I read this comment by Buster Olney on defensive ratings in his ESPN chat and thought some here might consider it interesting because I know several here put a lot of stock into them. I know some will not like his opinion and will downplay it simply because it's Olney, but I tend to agree with it.
Bill (Hartford): "In this era of stats worship, which stat do you think is the most overused, overanalyzed, and generally overrated predictor of actual performance?"
SportsNation Buster Olney: Bill: "A lot of the defensive stats are a joke and have an inherent flaw that has never really been addressed -- typically, these systems employ an evaluator stationed in each park to determine defensive positioning. That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city, to things like how infielders lean a step (or more) if they know a breaking ball is being thrown. And you can't tell me one evaluator can accurately track the movement of seven fielders (those besides the pitcher and the catcher). That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical. It's ironic that a lot of the statistical evaluators are very skeptical of scouts' evaluations, and yet when it comes to defensive numbers, they are, in effect, basing their results on a common denominator -- one evaluator making subjective judgments."