Olney's comments on defensive ratings......

[jimbo]

I read this comment by Buster Olney on defensive ratings in his ESPN chat and thought some here might consider it interesting because I know several here put a lot of stock into them. I know some will not like his opinion and will downplay it simply because it's Olney, but I tend to agree with it.

Bill (Hartford): "In this era of stats worship, which stat do you think is the most overused, overanalyzed, and generally overrated predictor of actual performance?"

SportsNation Buster Olney: Bill: "A lot of the defensive stats are a joke and have an inherent flaw that has never really been addressed -- typically, these systems employ an evaluator stationed in each park to determine defensive positioning. That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city, to things like how infielders lean a step (or more) if they know a breaking ball is being thrown. And you can't tell me one evaluator can accurately track the movement of seven fielders (those besides the pitcher and the catcher). That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical. It's ironic that a lot of the statistical evaluators are very skeptical of scouts' evaluations, and yet when it comes to defensive numbers, they are, in effect, basing their results on a common denominator -- one evaluator making subjective judgments."

[RedsManRick]

I love the "stats aren't perfect, therefore they are useless" argument. I hope Buster doesn't trust the strikes and balls data he gets back. Each game uses a different umpire, with a different set of eyes, to interpret the pitches. Therefore, any inferences about a pitchers ability to throw strikes based on a human umpire's decisions cannot be judged with any objectivity.

Yes, Buster, it's not perfect. Yes, there's subjectivity involved. But what is the alternative? I don't know of any statistician, or sabermatrician, who suggests that you should ignore the stating reports, or that scouts don't provide valuable insight. But how is this any different than the scout himself, who has his own set of biases and limitations?

[flyer85]

which is why you don't rely on just one inidicator. However, when you've have a number of them all agreeing about a player it is probably a good idea to assume where there is smoke there is fire.

[M2]

Well, there's some hyperbole in there, but I generally agree with Olney. First off, a lot of those systems use a grid, so it doesn't particularly matter where fielder X is standing. What's being measured is where the ball goes on the grid and whether the fielder reaches it.

Mind you, Olney's hit upon the problem with his "30 different perspectives" point. Not everyone perceives the grid the same way. A fielder with a lot of range might go off-grid an almost make a play, earning a demerit because he's made a ball look like it's in his grid while another guy may not get anywhere close to a ball that a rangier guy would offer at (Juan Castro leaps to mind here), suffering no penalty when he should get it.

Now, work has been done to whittle down that perceptual problem and teams pay big money for defensive data with the subjectivity better removed from the results. I've always thought that until you measure distance covered over time you're not going to get at the core of what constitutes defense. As far as I know, there's no public data on that.

Though Olney should know better than to tar "defensive efficiency" as it's wholly different from anything he's talking about. Team defensive efficiency measures how well a team fields balls in play. It might be the defensive metric ever as it involves no subjectivity and it uses big sets of data.

[Cedric]

Originally Posted by RedsManRick

I love the "stats aren't perfect, therefore they are useless" argument. I hope Buster doesn't trust the strikes and balls data he gets back. Each game uses a different umpire, with a different set of eyes, to interpret the pitches. Therefore, any inferences about a pitchers ability to throw strikes based on a human umpire's decisions cannot be judged with any objectivity.

Yes, Buster, it's not perfect. Yes, there's subjectivity involved. But what is the alternative? I don't know of any statistician, or sabermatrician, who suggests that you should ignore the stating reports, or that scouts don't provide valuable insight. But how is this any different than the scout himself, who has his own set of biases and limitations?

I don't think he is saying that a scout himself is perfect. The problem with ratings and numbers is they become official and there is no going back. I think that's what his point is.

[Redsland]

Originally Posted by RedsManRick

Each game uses a different umpire, with a different set of eyes, to interpret the pitches.

And a different official scorer.

Originally Posted by Buster Olney

That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city.... That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical.

Hit? Error? It's all very subjective. It's a good thing widely quoted stats like "fielding percentage" aren't affected by such bias. Why, such a thing could taint the sanctity of the Gold Glove.

[TeamSelig]

Agreed!

I've been saying defensive stats are pretty much useless for a while now

[RedsManRick]

My point is that there's very little in the game which isn't subjective on some level. I'm just a wary of the subjectivity of stats which tell me Derek Jeter isn't a great defensive SS based on his inability to go laterally as I am the people who claim he is great based on the dozens of plays they've seen him make. If the difference between a great fielder an a poor one, using errors is 30 players per 1000, I'm guessing that a sample of plays your average commentator sees isn't a great sample.

[SteelSD]

SportsNation Buster Olney: Bill: "A lot of the defensive stats are a joke and have an inherent flaw that has never really been addressed -- typically, these systems employ an evaluator stationed in each park to determine defensive positioning. That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city, to things like how infielders lean a step (or more) if they know a breaking ball is being thrown. And you can't tell me one evaluator can accurately track the movement of seven fielders (those besides the pitcher and the catcher). That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical.

It'd be helpful to Olney if he actually understood how things like Zone Rating are tracked. If he knew, he'd actually sound coherant and could possibly produce a reasonable argument against the more objective publicly published defensive metrics.

First, something like Zone Rating doesn't result from someone's objective opinion of "should have been caught". ZR only cares about areas of responsibility. Is there a potential issue with determining balls that fall in a player's area of responsibility? Sure. Slight. But nothing about the "movement of seven fielders" Olney notes. ZR doesn't freakin' care about movement. It only cares about areas of responsibility. Now, I'm not saying that Olney was talking about Zone Rating, mind you. Frankly, I'd be surprised if he had the first clue about it (or anything resembling objective analysis).

The ZR metric, while not perfect, can only be dramatically skewed by the following:

1. The person tracking the ball is a moron.
2. The positioning of the fielder is always bad (which is either his problem or the coaching staff's problem).
3. The player didn't catch a ball hit into his zone of responsibility.

Sorry, but I'm going to cite Occam's Razor on that one.

It's ironic that a lot of the statistical evaluators are very skeptical of scouts' evaluations, and yet when it comes to defensive numbers, they are, in effect, basing their results on a common denominator -- one evaluator making subjective judgments."

Yeah, Buster? Well, the Reds just finished a season in which Ken Griffey Jr. played Center Field and in which Juan Castro and Royce Clayton were considered solid defensive options at Shortstop. If that's what relying on scouts gets ya', I'll take the alternative every seven days a week.

[GOOCH]

What's sad is that Buster Olney is a young guy. The same nonsense Bill James was fighting 20 years ago...from a twenty-something. Sigh. D.GOOCH

[Ron Madden]

I know a lot of fans who believe in statisical evaluation. Not one of them has ever claimed thier studies to be 100% fool proof. Each and everyone of them will tell us we need a mixture of both, Scouts and Sabermetric Tools.

Most fans I know discredit all stats other than a hitters BA, HR and RBI. Or a pitchers W-L record and ERA. This is understandable, since we all grew up believing these to be the only numbers that really matter.

Baseball like no other sport is linked to tradition.

Tradition=the handing down of customs, practices, and beliefs that are valued by a particular culture.

This particukar culture is made up of players, coaches, managers, scouts, gm's owners, broadcasters, writers and baseball fans.

New ideas are seldom embraced.

[BuckeyeRedleg]

I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

[LoganBuck]

Originally Posted by BuckeyeRedleg

I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

Not necessarily a bad idea though! You could put a radio frequency ID chip in the glove, without affecting the play of the game. Actually a darn good idea.

[Johnny Footstool]

Originally Posted by BuckeyeRedleg

I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

GPS is only accurate up to a few feet, though. Use a laser grid instead. Just make sure the players wear eye protection to avoid getting their retinas burned.

Buster Olney has a legitimate complaint about that particular system of evaluation. Of course, no decent team would rely on a system like that, and no decent stathead would put any more stock into it than, say, a scout's evaluation.

There are a number of other systems that involve significantly less subjectivity -- The Fielding Bible uses a pretty good system, and their evaluators watch every play of every game. Olney should educate himself before making a misleading statement like the one he just made.

Of course he won't, though. He's trying to make a name for himself as an "always goes against the grain" journalist.

[TC81190]

As far as fielding goes, I'd take what a scout says 9.9 times out of 10 over some statiscal analysis.