PDA

View Full Version : Olney's comments on defensive ratings......



jimbo
10-30-2006, 06:21 PM
I read this comment by Buster Olney on defensive ratings in his ESPN chat and thought some here might consider it interesting because I know several here put a lot of stock into them. I know some will not like his opinion and will downplay it simply because it's Olney, but I tend to agree with it.


Bill (Hartford): "In this era of stats worship, which stat do you think is the most overused, overanalyzed, and generally overrated predictor of actual performance?"

SportsNation Buster Olney: Bill: "A lot of the defensive stats are a joke and have an inherent flaw that has never really been addressed -- typically, these systems employ an evaluator stationed in each park to determine defensive positioning. That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city, to things like how infielders lean a step (or more) if they know a breaking ball is being thrown. And you can't tell me one evaluator can accurately track the movement of seven fielders (those besides the pitcher and the catcher). That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical. It's ironic that a lot of the statistical evaluators are very skeptical of scouts' evaluations, and yet when it comes to defensive numbers, they are, in effect, basing their results on a common denominator -- one evaluator making subjective judgments."

RedsManRick
10-30-2006, 06:52 PM
I love the "stats aren't perfect, therefore they are useless" argument. I hope Buster doesn't trust the strikes and balls data he gets back. Each game uses a different umpire, with a different set of eyes, to interpret the pitches. Therefore, any inferences about a pitchers ability to throw strikes based on a human umpire's decisions cannot be judged with any objectivity.

Yes, Buster, it's not perfect. Yes, there's subjectivity involved. But what is the alternative? I don't know of any statistician, or sabermatrician, who suggests that you should ignore the stating reports, or that scouts don't provide valuable insight. But how is this any different than the scout himself, who has his own set of biases and limitations?

flyer85
10-30-2006, 06:54 PM
which is why you don't rely on just one inidicator. However, when you've have a number of them all agreeing about a player it is probably a good idea to assume where there is smoke there is fire.

M2
10-30-2006, 06:56 PM
Well, there's some hyperbole in there, but I generally agree with Olney. First off, a lot of those systems use a grid, so it doesn't particularly matter where fielder X is standing. What's being measured is where the ball goes on the grid and whether the fielder reaches it.

Mind you, Olney's hit upon the problem with his "30 different perspectives" point. Not everyone perceives the grid the same way. A fielder with a lot of range might go off-grid an almost make a play, earning a demerit because he's made a ball look like it's in his grid while another guy may not get anywhere close to a ball that a rangier guy would offer at (Juan Castro leaps to mind here), suffering no penalty when he should get it.

Now, work has been done to whittle down that perceptual problem and teams pay big money for defensive data with the subjectivity better removed from the results. I've always thought that until you measure distance covered over time you're not going to get at the core of what constitutes defense. As far as I know, there's no public data on that.

Though Olney should know better than to tar "defensive efficiency" as it's wholly different from anything he's talking about. Team defensive efficiency measures how well a team fields balls in play. It might be the defensive metric ever as it involves no subjectivity and it uses big sets of data.

Cedric
10-30-2006, 07:00 PM
I love the "stats aren't perfect, therefore they are useless" argument. I hope Buster doesn't trust the strikes and balls data he gets back. Each game uses a different umpire, with a different set of eyes, to interpret the pitches. Therefore, any inferences about a pitchers ability to throw strikes based on a human umpire's decisions cannot be judged with any objectivity.

Yes, Buster, it's not perfect. Yes, there's subjectivity involved. But what is the alternative? I don't know of any statistician, or sabermatrician, who suggests that you should ignore the stating reports, or that scouts don't provide valuable insight. But how is this any different than the scout himself, who has his own set of biases and limitations?

I don't think he is saying that a scout himself is perfect. The problem with ratings and numbers is they become official and there is no going back. I think that's what his point is.

Redsland
10-30-2006, 09:32 PM
Each game uses a different umpire, with a different set of eyes, to interpret the pitches.
And a different official scorer.

That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city.... That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical.
Hit? Error? It's all very subjective. It's a good thing widely quoted stats like "fielding percentage" aren't affected by such bias. Why, such a thing could taint the sanctity of the Gold Glove. ;)

TeamSelig
10-30-2006, 11:56 PM
Agreed!

I've been saying defensive stats are pretty much useless for a while now

RedsManRick
10-31-2006, 12:06 AM
My point is that there's very little in the game which isn't subjective on some level. I'm just a wary of the subjectivity of stats which tell me Derek Jeter isn't a great defensive SS based on his inability to go laterally as I am the people who claim he is great based on the dozens of plays they've seen him make. If the difference between a great fielder an a poor one, using errors is 30 players per 1000, I'm guessing that a sample of plays your average commentator sees isn't a great sample.

SteelSD
10-31-2006, 12:37 AM
SportsNation Buster Olney: Bill: "A lot of the defensive stats are a joke and have an inherent flaw that has never really been addressed -- typically, these systems employ an evaluator stationed in each park to determine defensive positioning. That means there are 30 different evaluators, meaning 30 different standards (in practice), 30 different perspectives, 30 different sets of eyes. The evaluator in one town might be much more attentive than the evaluator in another city, to things like how infielders lean a step (or more) if they know a breaking ball is being thrown. And you can't tell me one evaluator can accurately track the movement of seven fielders (those besides the pitcher and the catcher). That's why, when you see something about defensive efficiency -- 'These are the balls he should've caught' -- then you should be very, very, very skeptical.

It'd be helpful to Olney if he actually understood how things like Zone Rating are tracked. If he knew, he'd actually sound coherant and could possibly produce a reasonable argument against the more objective publicly published defensive metrics.

First, something like Zone Rating doesn't result from someone's objective opinion of "should have been caught". ZR only cares about areas of responsibility. Is there a potential issue with determining balls that fall in a player's area of responsibility? Sure. Slight. But nothing about the "movement of seven fielders" Olney notes. ZR doesn't freakin' care about movement. It only cares about areas of responsibility. Now, I'm not saying that Olney was talking about Zone Rating, mind you. Frankly, I'd be surprised if he had the first clue about it (or anything resembling objective analysis).

The ZR metric, while not perfect, can only be dramatically skewed by the following:

1. The person tracking the ball is a moron.
2. The positioning of the fielder is always bad (which is either his problem or the coaching staff's problem).
3. The player didn't catch a ball hit into his zone of responsibility.

Sorry, but I'm going to cite Occam's Razor on that one.


It's ironic that a lot of the statistical evaluators are very skeptical of scouts' evaluations, and yet when it comes to defensive numbers, they are, in effect, basing their results on a common denominator -- one evaluator making subjective judgments."

Yeah, Buster? Well, the Reds just finished a season in which Ken Griffey Jr. played Center Field and in which Juan Castro and Royce Clayton were considered solid defensive options at Shortstop. If that's what relying on scouts gets ya', I'll take the alternative every seven days a week.

GOOCH
10-31-2006, 02:18 AM
What's sad is that Buster Olney is a young guy. The same nonsense Bill James was fighting 20 years ago...from a twenty-something. Sigh. D.GOOCH

Ron Madden
10-31-2006, 03:13 AM
I know a lot of fans who believe in statisical evaluation. Not one of them has ever claimed thier studies to be 100% fool proof. Each and everyone of them will tell us we need a mixture of both, Scouts and Sabermetric Tools.

Most fans I know discredit all stats other than a hitters BA, HR and RBI. Or a pitchers W-L record and ERA. This is understandable, since we all grew up believing these to be the only numbers that really matter.

Baseball like no other sport is linked to tradition.

Tradition=the handing down of customs, practices, and beliefs that are valued by a particular culture.

This particukar culture is made up of players, coaches, managers, scouts, gm's owners, broadcasters, writers and baseball fans.

New ideas are seldom embraced.

BuckeyeRedleg
10-31-2006, 11:44 AM
I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

:)

LoganBuck
10-31-2006, 11:58 AM
I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

:)

Not necessarily a bad idea though! You could put a radio frequency ID chip in the glove, without affecting the play of the game. Actually a darn good idea.

Johnny Footstool
10-31-2006, 12:07 PM
I know what would solve the subjective defensive measurement issue.

Implant a chip into each player and track their movements by GPS. That with the help of video coverage from each game would pretty much remove any subjectivity from the equation.

:)

GPS is only accurate up to a few feet, though. Use a laser grid instead. Just make sure the players wear eye protection to avoid getting their retinas burned.

Buster Olney has a legitimate complaint about that particular system of evaluation. Of course, no decent team would rely on a system like that, and no decent stathead would put any more stock into it than, say, a scout's evaluation.

There are a number of other systems that involve significantly less subjectivity -- The Fielding Bible uses a pretty good system, and their evaluators watch every play of every game. Olney should educate himself before making a misleading statement like the one he just made.

Of course he won't, though. He's trying to make a name for himself as an "always goes against the grain" journalist.

TC81190
10-31-2006, 04:52 PM
As far as fielding goes, I'd take what a scout says 9.9 times out of 10 over some statiscal analysis.

Johnny Footstool
10-31-2006, 05:06 PM
As far as fielding goes, I'd take what a scout says 9.9 times out of 10 over some statiscal analysis.

That's fine.

I'll search for a less subjective way to evaluate fielding.

M2
10-31-2006, 05:18 PM
As far as fielding goes, I'd take what a scout says 9.9 times out of 10 over some statiscal analysis.

I think scouts do a good job on fielding analysis. Though, like anything, it helps to verify that analysis. For instance, if a scout is telling you that Jr.'s all right in CF despite the overwhelming mass of evidence to the contrary then it's time for that scout to look again.

For the most part, defensive statistics are clues that should form part of the entire fielding analysis done on the player. They can point to certain strengths or weaknesses. If you know that Jr. has made significantly fewer plays in CF for the Reds than everybody else who's manned the position for the Reds for at least 100 innings in a given season (seven players total) for five seasons running, then I'd think a good scout would start paying attention to the jumps he gets and the ground he covers.

The one thing I'd want no part of is a scout who rejects good information. It seems to me that anyone who can't absorb the information the game provides, whether it be the numbers derived from the game or from the physics of playing the game, probably isn't a very good scout, or at least is a scout whose limitations a franchise needs to understand.

Rojo
10-31-2006, 06:27 PM
Is there a potential issue with determining balls that fall in a player's area of responsibility? Sure. Slight.

How slight? We know that Range Factor is pitching dependent (less K's, more chances) but I wonder if ZR is as well. Seems that a groundball pitcher is going to induce more easy pickups in a given zone. Could be a slight difference, I haven't seen anything.

RedsManRick
10-31-2006, 06:52 PM
I think scouts do a good job on fielding analysis. Though, like anything, it helps to verify that analysis. For instance, if a scout is telling you that Jr.'s all right in CF despite the overwhelming mass of evidence to the contrary then it's time for that scout to look again.

For the most part, defensive statistics are clues that should form part of the entire fielding analysis done on the player. They can point to certain strengths or weaknesses. If you know that Jr. has made significantly fewer plays in CF for the Reds than everybody else who's manned the position for the Reds for at least 100 innings in a given season (seven players total) for five seasons running, then I'd think a good scout would start paying attention to the jumps he gets and the ground he covers.

The one thing I'd want no part of is a scout who rejects good information. It seems to me that anyone who can't absorb the information the game provides, whether it be the numbers derived from the game or from the physics of playing the game, probably isn't a very good scout, or at least is a scout whose limitations a franchise needs to understand.

M2 wins. It's not about either-or. It's about both. The "stats" will show you things you might not be able to tell by watching a select handful of games. They'll tell you exactly what they tell you "So and so fielded x amount of balls hit in to his fieldable area defined as blah blah blah". Take them for what they are with all the proper considerations. But they also only tell you what a player has done, not what he may be capable of doing.

Scouting will give you a better feel for things for how & where those numbers are coming from. IE. Junior reads the ball well off the bat and takes great routes but lacks accelleration and his top speed has diminished. There's also some inherent problems with the way people perceive and remember things that's just a problem with any observation. Human memory isn't perfect and even the most objective person has biases they can't always account for or may not even be aware of.

We don't and can't have perfect information. Information comes in many forms. Any person wishing to make a fair judgement would be silly to exclude information. To me, the reluctance to use fielding measurements is a reflection on a lack of familiarity with it and a fear that they don't understand exactly what the stat means.

Cooper
10-31-2006, 07:41 PM
As an observer of the debate- scouts vs stats....it seems to me the stats folks make a lot more room for the scouts perspective than vice versa (i'm talking in general-not this debate specifically).

Johnny Footstool
11-01-2006, 12:13 PM
As an observer of the debate- scouts vs stats....it seems to me the stats folks make a lot more room for the scouts perspective than vice versa (i'm talking in general-not this debate specifically).

To be fair, scouts use a lot of stats. They just tend to use traditional stats like RBI, batting average, pitching wins, etc.

In this particular debate, I don't think anyone can discount the scout -- you've got to have someone making observations. The thing is, you need to figure out a way to take all those observations and quantify them using a standard system of measurement.

Cooper
11-01-2006, 03:18 PM
The folks that make up the Fielding Bible are doing just what you say...they are making observations and quantifying them. I wouldn't be surprised at all if the tradional scouting community discounts the work they do because they view scouting as an art. And only artists deemed exceptable by the scouts themselves are in the "the know".

I watched the Andy Warhol story on PBS a couple of weeks ago. He was a commercial artist for the beginning of his career and other artists went out of their way to ostrasize him and his work. They felt he had made too much money and they just didn't like his work or him as a person. For years he struggled for exceptance until he decided he just didn't care...he began to say "so what" to any hardships that came along.

Imo, scouts are a lot like artists. They pick and choose what is art, who is exceptable, what is talent. They, also, believe they are uniquely qualified in determining even what they are about....moving the boundaries around for those they deem special. To gain entrance usually requires the person to give up some of their freedom and ideas. My guess is a lot of scouts are secret sabermetricians, but are too scared to admit it.

Humans do things like this so we can feel special-i understand that. In a better world there would be more freedom of ideas. More access.

Johnny Footstool
11-01-2006, 03:47 PM
The folks that make up the Fielding Bible are doing just what you say...they are making observations and quantifying them.

I'm very happy with the direction the producers of the Fielding Bible have taken. They're hacking away big chunks of subjectivity and producing a more objective vision of fielding prowess. They're not perfect -- not yet, but they're certainly headed in the right direction.


I wouldn't be surprised at all if the tradional scouting community discounts the work they do because they view scouting as an art. And only artists deemed exceptable by the scouts themselves are in the "the know".

I watched the Andy Warhol story on PBS a couple of weeks ago. He was a commercial artist for the beginning of his career and other artists went out of their way to ostrasize him and his work. They felt he had made too much money and they just didn't like his work or him as a person. For years he struggled for exceptance until he decided he just didn't care...he began to say "so what" to any hardships that came along.

Imo, scouts are a lot like artists. They pick and choose what is art, who is exceptable, what is talent. They, also, believe they are uniquely qualified in determining even what they are about....moving the boundaries around for those they deem special. To gain entrance usually requires the person to give up some of their freedom and ideas. My guess is a lot of scouts are secret sabermetricians, but are too scared to admit it.

Humans do things like this so we can feel special-i understand that. In a better world there would be more freedom of ideas. More access.

I know that a lot of people still enjoy the poetry of baseball -- the mysticism, the "secret knowledge" that only a lifetime in the game can impart. I think the major reason these people resist statistical analysis is because it breaks that secret knowledge down into numbers that anyone can understand.

M2
11-01-2006, 03:49 PM
Nice post Coop. Less than a century ago, Ivory Tower lit professors glommed onto something called New Criticism as a reaction to the success of certain writers like Ernest Hemingway and F. Scott Fitzgerald who were gaining notoriety as great writers thanks to the explosion of the publishing industry.

Think about it, people were deciding all on their own that these writers were great without ever asking a lit professor. Well, the lit professors made sure they came up with their little system of making sure no one else leapt over that threshhold without Ivory Tower approval. BTW, one of the tenets was that prose can never be as good as poetry. Seriously, that was one of the core rules they set up. Basically what they did was set up this little game where they insisted that everything new that you thought might have been great wasn't really as great as you thought it was and that you better check back with them the next time you think you've read something good.

You can sort of see where the same thing is happening in baseball. People have figured out how to get some serious insight from the numbers generated by the game (and it can't be stressed enough that the stats are the result of what's actually happened on the field). There's really no stuffing the genie back in the bottle. I figure the good scouts have mostly learned to incorporate the objective data with their subjective insights.

lollipopcurve
11-01-2006, 04:26 PM
Well, the lit professors made sure they came up with their little system of making sure no one else leapt over that threshhold without Ivory Tower approval. BTW, one of the tenets was that prose can never be as good as poetry.



"Even if Homer Bailey develops into an ace, he will still have been a bad draft choice." -- somewhere on Redszone

College pitcher always > high school pitcher = poetry always > prose.

Ideology runs on both sides of the debate.

Cooper
11-01-2006, 04:36 PM
red herring.

lollipopcurve
11-01-2006, 04:43 PM
red herring.

Not at all.

I'm responding to M2's post that suggests resistance to new defensive metrics is encased in reactionary ideology by pointing out that some stat-based arguments are also encased in reactionary ideology. I happen to agree with M2's point, but I want to add that the stat crowd does its share of ignoring evidence staring them in the face.

M2
11-01-2006, 04:46 PM
"Even if Homer Bailey develops into an ace, he will still have been a bad draft choice." -- somewhere on Redszone

College pitcher always > high school pitcher = poetry always > prose.

Ideology runs on both sides of the debate.

Boy, you still don't understand the whole college pitcher vs. HS pitcher thing. I doubt there's even half a dozen people on this site who think it's always better to pick a college pitcher over a HS pitcher.

Just for the record, the Reds probably would have won the division this year had they picked Jered Weaver instead of Bailey. Mind you, I'm wincing too much over the team not taking Tim Lincecum this year to fret much over that one these days.

lollipopcurve
11-01-2006, 04:57 PM
Boy, you still don't understand the whole college pitcher vs. HS pitcher thing. I doubt there's even half a dozen people on this site who think it's always better to pick a college pitcher over a HS pitcher.

Just for the record, the Reds probably would have won the division this year had they picked Jered Weaver instead of Bailey. Mind you, I'm wincing too much over the team not taking Tim Lincecum this year to fret much over that one these days.

If you can't divine that I was making a point about ideology -- i.e., that ANY HS pitcher is necessarily a poor choice (where Bailey was chosen), no matter how he develops -- not about the relative merits of Bailey vs. Weaver, because, after all, you can pretty much always find a player of one ilk (HS pitcher or HS hitter, college pitcher or college hitter) who outperforms a player of another ilk taken earlier in the draft, then I'll just chalk this response of yours up to another outburst of arrogance and defensiveness. Because my point was pretty clear.

lollipopcurve
11-01-2006, 05:01 PM
I doubt there's even half a dozen people on this site who think it's always better to pick a college pitcher over a HS pitcher.

Ah yes, from a guy who was telling me about a year ago that he'd rather have ANY of the college pitchers taken in the 04 first round over Bailey. I wonder what that was based on?

M2
11-01-2006, 05:14 PM
If you can't divine that I was making a point about ideology -- i.e., that ANY HS pitcher is necessarily a poor choice (where Bailey was chosen), no matter how he develops -- not about the relative merits of Bailey vs. Weaver, because, after all, you can pretty much always find a player of one ilk (HS pitcher or HS hitter, college pitcher or college hitter) who outperforms a player of another ilk taken earlier in the draft, then I'll just chalk this response of yours up to another outburst of arrogance and defensiveness. Because my point was pretty clear.

I thought it was pretty clear that your point was confused. Now apparently the criteria you're using to laud the Bailey pick can't be used to demonstrate that there was a better-regarded college arm on the board who would have made an even better pick.

Weaver wasn't the guy I'd have picked at the time, but the argument for him (best prospect on the board and you can pay him instead of some lousy veteran like Eric Milton or Paul Wilson or Ramon Ortiz and you'd have a pitcher who could move to the front of your rotation in the near term) has turned out to be flawless. Sorry that I've arrogantly and defensively bought into someone else's line of thinking on this.

Johnny Footstool
11-01-2006, 05:30 PM
College pitcher always > high school pitcher = poetry always > prose.

That's a false analogy.

There is significant data to support the position that college pitchers are better draft picks than HS pitchers.

There is no data to support the subjective poetry vs. prose argument.

If you're trying to argue that both statheads and seamheads are guilty of making absolute judgements, you need to come up with a better example.

M2
11-01-2006, 05:31 PM
Ah yes, from a guy who was telling me about a year ago that he'd rather have ANY of the college pitchers taken in the 04 first round over Bailey. I wonder what that was based on?

Not any. Down in the bottom of the first round I'd have been all over a kid like Bailey. The Dodgers had a sweet pick down there with Scott Elbert and Phillip Hughes was a nice grab by the Bombers. IIRC, I've done nothing but praise those picks since the moment they were made. You know we had a mock draft on the site this summer and I picked a HS pitcher with my first pick.

But if you want to play the game of who the Reds should have picked first in 2004 with the full benefit of hindsight, the answer is Jered Weaver. Personally, I give credit to the folks who called that one beforehand.

Anyway, I'm thrilled Bailey did what he did in 2006. I think he's particularly intersting at the moment because he's probably the most valuable talent in the organization. In that way he's turned out to be a very good pick because he presents the opportunity to get frontline production out of that pick. The billion dollar question is whether the way to get that production by dealing him (something that would be nearly unprecedented) or by keeping him (begging the question of whether he's Josh Beckett or Edwin Jackson).

Point here being that all the scouting director can do is deliver opportunity to the franchise. By that criteria Terry Reynolds made an excellent pick on Homer Bailey. It's now up to Wayne Krivsky to determine what to do with that opportunity. Hopefully it's a case of figuring out the best of two good choices.

SteelSD
11-01-2006, 11:14 PM
If you can't divine that I was making a point about ideology -- i.e., that ANY HS pitcher is necessarily a poor choice (where Bailey was chosen), no matter how he develops -- not about the relative merits of Bailey vs. Weaver, because, after all, you can pretty much always find a player of one ilk (HS pitcher or HS hitter, college pitcher or college hitter) who outperforms a player of another ilk taken earlier in the draft, then I'll just chalk this response of yours up to another outburst of arrogance and defensiveness. Because my point was pretty clear.

Considering how badly you've convoluted the opposing position at the time of the trade, you really have zero footing right now.

Virtually no one, myself included, who argued that the Reds took a bad risk with the Bailey selection has also argued that they would never take a HS arm in the first round. That's been explained enough to you in no uncertain terms so color me confused as to why you'd need it explained again. You're obviously not forgetful and I think you have the ability to comprehend the written word so the only alternative is that you are willingly and knowingly misrepresenting the opposing position.

Heck, in the 2006 Redszone mock draft, I selected Clayton Kershaw with the 14th pick. If your made-up opposing position were accurate, that couldn't have happened.

Not that I don't appreciate the creativity it takes to weave a strawman, a red herring, and an example of argumentum ad hominem into a post, but do you actually feel that such a trifecta of fallacy furthers your argument?