Originally Posted by M2
I think you have to tackle defense with a scattershot. I've yet to see a methodology without critical flaws. I like the idea of PBP assessment, but I think they measure the wrong thing. I like Dewan's idea of measuring plays made instead of runs makes sense. I think Bill James has lapped the field about two times over in terms of equating defensive value with performance at the plate. I think, given the holes in the methodology, that common sense and observation have a huge role to play.
I also think FCB has an excellent eye (he's proven it time and again) and that he does fantastic job of cutting through the noise to the heart of the matter in most cases. He's got horse sense. I respect the hell out of that.
In regards to scatter-shot, Justin did use 3 different systems in his take on plus/minus. His weight formula was admittedly arbitrary, but that said:
+-Fielding = 0.375*PMR + 0.375*UZR + .25*FSR
PRM: Probabilistic Model of Range (using BIS data)
UZR: Ultimate Zone Rating (using STATS Inc data)
FSR: Fan Scouting Report (run by Tom Tango)
That's two different quantitative systems, based on qualitative coding, and one purely qualitative source generated from a spread of independent observers.
I don't mean to demean FBC's eye and should not have gone down that road. Rather, I question the ability of anybody's eye to assess the specific question being assessed by the study presented here. It is very difficult for any person to
- neutralize a single season's performance of a given player from their established body of work
- make an assessment of that player's performance (not ability) relative to a dozen or more peers
- compare two players at different positions on this basis
I just don't think you can make all the necessary adjustments in your head. We hear all the time evaluations which are summary assessments of ability, rather than performance. "Ken Griffey Jr. is a bad RF" -- rather than "In 2007, Ken Griffey Jr. made fewer plays than the average RF, thus resulting in fewer runs prevented". Qualitative assessments tend to lend themselves to more holistic evaluations, whereas quantitative ones are necessarily specific. Neither approach is right or wrong per se', but the conclusions are different nonetheless.
I think we can all agree that both Griffey and Dunn are below average fielders at their positions, and overall, that their defensive value (or lackthereof) is in the same ballpark -- pardon the pun. The metric difference in terms of Justin's +- Runs calculation is on the order of a 1 play difference a month.
I think we'd be very well served to treat any +- metric with a healthy confidence interval and I'm pretty sure that the measured difference between Dunn and Junior would fall within that such that we can say they are not significantly different.