PDA

View Full Version : Sabermetrics in action

RedsManRick
03-12-2010, 11:19 AM
Tango has a good post up over on his blog discussion the ability of models to predict HR totals. Here's the part I found most illuminating regarding how variation and regression work in practice, but the whole article is worth a read if you are dubious about systematic predictions. (the short: they are as accurate as a systematic process can be, but still result in pretty big error bars on an individual basis)

Tango's Blog Post
(If you're not familiar with Tom Tango, aka Tangotiger, he's one of the leading sabermetricians, more heavily focused on the detailed quant stuff. He and some colleagues wrote a great book called, THE BOOK--Playing The Percentages In Baseball and done some work for various professional teams, including most recently the Mariners. The stuff on his blog isn't always the most accessible to the non math geeks among us, but it's usually pretty high quality stuff.)

Can we prove that? A simple forecasting system I developed is called Marcel The Monkey Forecasting System, or The Marcels, for short. It’s named after the monkey from the TV show Friends. I also like the name Marcel for the hockey great Marcel Dionne so even if the name looks dated, you can think of Dionne instead. Anyway, Marcel listed 13 players as having a forecasted mean of 28 or more home runs for the 2009 season. Here are those hitters:

40 Howard, Ryan
32 Rodriguez, Alex
32 Fielder, Prince
32 Braun, Ryan
31 Pujols, Albert
31 Pena, Carlos
30 Thome, Jim
29 Dye, Jermaine
28 Cabrera, Miguel
28 Berkman, Lance
28 Beltran, Carlos

Now, remember what I said, and this is important: we are NOT forecasting Pujols to hit 31 HR in 2009. We forecasted him to hit 31 HR give or take 20 or 30 HR. You apply that same kind of thinking for each of the above players. And, we are NOT forecasting Ryan Howard to led the league with 40 HR. We ARE forecasting SOMEONE to hit around 50 HR. And these guys our among our best bets. With the top-end of each of these hitters close to 50 HR, obviously the average will be much lower.

How many HR did these players hit in 2009?

47 Pujols, Albert
46 Fielder, Prince
45 Howard, Ryan
39 Pena, Carlos
34 Cabrera, Miguel
32 Braun, Ryan
30 Rodriguez, Alex
27 Dye, Jermaine
25 Berkman, Lance
23 Thome, Jim
10 Beltran, Carlos

As you can see, it runs the gamut from Delgado’s 4 to Pujols’ league-leading 47. These 13 hitters were forecasted to hit a combined 401 HR in 2009. And how many HR did they actually hit in 2009? 400. That’s right, Marcel nailed it.

So, the forecasting systems work… if you know how to properly interpret what it is they are trying to tell you.

Plus Plus
03-12-2010, 11:29 AM
Thanks for sharing this, Rick. This is interesting- can you post a link to his blog?

gonelong
03-12-2010, 11:32 AM
Frankly, that's something my 6 year old could tell me.

GL

RedsManRick
03-12-2010, 11:32 AM
Thanks for sharing this, Rick. This is interesting- can you post a link to his blog?

Done. Thanks for the heads up.

RedsManRick
03-12-2010, 11:33 AM
Frankly, that's something my 6 year old could tell me.

GL

Your six year old would have predicted Dunn for just 32 HR? You have a very smart kid.

jojo
03-12-2010, 11:35 AM
Frankly, that's something my 6 year old could tell me.

GL

I think the snark is missing the point... the model is capturing the cohort really well...i.e. it works and the population is being accurately defined....

That actually isn't a trivial thing....

edabbs44
03-12-2010, 11:41 AM
If someone can explain how this could be utilized in any way, it would be appreciated. I am struggling to see the value of something like this, if this wasn't a fluke.

Cedric
03-12-2010, 11:42 AM
If someone can explain how this could be utilized in any way, it would be appreciated. I am struggling to see the value of something like this, if this wasn't a fluke.

It could make you the king of fantasy baseball for a year!

lollipopcurve
03-12-2010, 11:44 AM
Frankly, that's something my 6 year old could tell me.

I know. I'm sure the math used to develop the projections is sophisticated and worthy of praise, but how hard is it really to list a bunch of guys who are likely to hit a bunch of home runs?

Claiming that "Marcel nailed it" because as a group the players hit almost exactly the number of HRs predicted seems like a bit of gratuitous self-congratulation to me. How likely is it that such a result would repeat itself year by year? And, even if it did, what value does it have to anyone that there's a formula to forecast the total number of HRs hit by players forecast to hit 28 HRs or more?

Plus Plus
03-12-2010, 11:48 AM
Done. Thanks for the heads up.

Thanks, Rick! :thumbup:

jojo
03-12-2010, 11:49 AM
Marcels is the "mean" without percentiles (i.e. Pecota's 50th percentile).

Its a projection system... what is shared above is basically a test of how well the aggregate model captured the population.

RedsManRick
03-12-2010, 11:53 AM
If someone can explain how this could be utilized in any way, it would be appreciated. I am struggling to see the value of something like this, if this wasn't a fluke.

It's not really about this specific analysis or model. It's about understanding the nature of projection and the limits of modeling. With just 1 year of data, we can get really, really good about describing a system or large group of observations.

But the most accurate model still results in big, big error bars around individual players. Opponents of the sabermetric approach often like to cherry pick specific examples to question the credibility of a model. This hopefully sheds some light on why that is unfair.

That said, the question about utility is fair. I would actually argue that such a model actually helps us better understand the limits of projection. Too often people like to make projections about individual players and add them together -- but what usually happens in those cases is an across the board optimism bias. Models like this help us better understand the true probabilit

To your question, lollipop, "How likely is it that such a result would repeat itself year by year?" -- the answer is extremely. That's what makes it so powerful. What's the value? Well, if the Mariners had run a basic Marcel on their roster after 2007 and seen that, in the aggregate they were due for a boatload of regression, they might not have the disastrous Bedard trade.

Yeah, it's really simple stuff. And yet it still eludes a TON of people.

_Sir_Charles_
03-12-2010, 11:55 AM
Fluke is how I would describe it too. I wonder just how often they make projections like this and they come out so far off it's laughable. But get one right and they write up a big synopsis on the value of it. Sorry, ain't buyin' it. Not to mention...it has no use what-so-ever.

jojo
03-12-2010, 11:59 AM
Fluke is how I would describe it too. I wonder just how often they make projections like this and they come out so far off it's laughable.

You asked the question.... but the next step is to find the answer..... (and you can).

But get one right and they write up a big synopsis on the value of it. Sorry, ain't buyin' it. Not to mention...it has no use what-so-ever.

The conclusion seems premature (i.e. a step too early).

lollipopcurve
03-12-2010, 12:04 PM
To your question, lollipop, "How likely is it that such a result would repeat itself year by year?" -- the answer is extremely.

Are you saying it's extremely likely that the model will produce a result that's 1 HR off, or exactly right, for total # of HRs by the group? I would find that pretty amazing.

What's the value? Well, if the Mariners had run a basic Marcel on their roster after 2007 and seen that, in the aggregate they were due for a boatload of regression, they might not have the disastrous Bedard trade.

I buy this, more or less. But part of the mistake the Mariners made was in undervaluing their prospects (not just overvaluing their major league roster).

RedsManRick
03-12-2010, 12:12 PM
Are you saying it's extremely likely that the model will produce a result that's 1 HR off, or exactly right, for total # of HRs by the group? I would find that pretty amazing.

Will it be exactly on or 1 HR off? Not always. But what do you consider accurate? What if it's within 3 HR 90% of the time? There are plenty of studies out there that show the accuracy of the various projections systems. You might be surprised how accurate a simple model based on regression the mean can be.

I buy this, more or less. But part of the mistake the Mariners made was in undervaluing their prospects (not just overvaluing their major league roster).

The specific prospects they chose to trade were a result of what you've cited. But the logic of the trade was based on a belief that they were 1 big piece away when reasonable projections suggested that they really weren't even close.

lollipopcurve
03-12-2010, 12:21 PM
Will it be exactly on or 1 HR off? Not always. But what do you consider accurate? What if it's within 3 HR 90% of the time? There are plenty of studies out there that show the accuracy of the various projections systems. You might be surprised how accurate a simple model based on regression the mean can be.

I do find that interesting, but primarily as a fan of math, not baseball.

The specific prospects they chose to trade were a result of what you've cited. But the logic of the trade was based on a belief that they were 1 big piece away when reasonable projections suggested that they really weren't even close.

I think you're mainly right. Though you could argue that an ownership-driven "win-now" approach had long-since poisoned the well, thus dooming any develop-from-within strategy.

Ltlabner
03-12-2010, 02:26 PM
Why people get stuck on "it won't tell me exactly hits a player will have next year" as a way to dismiss sabermetrics is beyond me.

The use of statistics paints a picture of what is *possible* in the future in broad strokes. They won't tell you that player X will hit into exactly 67.8 double-plays but they can tell you if you are overestimating the power of your offense. Seems to me that is an incredibly powerful tool.

Yes, the games need to be played and a players performance is not locked in stone. But using some of these tools as a check against self-delusion is an incredibly powerful tool for those willing to use them.

Hoosier Red
03-12-2010, 02:40 PM
Okay so explain the value in thisl;

Now, remember what I said, and this is important: we are NOT forecasting Pujols to hit 31 HR in 2009. We forecasted him to hit 31 HR give or take 20 or 30 HR. You apply that same kind of thinking for each of the above players. And, we are NOT forecasting Ryan Howard to led the league with 40 HR. We ARE forecasting SOMEONE to hit around 50 HR. And these guys our among our best bets. With the top-end of each of these hitters close to 50 HR, obviously the average will be much lower.

Basically the system is projecting Albert Pujols to hit somewhere between 1 and 61 home runs? Even taken as a whole, it projected the sub set of 13 hitters to hit 401 home runs and says it nailed the prediction because they hit 400. If they had combined to hit 300 would it have been invalidated?

kpresidente
03-12-2010, 02:42 PM
Well, the problem is that *some* people might misuse the model to try and claim that they DO know exactly how many hits a player will have next year, and use the credibility of the modeling to back up their claim. The layman they're speaking with may not understand that the model is being misused, but does understand that modeling in general can be misused, and so reserves the right to dismiss the entire thing, not because they necessarily doubt the math, but rather because they don't understand it, and they don't trust the honesty or competence of the person expounding it to them.

An example in another arena that illustrates the point: Your credit card company understands finance a lot better than you do. And their models are no doubt correct. That doesn't mean you should believe everything they tell you.

edabbs44
03-12-2010, 02:45 PM
Okay so explain the value in thisl;

Now, remember what I said, and this is important: we are NOT forecasting Pujols to hit 31 HR in 2009. We forecasted him to hit 31 HR give or take 20 or 30 HR. You apply that same kind of thinking for each of the above players. And, we are NOT forecasting Ryan Howard to led the league with 40 HR. We ARE forecasting SOMEONE to hit around 50 HR. And these guys our among our best bets. With the top-end of each of these hitters close to 50 HR, obviously the average will be much lower.

Basically the system is projecting Albert Pujols to hit somewhere between 1 and 61 home runs? Even taken as a whole, it projected the sub set of 13 hitters to hit 401 home runs and says it nailed the prediction because they hit 400. If they had combined to hit 300 would it have been invalidated?

I'm also curious as to what the post-article parade would have been like if Delgado didn't miss 140 games and if ARod didn't have hip surgery.

edabbs44
03-12-2010, 02:45 PM
Well, the problem is that *some* people might misuse the model to try and claim that they DO know exactly how many hits a player will have next year, and use the credibility of the modeling to back up their claim. The layman they're speaking with may not understand that the model is being misused, but does understand that modeling in general can be misused, and so reserves the right to dismiss the entire thing, not because they necessarily doubt the math, but rather because they don't understand it, and they don't trust the honesty or competence of the person expounding it to them.

An example in another arena that illustrates the point: Your credit card company understands finance a lot better than you do. And their models are no doubt correct. That doesn't mean you should believe everything they tell you.

I think the bolded part could be questioned. The same could have been said for a lot of Wall St traders just 5 years ago.

kpresidente
03-12-2010, 02:53 PM
I think the bolded part could be questioned. The same could have been said for a lot of Wall St traders just 5 years ago.

But doesn't that bring up another point? The models may be consistent and rigorous, but they're only a representation of the real world. Mistaken assumptions can lead to gross errors.

nate
03-12-2010, 03:02 PM
Why people get stuck on "it won't tell me exactly hits a player will have next year" as a way to dismiss sabermetrics is beyond me.

The use of statistics paints a picture of what is *possible* in the future in broad strokes. They won't tell you that player X will hit into exactly 67.8 double-plays but they can tell you if you are overestimating the power of your offense. Seems to me that is an incredibly powerful tool.

Yes, the games need to be played and a players performance is not locked in stone. But using some of these tools as a check against self-delusion is an incredibly powerful tool for those willing to use them.

So powerful, I'd wager every front office uses their own version of them.

That's in addition to "the scouts."

westofyou
03-12-2010, 03:08 PM
I was afraid their was going to be an image like this and not an article.

http://www.rose-hulman.edu/thorn/media/images/2008/03/21-nerd-run.jpg

Falls City Beer
03-12-2010, 03:23 PM
What is the difference between a statistical approach and a sabremetric approach? Obviously every MLB FO uses statistics in their judgments, but does every FO use a sabremetric approach?

Isn't one inclusive of the other? Is there any exclusivity in these domains? Should we care?

Just seems like kind of a red herring. M2's point in the other thread is worth revisiting: maybe there's a point at which new numbers cease to add much to the decisions made by FOs. Maybe the progress narrative doesn't apply in all cases.

jojo
03-12-2010, 03:37 PM
Just seems like kind of a red herring. M2's point in the other thread is worth revisiting: maybe there's a point at which new numbers cease to add much to the decisions made by FOs. Maybe the progress narrative doesn't apply in all cases.

I think it's safe to say that the threshold isn't at ERA....

Falls City Beer
03-12-2010, 04:12 PM
I think it's safe to say that the threshold isn't at ERA....

Probably true. But this point doesn't ultimately answer my overarching question.

gonelong
03-12-2010, 04:35 PM
Your six year old would have predicted Dunn for just 32 HR? You have a very smart kid.

give or take 20 or 30 HR? I'll take my chances with the 6 year old. :D

Modeling? Interesting.

This model? meh.

GL

mth123
03-12-2010, 09:15 PM
give or take 20 or 30 HR? I'll take my chances with the 6 year old. :D

Modeling? Interesting.

This model? meh.

GL

:thumbup:

RedsManRick
03-13-2010, 12:08 AM
Feel free to stick with your six year old. But I bet they'd predict a bunch more homers than actually end up getting hit. It's not the models fault that you can't predict homers any better than that -- it's the reality of the variability of home run hitting. It's just that are brains aren't wired to naturally intuit variability.

And just to clarify, the 20 to 30 HR spread point isn't suggesting equal likelihood across the span. It's much more of a bell curve around the number you see projected. Some guys go over, some under, most get in the ballpark. If you had your 6 year old do the projections, it's quite likely he won't be allowing for the under-performance that we're bound to see from some people in the group.

What's amazing about Marcel is not it's ability to project perfectly. What's amazing is that such a simple model using just 3 years of past performance, an age adjustment, and a heavy dose of regression to the mean can do almost as well as the most sophisticated and accurate projection methods in the world. And sorry, but your 6 year old, or even "the experts", don't do nearly as well as you'd think they'd do by comparison.

If modeling interests you, Marcel should interest you -- even if this example doesn't terribly excite you.

For anybody so inclined, here's an analysis of well the most popular projection systems performed in 2009. But beware, there's statistics involved: http://sites.google.com/site/steamerprojections/2009-forecast-evalutions

ochre
03-13-2010, 01:12 AM
It's just a Fermi Problem with mean HRs for the individual players as the estimates. It would probably be less accurate with fewer players.

nate
03-13-2010, 08:29 AM
There's a great deal of supplemental info in the comments at the link.

It's not that "he'll hit x +/- 20-30" HR. It's that it's nearly 100% likely that he'll do so. Higher probabilities center around the likeliest amount.

Projections vs. predictions, folks.

lollipopcurve
03-13-2010, 11:15 AM
For anybody so inclined, here's an analysis of well the most popular projection systems performed in 2009. But beware, there's statistics involved: http://sites.google.com/site/steamer...ast-evalutions

First sentence of the article's conclusion:

It's hard to know exactly what to take from this.

Yeah.

jojo
03-13-2010, 11:49 AM
To me it's pretty straightforward.

Projections systems like Marcels, CHONE, Pecota, ZiPs etc are all pretty good good at estimating a player's true skill level and accounting for the effects of aging. They each take some different paths to getting the final answer so each has some strengths and some blind spots. All of them more or less end up in a relatively similar place in the aggregate (though they may disagree about certain players).

Projection systems like Marcels, CHONE, Pecota, ZiPs etc suck at estimating playing time or luck (i.e BABIP, LOB% etc).

That's kind of the point of the Tango link precipitating this thread.

It's a very significant thing to be able to accurately estimate a player's true skill level-its a huge advance. That is the essence of sabermetrics and really answers the question posed earlier about the value of and need/use for these second generation metrics.

So, projecting the potential to produce at a certain level is big deal because it estimates the reasonable expectation for that player going forward which allows for his impact to be "budgeted" for when roster building (both in terms of how much effect on RS/RA and what it should be worth). In other words, the value is that they (projection systems) can summarize a true skill level in a rate stat like wOBA not that they can also translate that into counting stats (i.e. predict the exact number of homers or doubles a guy will hit). Second generation metrics basically add more precision to projection-something that scouting lacks.

The modern projection systems do a pretty good job of it too despite "stuff happening" like injury and luck.....