View Full Version : A question about small sample sizes

BCubb2003

04-15-2012, 12:17 PM

I heard from an engineering acquaintance recently that when we talk about a small sample size in baseball, we usually mean a small universe. As in, looking at the first week of at-bats would be a small universe. Looking at one in five at-bats (or maybe the more common "throw out this game and that game and he looks pretty good") would be a small sample size.

What say you guys?

kaldaniels

04-15-2012, 12:28 PM

It depends what stat you are talking about. Fangraphs had a nice article on it a while back.

757690

04-15-2012, 12:33 PM

Well, I have always been told that size doesn't matter, but then again, they might just have been trying not to hurt my feelings :cool:

RedsManRick

04-15-2012, 01:25 PM

I heard from an engineering acquaintance recently that when we talk about a small sample size in baseball, we usually mean a small universe. As in, looking at the first week of at-bats would be a small universe. Looking at one in five at-bats (or maybe the more common "throw out this game and that game and he looks pretty good") would be a small sample size.

What say you guys?

Usually, when we're analyzing a players' stats, we're trying to guess about what he's likely to do moving forward. The way we do that is by estimating his "true skill".

Of course, we can't directly measure a guys' physical and abilities. The only way we could know for sure is if we had essentially an infinite number of plate appearances including every possible situation. But that's not exactly feasible.

So instead, we get a sample of plate appearances -- be it 5, 50, or 500. The size of sample you need to get a reliable estimate of the "true skill" depends on how much variability there is in the thing you're trying to estimate. There's no magic number; it's just that as your sample gets bigger the results in your sample become a more accurate reflection. And how quickly that happens depends on how much noise (variance) there is in that stat.

(This assumes your sample is representative of the full population -- for example, if you get 500 PA against Roy Halladay, your performance in those PA is not going to be reflective of your overall skill.)

So, yes, when we're talking about small sample, we do mean sample -- a sample from the universe of all possible matchups, situations, etc. that a better can face.

Rick, it might be a good time for your annual posting of the sample size list.

BCubb2003

04-15-2012, 04:34 PM

So, the weeks of the season would be the universe and the first two weeks would be the sample size? Similarly, a full season's approximate number of at-bats and and the at-bats so far?

So, the weeks of the season would be the universe and the first two weeks would be the sample size? Similarly, a full season's approximate number of at-bats and and the at-bats so far?

I think he's trying to explain the concept of population (I.e. the complete data set of all possible observations- the "universe" in his terms) versus the sample meant to estimate the population parameters.

I guess the definition of population depends upon the goal of the analysis.

Sample size is just the size or magnitude of the sample with a sample generally considered a better estimate of the population the larger the sample becomes (I.e. the closer it's size gets to the population).

Typically when the goal is to estimate a player's true skill, a season may actually be the sample (so sample size might be 600 PAs) and the population would be a cohort of all similar players.

So when an announcer evokes that batter A is 1 or 7 against pitcher B, it's a population in the sense it's the complete data set but it's worthless because as n estimate of true talent it's a hopelessly small sample that is completely beholden to randomness.

That said we do know certain peripherals of a hitter or pitcher stabilizes after specific numbers of events so as a thumb in the air, we can assume that the parameter is a good estimate of true skill. For instance, a player's contact rate might stabilize after 100 PAs (I don't honestly remember if that's true but I'm trying to illustrate the point), so after 100 PAs our sample could be thought to estimate the population with little error.

I don't know if this helps you or just adds to any confusion....

BCubb2003

04-15-2012, 10:38 PM

I think the engineer's point was that when you sample, you poll one out of 1,000 people, or you inspect one out of 100 parts, but do you look at one of every 10 at-bats? No. You look at all of the at-bats, but "all of the at-bats" happens to be small right now.

I think the engineer's point was that when you sample, you poll one out of 1,000 people, or you inspect one out of 100 parts, but do you look at one of every 10 at-bats? No. You look at all of the at-bats, but "all of the at-bats" happens to be small right now.

Well the second law of thermodynamics is often interpretted as meaning the universe tends toward maximum disorder and a sample of ten at bats is essentially hopelessly beholden to randomness, so ya, maybe a sample of ten at bats is a universe.

My sense is that the engineering definition doesn't align well with a sabermetric one... Also your engineering buddy would probably punch me in the nose for conflating entropy and disorder....

Blimpie

04-17-2012, 04:06 PM

Let's just simplify this discussion.

If you don't believe in the concept of sampling, then the next time you visit your Doctor--ask them to take all of your blood.

:cool:

Let's just simplify this discussion.

If you don't believe in the concept of sampling, then the next time you visit your Doctor--ask them to take all of your blood.

:cool:

Sampling is perfectly appropriate so long as the sample taken is sufficient to estimate the target end point.

BCubb2003

04-17-2012, 10:10 PM

My question was really only about the terminology: Are we looking at every at-bat, say, even if it's small, or are we taking one out of every 10 or so, which would be a true sample. It seems like when we talk about a "small sample size," we aren't really talking about a sampling in the technical sense.

My question was really only about the terminology: Are we looking at every at-bat, say, even if it's small, or are we taking one out of every 10 or so, which would be a true sample. It seems like when we talk about a "small sample size," we aren't really talking about a sampling in the technical sense.

Generally an estimate of true skill is being inferred in those types of conversations so say if it's 30 at bats, it's a sample....

An example would be the "Gomes has changed his approach" argument when he started the season walking at an obscene rate. He hadn't become a Prince Charming.

But this isn't like an assembly line where you're grabbing every 100th widget to make sure it's within acceptable tolerances.

Reds/Flyers Fan

04-17-2012, 11:36 PM

Small sample size or not, if any of you don't believe this club has serious deficiencies in its batting order, you're whistling while Rome burns.

This isn't a small sample size at all (11 games) ... this has been an ongoing problem since the 2010 All-Star break. We have a "clean-up hitter" who has all of eight home runs in the last two years. Even Paul Janish scoffs at those numbers.

But hey ... they'll turn it around, huh? Just one offensive, stat-padding explosion and all is OK, right?

Powered by vBulletin® Version 4.2.3 Copyright © 2015 vBulletin Solutions, Inc. All rights reserved.