Sampling

Sampling is like taking a few sips to judge a bottle of wine. In math, it means picking a small, manageable bunch from a huge group to learn something about the whole. It’s a shortcut that when done correctly gets you pretty close to the full picture and when done poorly leads you to the wrong conclusions. Understanding the influence and importance of sampling is a fundamental key to understanding the world better.

The mental model of sampling is a first principle for a number of concepts in mathematics, especially in statistics. Seeing as we often use statistics to gain a picture of reality, taking sampling into account will broaden your knowledge of other areas involving measurement, such as psychology. It will also help you think about risk and reward while disentangling luck from skill.

Your personal experiences with money make up maybe 0.00000001 % of what’s happened in the world, but maybe 80 % of how you think the world works.

—Morgan Housel

When we want to get information about a population (meaning a set of alike people, things, or events), we typically need to look at a sample (meaning a part of the population). It is usually not possible or even desirable to consider the entire population, so we aim for a sample that represents the whole. We use samples to tell us about the world. The exception is a census, which aims to include everyone, not just a sample.

Sample size refers to the number of people, things, or events we look at from a population. It can have an enormous influence on the results we get. According to the law of large numbers, the larger the sample size, the more the result obtained will converge with the true value. For instance, the likelihood of getting any of the numbers from one to six when you roll a fair die is 1 in 6. If you roll a die 6 times, it would be unusual for each number to show up once. If you roll it 600 times, the frequency of each number would be closer to 1 in 6, and even closer if you roll it 6,000 times or 60,000 times.

Sampling gives you an idea of the possible values in your data set. The law of large numbers helps identify the probability of any of those values occurring.

As a rule of thumb, more measurements mean more accurate results, all else being equal. Small sample sizes can produce skewed results. If your sample for assessing the color of swans is the white birds in your neighborhood pond, you might deduce all swans are white. But if you looked at a larger sample of swans from different places, you’d discover some are black. Even if you took a small sample from your local pond dozens of times, it might not be representative and would always mislead you. Small sample sizes can fail to include rare or outlier results, making it seem like these don’t exist. If you are once again just looking at the birds in a particular pond, you might ascertain that all swans have two legs. But if you took a much larger sample (or visited a wildlife sanctuary), you could well discover a swan that has had an accident and has only one leg.

One area where sampling is especially salient is in scientific studies of people, in which case the sample size is the number of participants. The larger the sample, the lower the margin of error and the higher the sampling confidence level, meaning the more probable it is the results will generalize to the whole population.

But we have to make trade-offs. Studies with small sample sizes are not useless. Managing the funding and logistics for a larger study might be impractical, so a small one can provide evidence that further research is worthwhile. Researchers can combine a number of small studies in a meta-analysis to get a broader overview. If the costs of a study are high, such as a psychological study in which participants are subjected to a great deal of distress, a small sample may be more ethical. Researchers have to consider a range of factors to establish the right sample size, including the expected effect size and the expected dropout ratio.

In addition to being an appropriate size, samples need to be random to be representative of a varied population. This means every person, thing, or event within the population has an equal chance of ending up in the sample. George Gallup, who could be called the inventor of the opinion poll, once gave a useful analogy for the necessity of a random sample. He said you can decide if a pot of soup needs more seasoning by tasting a single spoonful—provided you’ve stirred it well first.

A large sample size is certain to be more accurate only if it is representative, which means it’s important to be aware of sampling biases. For instance, the healthy worker effect refers to the fact that people who are unwell are less likely to be employed, meaning that workers within a field or for a company will be, on average, healthier than the general population. If you’re trying to measure the impact of a dangerous chemical on factory workers who handle it, a sample of those currently employed in that type of factory will not be representative—no matter how large. You’ll need to factor in those whose health has been damaged so much they no longer work or have moved to another field.

Thinking about sampling can help us overcome some forms of bias. For example, we tend to place too much emphasis on anecdotes, in particular those coming from people close to us or that make good stories. We’re all the more inclined to respect anecdotes if they confirm what we already believe. An anecdote is a sample size of one, and we should collect more data points if possible. The exception is when one result indicates something is possible. The first person to survive a heart transplant was more than an anecdote.

Thinking about your samples also teaches that sometimes what you need to do to see the world in a more accurate way is to obtain a larger or more representative sample. Traveling, living in a big city, or otherwise finding a way to meet more diverse people may make you more tolerant. Exposing yourself to a broader range of ideas through interdisciplinary, far-reaching reading may make you more open-minded. Learning more about the history of your industry and accumulating more experience may make you more risk-aware as you discover rarer and more extreme possibilities.