Generalization: Why You Make Dumb Decisions

Generalizability is the degree to which your study results can be applied to a broader context. Research results are generalizable when they apply to most contexts, most people, most of the time. Suppose you want to study the fitness activity level of people in your city. You randomly ask passersby in a popular area whether they would answer a few questions for your survey. After you gather what you think is enough data, you synthesize it, and present your results. Would that be an accurate representation of the fitness of your region?

You know the answer. Not only would you miss a huge proportion of the population, but surveys tend to be lied to. Especially when they are open-ended, fitness-related questions. Who would publicly state themselves physically inactive? In other words, this makes your little investigation non-generalizable. Your sample is not representative of your target population and the variable you are studying. The goal of research is to produce knowledge that can be applied widely. But since it is impossible for you or any other researcher to analyze an entire city, studies are done analyzing portions of it, and making statements about a given portion.

This is often how people run into incorrect conclusions when they read scientific papers through their uneducated lenses. They apply niche statements to large groups which don’t represent the studied sample. Three things dictate how generalizable a study is: the randomness of the sample, how representative the sample is of the studied population, and the size of the sample. The more random, representative, and large the sample is, the more statistically significant the results. It’s crucial to establish just how valid and reliable any study is.

People who lack the necessary skill to examine scientific papers shouldn’t form robust conclusions after reading them. Take the sampling error. With a small sample, the random variation is high. Thus the observed characteristics can deviate significantly from the true characteristics of the population. If you interview 10 people in a town and find that 90 percent like a local pizzeria, that does not mean 90 percent of the town prefers that restaurant. Small samples result in overconfidence in the patterns observed, since limited data often exhibits clear and attractive trends. A startup might conduct a pilot test with 20 users. They all like it. Does this mean the product is universally loved? No. Would the flattered founder think so? Of course.

Real-life implications of generalizability:

• Consumerism: a company surveys 15 customers about a new product. 12 out of 15 say they love it. Management hastily concludes that the product is a hit. That small survey fails to capture the diverse opinions in a large market. The overinvestment in the product could be a disaster.

• Political Polling: a local news outlet polls 30 voters in a district. They find overwhelming support for a candidate. With their tiny sample, the results can of course be skewed by a sampling error. What do you see in local news articles however?

• Health and Medicine: a preliminary clinical trial with a small number of patients shows promising results for a new drug. Early results don’t hold up in larger, more rigorous studies. But by then, its too late. The generalization from a small sample resulted in many people taking the drug and experiencing its undocumented side effects.

• Workplace Feedback: a manager receives positive feedback from a handful of team members on a new policy. They absolutely loved it. Since the sample is both small and non-representative (only the most engaged employees who are under this manager can give them feedback), the manager will overestimate how effective the initiative was.

How you might use generalizability as a mental model: (1) numbers matter — asses the sample size before you draw conclusions, considering whether your data is statistically significant and reliable; (2) change your sources — combine multiple streams of data to avoid reliance on possibly biased samples, supplementing surveys with focus groups and conversational studies; (3) beware of anecdote — recognize that compelling stories don’t always represent the general trend, and arguments from personal experience should never take stead over objective information; (4) be cautious of your findings — when presenting something with a small sample, clearly communicate the limitation, such as “based on a limited number of people”.