Sample Bias
Also known as: Survivor Bias
A group of people asked about something might not be a valid, random sample. Asking people in front of the library how many books they read last year is a good way to get an invalid answer. Related to this, people who offer to participate in surveys (who “self-select”) are often biased – they want to respond to a surveyed or express an opinion precisely because they have an opinion to share.
Examples
What Sharks Can Teach Us About Survivorship Bias
Great White sharks are thought to be the deadliest in the world. But what if they’re just less good at killing people than other sharks, so they leave more survivors to identify them?
Several studies have indicated that people get happier as they enter old age (their 70s and 80s). However, this could be simply because unhappier people die at earlier ages so they’re not around to take a survey and drag the results down. Ironically, this makes the original premise true: people do get happier as they get older, but only because unhappier people tend to not get older.
During World War II, the Statistical Research Group was tasked with deciding what parts of a fighter plane to re-enforce with steel plating. While initially tempted to apply armor to the areas where they found the most bullet holes, Hungarian mathematician Abraham Wald determined that what they should actually be looking for were areas on the plane that had no bullet holes, since it was those areas (i.e. – the engine) where a single penetrating bullet downed the plane and prevented it from returning and being examined. The existence of bullet holes in a plane that survived actually proved the plane could absorb bullets in that location and survive.
People who write reviews for websites are almost always on the extremes, and often negative, because people in the middle couldn’t be bothered so the weakness of their opinion didn’t overcome the level of effort required to write a review.
We don’t know who the greatest art forger in the world is because they’re so good, they’ve never been detected. If someone was caught, then they are – by definition – not the greatest forger. The pool of forgers that we have identified is biased towards forgers who have been caught.
Someone tweeted once:
Parachutes sold on Amazon only have 5-star reviews.
Consider: why wouldn’t there be a 1-star review for a parachute?