Non-Generalizable Objectivity
Banasiewicz uses non-generalizable objectivity to describe treating anecdotes or unrepresentative samples as a basis for broad conclusions. A large collection of reviews can still mislead if the reviewers’ circumstances differ from those of the people the conclusion concerns.
THE IDEA
Non-Generalizable Objectivity
Anecdotal evidence can be irresistibly compelling. After all, there are entire product categories, even industries that owe their existence to consumers relying on anecdotes rather than representative evidence. Antiaging or skin care products are commonly bought on recommendations of friends, diet plans and miracle pills that promise to deliver a lot (of weight loss) and fast are typically sold by passionate spokespersons – in fact, one of the pillars of advertising, a compensated endorser, is rooted in the persuasive power of anecdotal evidence. Consumers tend to conclude that a product or a brand is 'the best' or 'the worst' not based on any kind of scientific evidence, but on word-of-mouth, television, and other similar forms of influence.
A more recent phenomenon of online product reviews seemingly fixed the problem associated with relying on a single, or a very small number of influencers by bringing to bear often large pools of ad hoc reviewers (which created a cottage industry of professional product reviewers). The idea of 'verified' product purchasers offering their independent evaluations is now one of the mainstays of online commerce, and countless consumers base their brand choices on the resultant buyer satisfaction comparisons. After all, how can one not trust hundreds of independent impressions?
And while relying on a compensated opinion of a star professional athlete when buying, let's say, a car or car insurance is very visibly questionable, relying on (sometimes compensated and sometimes not) opinions of hundreds of anonymous buyers may seem reasonable, or at least until the representativeness of a particular opinion pool is called into question. Would brand reviews sourced mostly from younger, let's say under 30 years of age, drivers offer meaningful decision-related insights to a retired baby boomer looking for a new automobile? They may answer some questions, but the usefulness of the reviews depends on the buyers’ needs and circumstances, not just the number of reviews. Although the difference may seem obvious to a statistician, many decision-makers tend not to be adequately cognizant of the difference between the number of product reviewers and the degree to which opinions of those reviewers may or may not be representative of a larger universe of all product users.
Sample size and representativeness are different questions. Before generalizing, ask who contributed the data, who is missing and whether the sample fits the population or decision at hand.
When to use it
Deciding on a medical treatment based on a friend's success story rather than clinical trial results.
Choosing a university or college based on a few highly successful alumni stories.
Investing in a startup based on a charismatic founder's pitch and a few anecdotal successes.
Relying on aggregated product ratings without examining the demographic profile of the reviewers.
FROM THEORY TO EVERYDAY LIFE
In practice
The Miracle Diet Pill
A friend loses 30 pounds in a month using a miracle diet pill. Based solely on this persuasive anecdote, you purchase the expensive pills, ignoring the lack of scientific evidence and the possibility that your friend's outcome was due to other factors or luck.
Car Reviews by Age Group
In a hypothetical car-shopping example, a retired couple discovers that 90% of the reviews come from drivers under 30. They examine the reviews and find that those reviewers emphasize sporty handling and infotainment, while they care more about ride comfort and ease of entry. The mismatch in priorities limits the usefulness of the overall rating; age alone would not establish that mismatch.
A MOMENT TO REFLECT
Put it to work.
When was the last time you made a significant decision based primarily on a single anecdote or a non-representative set of reviews?
How can you determine whether the reviews or testimonials you are reading are truly representative of the general user population you belong to?
What are the potential adverse consequences of generalizing from a few vivid stories instead of consulting statistical data?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)