Skip to content

Confirmation bias and inconvenient facts

November 27, 2011

Libertarian Daniel Klein (George Mason) received a great deal of notice for his startling Wall Street Journal Op-Ed and study claiming that knowledge of basic economic facts was (a) not correlated to attending college; and (b) was correlated with political ideology (with conservatives doing better than liberals).

Sharp criticism over the nature of his study caused him to do a second study, which revealed rather different results.  I’ll let Klein tell his story from his latest mea culpa piece in The Atlantic Magazine:

Back in June 2010, I published a Wall Street Journal op-ed arguing that the American left was unenlightened, by and large, as to economic matters. Responding to a set of survey questions that tested people’s real-world understanding of basic economic principles, self-identified progressives and liberals did much worse than conservatives and libertarians, I reported. To sharpen the ax, The Journal titled the piece “Are You Smarter Than a Fifth Grader?”—the implication being that people on the left were not.

The op-ed set off fireworks. On The Journal’s Web site, the piece peaked at No.2 in most-e-mailed for the month it was published. The Examiner, in Washington, D.C., ran two opinion pieces in response, one approving and one critical. (The latter noted, correctly, that conservatives were “happily disseminating the results across the right-wing blogosphere.”) The Washington Times reported, “Liberals Livid Over Economic Enlightenment Gauge.” My inbox exploded with messages haranguing me for cynically rigging my results or blessing me for providing proof of a long-suspected truth.

The Wall Street Journal piece was based on an article that Zeljka Buturovic and I had published in Econ Journal Watch, a journal that I edit. In short order, more than 10,000 people downloaded a PDF of the scholarly article. The attention, while slightly unnerving, was also pleasing, and I’ll confess that I found the study results congenial: I’m a libertarian, and I found it easy to believe that people on the left had an especially bad grasp of economics.

But one year later, in May 2011, Buturovic and I published a new scholarly article reporting on a new survey. It turned out that I needed to retract the conclusions I’d trumpeted in The Wall Street Journal. The new results invalidated our original result: under the right circumstances, conservatives and libertarians were as likely as anyone on the left to give wrong answers to economic questions. The proper inference from our work is not that one group is more enlightened, or less. It’s that “myside bias”—the tendency to judge a statement according to how conveniently it fits with one’s settled position—is pervasive among all of America’s political groups. The bias is seen in the data, and in my actions….

Shouldn’t a college professor have known better? Perhaps. But adjusting for bias and groupthink is not so easy, as indicated by one of the major conclusions developed by Buturovic and sustained in our joint papers. Education had very little impact on responses, we found; survey respondents who’d gone to college did only slightly less badly than those who hadn’t. Among members of less-educated groups, brighter people tend to respond more frequently to online surveys, so it’s likely that our sample of non-college-educated respondents is more enlightened than the larger group they represent. Still, the fact that a college education showed almost no effect—at least for those inclined to take such a survey—strongly suggests that the classroom is no great corrective for myside bias. At least when it comes to public-policy issues, the corrective value of professional academic experience might be doubted as well.

Discourse affords some opportunity to challenge the judgments of others and to revise our own. Yet inevitably, somewhere in the process, we place what faith we have.

Ouch.  “The corrective value of professional academic experience might be doubted.”

It is worth reading the full Atlantic piece to see the details of how Klein went down the academic primrose path of confirmation bias (in Klein’s case, that libertarians knew more about economics than liberals).  (However, Klein hardly went far enough.  A more careful analysis of his questions would have revealed that a large number of them had at least some ambiguity, and that his sampling technique was lacking, raising fundamental doubts about his survey and his method of “discovering truth.”)

And Klein did a half-self-serving and half-apology interview with WNYC’s On the Media (MP3 download link).

2 Comments leave one →
  1. November 28, 2011 11:46 am

    A more careful analysis of his questions would have revealed that a large number of them had at least some ambiguity, and that his sampling technique was lacking, raising fundamental doubts about his survey and his method of “discovering truth.”

    This can be illustrated with just the one new survey question that Klein does share with Atlantic readers. The question is loaded (and yes is ambiguous):

    “A dollar means more to a poor person than it does to a rich person”

    You may find this in an endnote in Tal Scriven’s Wrongness, Wisdom, and Wilderness: Toward a Libertarian Theory of Ethics and the Environment, where (on page 194) the author writes:

    Simple though it may be, the empirically verifiable claim that a dollar means more to a poor person than it does to a rich person (i.e., the poor person is willing to perform more labor for it than a rich person is) implies a resolution of a common complaint against utiliarianism (especially welfare utiliarianism). The complaint centers around the possiblity of what Nozick calls “utility monsters” (see his Anarchy, State and Utopia, p. 41). Nozick’s objection is based on the hypothesis of the existence of a being who is capable of absorbing so much pleasure that the suffering of a large number of other beings is of little hedonic importance. In other words, this being is so insatiable that, no matter how much of any kind of wlefare others may recive from a distribution of a bundle of goods to them, the utility monster receives more. The resolution of this problem I have in mind follows from the simple denial of the existence of such a being; sentient beings on this planet just don’t operate the way Nozick is supposing them to, and utilitarianism was never proposed to do more than work for all sentient beings on this planet.

    Notice the leaps of faith Scriven must make, at this one point, as he tries to lead his own readers toward a libertarian theory. The “claim” is “empirically verifiable.” There’s an implication of a resolution of a complaint that is “common.” There’s a special case of this common complaint, the “objection” by another libertarian Robert Nozick. There’s the ability to see Nozick’s complaint as wrong when it is simply denied.

    Or notice what Klein does to his larger audience when picking on his fellow libertarians. He simply has to exclaim, “c’mon, people!” to object that “30 percent” of them are wrong about this question that Scrivens and Nozick seem not to agree on.

  2. November 28, 2011 1:18 pm

    Robert Nozick (whom I knew) still continues to be a significant thorn in the side of many moral philosophers and libertarians alike. The libertarians who admire him tend to read him selectively, and then retreat back to an easy ideology (such as Ayn Rand.)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: