Confirmation bias and inconvenient facts

Libertarian Daniel Klein (George Mason) received a great deal of notice for his startling Wall Street Journal Op-Ed and study claiming that knowledge of basic economic facts was (a) not correlated to attending college; and (b) was correlated with political ideology (with conservatives doing better than liberals).

Sharp criticism over the nature of his study caused him to do a second study, which revealed rather different results.  I’ll let Klein tell his story from his latest mea culpa piece in The Atlantic Magazine:

Back in June 2010, I published a Wall Street Journal op-ed arguing that the American left was unenlightened, by and large, as to economic matters. Responding to a set of survey questions that tested people’s real-world understanding of basic economic principles, self-identified progressives and liberals did much worse than conservatives and libertarians, I reported. To sharpen the ax, The Journal titled the piece “Are You Smarter Than a Fifth Grader?”—the implication being that people on the left were not.

The op-ed set off fireworks. On The Journal’s Web site, the piece peaked at No.2 in most-e-mailed for the month it was published. The Examiner, in Washington, D.C., ran two opinion pieces in response, one approving and one critical. (The latter noted, correctly, that conservatives were “happily disseminating the results across the right-wing blogosphere.”) The Washington Times reported, “Liberals Livid Over Economic Enlightenment Gauge.” My inbox exploded with messages haranguing me for cynically rigging my results or blessing me for providing proof of a long-suspected truth.

The Wall Street Journal piece was based on an article that Zeljka Buturovic and I had published in Econ Journal Watch, a journal that I edit. In short order, more than 10,000 people downloaded a PDF of the scholarly article. The attention, while slightly unnerving, was also pleasing, and I’ll confess that I found the study results congenial: I’m a libertarian, and I found it easy to believe that people on the left had an especially bad grasp of economics.

But one year later, in May 2011, Buturovic and I published a new scholarly article reporting on a new survey. It turned out that I needed to retract the conclusions I’d trumpeted in The Wall Street Journal. The new results invalidated our original result: under the right circumstances, conservatives and libertarians were as likely as anyone on the left to give wrong answers to economic questions. The proper inference from our work is not that one group is more enlightened, or less. It’s that “myside bias”—the tendency to judge a statement according to how conveniently it fits with one’s settled position—is pervasive among all of America’s political groups. The bias is seen in the data, and in my actions….

Shouldn’t a college professor have known better? Perhaps. But adjusting for bias and groupthink is not so easy, as indicated by one of the major conclusions developed by Buturovic and sustained in our joint papers. Education had very little impact on responses, we found; survey respondents who’d gone to college did only slightly less badly than those who hadn’t. Among members of less-educated groups, brighter people tend to respond more frequently to online surveys, so it’s likely that our sample of non-college-educated respondents is more enlightened than the larger group they represent. Still, the fact that a college education showed almost no effect—at least for those inclined to take such a survey—strongly suggests that the classroom is no great corrective for myside bias. At least when it comes to public-policy issues, the corrective value of professional academic experience might be doubted as well.

Discourse affords some opportunity to challenge the judgments of others and to revise our own. Yet inevitably, somewhere in the process, we place what faith we have.

Ouch.  “The corrective value of professional academic experience might be doubted.”

It is worth reading the full Atlantic piece to see the details of how Klein went down the academic primrose path of confirmation bias (in Klein’s case, that libertarians knew more about economics than liberals).  (However, Klein hardly went far enough.  A more careful analysis of his questions would have revealed that a large number of them had at least some ambiguity, and that his sampling technique was lacking, raising fundamental doubts about his survey and his method of “discovering truth.”)

And Klein did a half-self-serving and half-apology interview with WNYC’s On the Media (MP3 download link).