Peer review, science, and politics

The problem is that as human beings our natural tendency is to look for evidence that confirms what we already think and to ignore evidence against it. Conservatives listen to Fox News. Liberals listen to NPR. Both of us (I listen to NPR) think we're getting objective, unbiased news and that the other guys are getting fed propaganda by right-wing nutcases or left-wing lunatics.

Mr. Manjoo cites a more recent study by Stanford University psychologists of students who either favored or opposed capital punishment. The students were shown the same two studies: one suggested that executions have a deterrent effect that reduces subsequent murders, and the other doubted that.

Whatever their stance, the students found the study that supported their position to be well-conducted and persuasive and the other one to be profoundly flawed.

“That led to a funny result,” Mr. Manjoo writes. “People in the study became polarized.”

Sound familiar?

Scientists are people, too. We have the same tendency, but we also have two processes that check it: the scientific method and peer review.

Philosophers devote entire careers to refining the details of how science is or ought to be practiced, but everyone agrees that observation and experiment are essential elements of scientific investigation. When we set up an experiment or design a set of observations, we usually have a pretty good idea of what we're going to find. But we don't know for sure. That's why we do the study. There's always the possibility that our expectation is wrong and that we'll get results we don't expect. If that happens, then we have to revise our hypothesis in light of the new data we've gathered.

But that's just the first step – convincing ourselves that we know what's going on. The next step is even more important – convincing our colleagues that we know what's going on. That's where peer review comes in.

As I constantly remind my students, our work isn't done until it's published and available, with the emphasis on published because “published” is shorthand for “published in a peer-reviewed journal.” No matter how careful I've been in designing my experiments or observations, it's always possible that I've overlooked some key feature of the system that causes my data to mean something other than what they really mean. I might not have controlled a key variable. I might have used an inappropriate statistical analysis. Or I might have made an assumption that seems reasonable to me that isn't reasonable at all.

So before my results appear in a scientific journal, my report will be sent to two or three experts in the field who will read it carefully and look for those kinds of deficiencies. Since they bring somewhat different experiences and biases to the problem than I do, they are virtually certain to find some parts of my report with which they disagree. They'll describe those differences to the editor, and if they're serious enough the editor will tell me “Sorry. This just isn't up to snuff. You need to do some more work here to correct some serious deficiencies.” I've been fortunate in that the more frequent response to my reports is something like “Look, Holsinger. You've got some decent stuff here, but there are a few significant points that you'll need to correct or rebut before this thing sees the light of day.” Usually that means that I've pushed my results too far and that I have to tone down or remove completely one of the conclusions I thought I could make. Sometimes it means I need to collect some new data to convince reviewers that what I think is going on really is going on. If so, it won't be simply collecting more data of the same kind. It will be conducting new experiments that address the specific questions the reviewer raised.

The scientific method and peer review don't guarantee that published results are right. But because peer reviewers are looking for flaws and because they are often selected because they're known to have a different view of the problem than the author, it does mean that peer reviewed science is free of obvious flaws, or at least free of flaws that weren't obvious to the authors of the paper and to three or four experts in the field.

Assertions are always challenged in peer-reviewed science. Authors always ask themselves, “Is there enough evidence of what I want to claim here that a reviewer will agree?” As a result, we're forced to look for evidence that's contrary to our initial expectations. Being human, we'll sometimes fail. But when we do, peer review will (usually) catch and correct our error.