Kristof on pundits

I've been writing a lot about about newspapers lately. It's not because I'm particularly knowledgeable about them. It's because I'm concerned. This post is a bit different.

I pointed out a few days ago that the George Will episode provides evidence that we need news organizations and editorial pages that check their facts. Today Nick Kristof has an interesting column on experts. Here's a paragraph I found particularly striking:

The marketplace of ideas for now doesn't clear out bad pundits and bad ideas partly because there's no accountability. We trumpet our successes and ignore failures -- or else attempt to explain that the failure doesn't count because the situation changed or that we were basically right but the timing was off.

Kristof's point is one that philosophers of science, starting with Karl Popper, have made for a long time. It's necessary to look for data that would contradict your hypothesis, not data that support it. Being accountable is good. Looking for evidence that shows you're wrong is even better. That's why peer review is so important.

It's really hard to evidence that you're wrong. It's much easier to look for evidence that someone else is wrong or pushing the conclusions beyond what the data justify.

The scientific method and peer review don't guarantee that published results are right. But because peer reviewers are looking for flaws and because they are often selected because they're known to have a different view of the problem than the author, it does mean that peer reviewed science is free of obvious flaws, or at least free of flaws that weren't obvious to the authors of the paper and to three or four experts in the field.