Uncommon Ground

Why I don’t like null hypothesis significance testing – Part 1

A couple of weeks ago, I argued against falsificationism. Empirical evidence is the foundation of scientific knowledge, but falsifying hypotheses is a less important part of building that knowledge than people often think. And even within the realm of falsifications, null hypothesis significance testing is less useful than many people think for several reasons. In this post I’ll focus on problems that arise for decision making posed by the difference between planned and post-hoc comparisons. I’ll use some real, but simplified, data on soybean yields to illustrate.

The first set of data I’ll discuss come from a field trial at a university research farm near Brownston, Illinois in 2015 that gave the following results:1

Treatment Yield (bushels per acre)
Control 61.1
Treatment (100 lb N/acre) 56.2

That’s a difference of 4.9 bushels per acre, with less yield when nitrogen is applied,2 and the difference is significant: P = 0.022. The price of soybean futures on the Chicago Board of Trade as I write this3 is $12.30 per bushel. In 2022, US farms harvested a little more than 300 acres of soybeans on average. So for an average soybean farmer, applying 100 pounds of nitrogen per acre would cost them a little over $18,000 (not counting the cost of the fertilizer).

So far so good, but the results I just reported are from one of 5 hypothesis tests the authors performed, and the particular test reported was not a planned comparison. Using Tukey’s HSD test for post-hoc comparisons, which is what you should do in this case, gives P > 0.1, which is far from significant, meaning that in a conventional statistical sense we have no evidence that applying nitrogen fertilizer has an effect. It still doesn’t make sense to apply nitrogen fertilizer, since it costs money to buy and apply it, but with this analysis the farmer wouldn’t expect to lose additional money because of reduced yields.

But of course the agronomists at the University of Illinois didn’t do the experiment only near Brownston, they also did it at several other experimental farms. Let’s look at the results from Urbana in 2015:

Treatment Yield (bushels per acre)
Control 91.0
Treatment (100 lb N/acre) 96.1

In this case, there’s a 5.1 bushel per acre increase in yield per acres when nitrogen is applied and the difference is significant: P = 0.028. But again, when we use Tukey’s HSD, we get P > 0.1, meaning that we don’t have statistical evidence of an effect, even though our best guess is that the farmer would make nearly $19,000 (minus the cost of fertilizer) if they did.

What are you going to tell a farmer to do? How much of what you tell them will depend on results of the null hypothesis tests I presented? Feel free to look at the additional results reported in the study from which these data were drawn. I’ll give you my answer in the next post.

  1. https://ilagronomy.info/2025/07/14/nitrogen-fertilizer-and-soybean-yield-what-we-learned-from-multi-year-trials-in-illinois/ The treatment mean is the PLT+R1+R3+R5 mean. ↩
  2. That might not be too surprising, since soybean fixes nitrogen. ↩
  3. About 9:00am EST on 16 November 2026 ↩

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.