{"id":106,"date":"2016-09-19T08:30:22","date_gmt":"2016-09-19T12:30:22","guid":{"rendered":"http:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/?p=106"},"modified":"2016-09-18T08:03:21","modified_gmt":"2016-09-18T12:03:21","slug":"even-an-informative-prior-doesnt-help-much","status":"publish","type":"post","link":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/blog\/2016\/09\/19\/even-an-informative-prior-doesnt-help-much\/","title":{"rendered":"Even an informative prior doesn’t help much"},"content":{"rendered":"

Two weeks ago<\/a> I pointed out that you should<\/p>\n

Be wary of results from studies with small sample sizes, even if the effects are statistically significant.<\/p><\/blockquote>\n

Last week I pointed out that being Bayesian won’t save you<\/a>. If you were paying close attention, you may have thought to yourself<\/p>\n

Holsinger’s characterization of Bayesian inference isn’t completely fair. The mean effect sizes he simulated were only 0.05, 0.10, and 0.20, but he used a prior with a standard deviation of 1.0 in his analyses. Any Bayesian in her right mind wouldn’t use a prior that broad, because she’d have a clue going into the experiment that the effect size was relatively small. She’d pick a prior that more accurately reflects prior knowledge of the likely results.<\/p><\/blockquote>\n

It’s a fair criticism, so to see how much difference more informative priors make, I re-did the simulations with a Gaussian prior on each mean with a prior mean of 0.0 (as before) and a standard deviation of 2 times the effect size used in the simulation. Here are the results:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
Mean<\/th>\nSample size<\/th>\nPower<\/th>\nWrong sign<\/th>\n<\/tr>\n
0.05<\/td>\n10<\/td>\n0\/1000<\/td>\nna<\/td>\n<\/tr>\n
<\/td>\n50<\/td>\n2\/1000<\/td>\n0\/2<\/td>\n<\/tr>\n
<\/td>\n100<\/td>\n7\/1000<\/td>\n2\/7<\/td>\n<\/tr>\n
0.10<\/td>\n10<\/td>\n1\/1000<\/td>\n0\/1<\/td>\n<\/tr>\n
<\/td>\n50<\/td>\n0\/1000<\/td>\nna<\/td>\n<\/tr>\n
<\/td>\n100<\/td>\n0\/1000<\/td>\nna<\/td>\n<\/tr>\n
0.20<\/td>\n10<\/td>\n22\/1000<\/td>\n2\/22<\/td>\n<\/tr>\n
<\/td>\n50<\/td>\n128\/1000<\/td>\n0\/158<\/td>\n<\/tr>\n
<\/td>\n100<\/td>\n265\/1000<\/td>\n0\/292<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

With a more informative prior, you’re not likely to say that an effect is positive when it’s actually negative. There are, however, a couple of things worth noticing when you compare this table to the last one<\/a>.<\/p>\n

    \n
  1. The more informative prior doesn’t help much, if at all, with a sample size of 10. The N(0,1) prior got the sign wrong in 7 out of 62 cases where the 95% credible interval on the posterior mean difference did not include 0. The N(0,0.4) prior made the same mistake in 2 out of 22 cases. So it didn’t make as many<\/em><\/strong> mistakes as the less informative prior, but it made almost the same proportion<\/em><\/strong>. In other words, you’d be almost as likely to make a sign error with the more informative prior as you are with the less informative prior.<\/li>\n
  2. Even with a sample size of 100, you wouldn’t be “confident” that there is a difference very often (only 7 times out of 1000) when the “true” difference is small, 0.05, but you’d make a sign error nearly a third of the time (2 out of 7 cases) .<\/li>\n<\/ol>\n

    So what does all of this mean? When designing and interpreting an experiment you need to have some idea of how big the between-group differences you might reasonably expect to see are relative to the within-group variation. If the between-group differences are “small”, you’re going to need a “large” sample size to be confident about your inferences. If you haven’t collected your data yet, the message is to plan for “large” samples within each group. If you have collected your data and your sample size is small, be very careful about interpreting the sign of any observed differences – even if they are “statistically significant.”<\/p>\n

    What’s a “small” difference, and what’s a “large” sample? You can play with the R\/Stan code in Github to explore the effects: https:\/\/github.com\/kholsinger\/noisy-data<\/a>. You can also read Gelman and Carlin (Perspectives on Psychological Science 9:641; 2014 http:\/\/dx.doi.org\/10.1177\/1745691614551642<\/a>) for more rigorous advice.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Two weeks ago I pointed out that you should Be wary of results from studies with small sample sizes, even if the effects are statistically significant. Last week I pointed… Read more ><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-106","post","type-post","status-publish","format-standard","hentry","category-statistics"],"_links":{"self":[{"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/posts\/106","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/comments?post=106"}],"version-history":[{"count":0,"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/posts\/106\/revisions"}],"wp:attachment":[{"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/media?parent=106"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/categories?post=106"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/darwin.eeb.uconn.edu\/uncommon-ground\/wp-json\/wp\/v2\/tags?post=106"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}