Uncommon Ground

Author Archive: kent

Designing exploratory studies: measurement

On Wednesday I argued that we need carefully done exploratory studies to discover phenomena as much as we need carefully done experimental studies to test explanations for the phenomena that have been discovered.1 Andrew Gelman suggests four preliminary principles:

  1. Validity and reliability of measurements.
  2. Measuring lots of different things.
  3. Connections between quantitative and qualitative data.2
  4. Collect or construct continuous measurements where possible.

Today I’m going to focus on #1, the validity and reliability of measurements.

If there happen to be any social scientists reading this, it’s likely to come as a shock to you to learn that most ecologists and evolutionary biologists haven’t thought too carefully about the problem of measurement, or at least that’s been my experience. My ecologist and evolutionary biologist friends are probably scratching their heads. “What the heck does Holsinger mean by ‘the problem of measurement.'” I’m sure I’m going to butcher this, because what little I think I know I picked up informally second hand, but here’s how I understand it.

(more…)

Designing exploratory studies: preliminary thoughts

Andrew Gelman has a long, interesting, and important post about designing exploratory studies. It was inspired by the following comment from Ed Hagen following a blog post about a paper in Psychological Science.

Exploratory studies need to become a “thing”. Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

I think Ed’s suggestion is too narrow. Exploratory studies are essential to good science, not just good social science. We often (or at least I often) have only a vague idea about how features I’m interested in relate to one another. Take leaf mass per area (LMA)

We often (or at least I often) have only a vague idea about how features I’m interested in relate to one another. Take leaf mass per area (LMA)1 and mean annual temperature or mean annual precipitation, for example. In a worldwide dataset compiled by IJ Wright and colleagues2, tougher leaves (higher values of LMA) are associated with warmer temperatures and less rainfall.

LMA vs. mean annual temperature and log(mean annual rainfall)

LMA, mean annual temperature, and log(mean annual rainfall) in 2370 species at 163 sites (from Wright et al. 2004)

We expected similar relationships in our analysis of Protea and Pelargonium,3 but we weren’t trying to test those expectations. We were trying to determine what those relationships were. We were, in other words, exploring our data, and guess what we found. Tougher leaves are associated with less rainfall in both general and with warmer temperatures in Protea. They were, however, associated with cooler temperatures in Pelargonium, exactly the opposite of what we expected. One reason for the difference might be that Pelargonium leaves are drought deciduous, so they avoid the summer drought characteristic of the regions from which our samples were collected. That is, of course, a post hoc explanation and has to be interpreted cautiously as a hypothesis to be tested, not as an established causal explanation. But that is precisely the point. We needed to investigate the phenomena to identify a pattern. Only then could we generate a hypothesis worth testing.

I find that I am usually more interested in discovering what the phenomena are than in tying down the mechanistic explanations for them. The problem, as Ed Hagen suggests, is that studies that explicitly label themselves as exploratory play little role in science. They tend to be seen as “fishing expeditions,” not serious science. The key, as Hagen suggests, is that to be useful, exploratory studies have to be done as carefully as explicit, hypothesis-testing confirmatory studies. A corollary he didn’t mention is that science will be well-served if we observe the distinction between exploratory and confirmatory studies.4

(more…)

Follow up on dodging a bullet

Last week I mentioned that I had considered participating in Benefunder and that an article in Science made me glad I’d decided not to participate.

I was tempted because it sounded like a very promising idea. In the end, though, I just couldn’t see investing $500 in the project. It seemed too unlikely that a donor would be interested in supporting the esoteric research that I do.

Christian Braemer, CEO/co-founder of Benefunder, left a comment on that post. If you’re interested in Benefunder, I encourage to visit the original post and read the whole comment. Here’s the key part of what he has to say about the article in Science:

The worst part is the title, which implies we’ve folded and that couldn’t be further from the truth as you’ll see in a string of press releases that are about to come out regarding new partners, board members, and investors (although this story certainly isn’t going to help with the latter).

Not to be overly dramatic, but this story is a major threat to finding new ways to fund research in this country. Consider this, with our new approach, if we get just 4% of current DAF distributions, that equates to over $1B in new funding for research. Crowdfunding (which we are not) doesn’t have the right alignment of interests, expectations, or volume to be able to pull this off – at least not for the foreseeable future.

By the way, Christian posted his comment on 8 November, the day after my original post went live. His comment only went live this morning, because it’s the first time in a week I’ve had a chance to check in.

 

Oxford Handbook on the Science of Science Communication

I just ran across an old (13 October 2015) press release from the Annenberg Public Policy Center of the University of Pennsylvania announcing that Oxford University Press will publish the Oxford Handbook on the Science of Science Communication in 2017.1 The very brief press release on the Annenberg Center’s website makes it sound very interesting. I look forward to learning more about it as its release date approaches. In the meantime, here’s a very brief description lifted from the press release.

The handbook, developed for scientists, academics and students of science communication, will grapple with the failure of widely accessible scientific evidence to effectively inform public controversies on issues including climate change, the human papilloma virus vaccine, and the safety of bio-engineered foods on the market in the United States.


1OUP doesn’t seem to have an announcement of it on its website yet.

How to think about replication

Caroline Tucker reviewed a paper by Nathan Lemoine and colleagues in late September and reminded us that inferring anything from small, noisy samples is problematic.1 Ramin Skibba now describes a good example of the problems that can arise.

In 1988, Fritz Strack and his colleagues reported that people found cartoons funnier if they held a pen in their teeth than if they held it between their lips. Why? Because holding a pen between your teeth causes you to smile, while holding one between your lips causes you to pout. This report spawned a series of studies on the “facial feedback hypothesis”, the hypothesis that facial expressions influence our emotional states. It seems plausible enough, and I know that I’ve read advice along this line in various places even though I’d never heard of the “facial feedback hypothesis” until I read Skibba’s article.

Unfortunately, the hypothesis wasn’t supported in what sounds like a pretty definitive study: 17 experiments and 1900 experimental subjects. Sixteen of the studies had large enough samples to be confident that the failure to detect an effect wasn’t a result of small sample size. Strack disagrees. He argues that (a) using a video camera to record participants may have made them self-conscious and suppressed their responses and (b) the cartoons were too old or too unfamiliar to participants to evoke an appropriate response.

Let’s take Strack at his word. Let’s assume he’s right on both counts. How important do you think the facial feedback to emotions is if being recorded by a video camera or being shown the wrong cartoons causes it to disappear (or at least to be undetectable)? I don’t doubt that Strack detected the effect in his sample, but the attempt to replicate his results suggest that the effect is either very sensitive to context or very weak.

I haven’t gone back to Strack’s paper to check on the original sample size, but the problem here is precisely what you’d expect to encounter if the original conclusions were based on a study in which the sample size was small relative to the signal-to-noise ratio. To reliably detect a small effect that varies across contexts requires either (a) very large sample size (if you want your conclusions to apply to the entire population) or (b) very careful specification of the precise subpopulation to which your conclusions apply (and extreme caution in attempting to generalize beyond it).

(more…)

Please vote

I votedToday is election day in the United States. The polls opened in Connecticut 2 hours ago, and there was a long line at my polling place by the time I arrived at 6:15.

If you are a registered voter and you aren’t one of the 41 million who have already cast a ballot in early voting, please vote today. If you don’t know where to vote, the League of Women Voters Education Fund has compiled information on the location of polling places at Vote411.org. All you need to do is put in your home address, and you’ll get a street address, hours of operation, and a Google map showing where to go. In addition to the Presidential race, there are important state and local races around the country. Please take the time to make your voice heard. Representative government works only if we select representatives who reflect our values. Voting is how we express those values.

It appears that I dodged a bullet

A couple of years ago, I received an unsolicited invitation to participate in Benefunder, a sort of Kickstarter for scientists. I talked with the people running it a couple of times. They proposed a very intriguing idea: All I needed to do was come up with a snappy description of my research, some compelling images, and $500. They would promote my research as part of a portfolio that wealthy investors would contribute to both because they were interested in the research and because the contributions were structured in a way that provided substantial tax benefits, a donor-advised fund. I was tempted because it sounded like a very promising idea. In the end, though, I just couldn’t see investing $500 in the project. It seemed too unlikely that a donor would be interested in supporting the esoteric research that I do.

It appears that my skepticism was well founded.

[E]ven as Benefunder bulged with projects, donors remained scarce. “We were never able to get off the ground,” [Christian] Braemer [one of the Benefunder founders] says. Donor funds “were not willing to take the reputational risk [on] an unknown entity,” he says. And the firm received just a few “small transactions … a bit out of the blue.”

To stay afloat, Benefunder ramped up sales of the profiles and videos. In 2014 and 2015, it earned more than $660,000 this way but attracted just $62,000 in gifts, tax forms show. In late 2015, as the firm ran out of cash, it abruptly stopped recruiting researchers, left some videos unfinished, and laid off all but three of the 12 employees who worked for it and an allied firm. (Ambitious web fundraising startup fails to meet big goals, by Mark Harris, Science 354: 534; 2016  doi: 10.1126/science.354.6312.534 )

Terry Tempest Williams – Excerpt from a letter to Major John Wesley Powell

John Wesley Powell and ten other men loaded food and provisions into four boats in Green River, Wyoming on 24 May 1869. They followed the Green River to the Colorado and traveled through the Grand Canyon. One man left after the first month. Three more left in the third month. Those who remained finished the expedition on on 30 August. They were the first Europeans known to pass through the Grand Canyon. Powell was appointed the second director of the US Geological Survey. His scientific study of the southwestern United States convinced him that agriculture and deserts should not mix.(Wikipedia).

Terry Tempest Williams writes him a letter in The Hour of Land: A Personal Topography of America’s National Parks. This is an excerpt.

I have learned from your history Major Powell, that it is only through the power of our own encounters and explorations of the wild that we can cultivate hope because we have experienced both the awe and humility in nature. We can passionately enter in to the politics of place, even the realm of public policy and change it, if we dare to speak from the authority of our own residencies.