Uncommon Ground

Monthly Archive: November 2016

Designing exploratory studies: measurement (part 2)

Remember Gelman’s preliminary principles for designing exploratory studies:

  1. Validity and reliability of measurements.
  2. Measuring lots of different things.
  3. Connections between quantitative and qualitative data.
  4. Collect or construct continuous measurements where possible.

I already wrote about validity and reliability. I admitted to not knowing enough yet to provide advice on assessing the reliability of measurements ecologists and evolutionists make (except in the very limited sense of whether or not repeated measurements of the same characteristic give similar results). For the time being that means I’ll focus on

  • Remembering that I’m measuring an indicator of something else that is the thing that really matters, not the thing that really matters itself.
  • Being as sure as I can that what I’m measuring is a valid and reliable indicator of that thing, even though the best I can do with that right now is a sort of heuristic connection between a vague notion of what I really think matters, underlying theory, and expectations derived from earlier work.

It’s that second part where “measuring lots of different things” comes in. Let’s go back to LMA and MAP. I’m interested in LMA because it’s an important component of the leaf economics spectrum. There are reasons to expect that tough leaves (those in which LMA is high) will not only be more resistant to herbivory from generalist herbivores, but that they will have lower rates of photosynthesis. Plants are investing more in those leaves. So high LMA is, in some vague sense, an indicator of the extent to which resource conservation is more important to plants than rapid acquisition of resources. So in designing an exploratory study, I should think about other traits plants have that could be indicators of resource conservation vs. rapid resource acquisition and measure as many of them as I can. A few that occur to me are leaf area, photosynthetic rate, leaf nitrogen content, leaf C/N ratio, tissue density, leaf longevity, and leaf thickness.

If I measure all of these (or at least several of them) and think of them as indicators of variation on the underlying “thing I really care about”, I can then imagine treating that underlying “thing I really care about” as a latent variable. One way, but almost certainly not the only way, I could assess the relationship between that latent variable and MAP would be to perform a factor analysis on the trait dataset, identify a single latent factor, and use that factor as the dependent variable whose variation I study in relation to MAP. Of course, MAP is only one way in which we might assess water availability in the environment. Others that might be especially relevant for perennials with long-lived leaves (like Protea) in the Cape Floristic Region rainfall seasonality, maximum number of days between days with “significant” rainfall in the summer, total summer rainfall, estimated potential evapotranspiration for the year, and estimated PET for the summer. A standard way to relate the “resource conservation” factor to the “water availability” factor would be a canonical correspondence analysis.

I am not advocating that we all start doing canonical correspondence analyses as our method of choice in designing exploratory studies, this way of thinking about exploratory studies does help me clarify (a bit) what it is that I’m really looking for. I still have work to do on getting it right, but it feels as if I’m heading towards something analogous to exploratory factor analysis (to identify factors that are valid, in the sense that they are interpretable and related in a meaningful way to existing theoretical constructs and understanding) and confirmatory factor analysis (to confirm that the exploration has revealed factors that can be reliably measured).

Stay tuned. It is likely to be a while before I have more thoughts to share, but as they develop, they’ll appear here, and if you follow along, you’ll be the first to hear about them.

On the importance of openness in scholarship

I was recently looking something up in Evernote, and I ran across a post by Eric Rauchway on Crooked Timber from July 2013. The post concerns the American Historical Association/s proposed recommendation for an embargo on dissertations. The AHA adopted a Statement on Policies Regarding the Option to Embargo Completed PhD Dissertations on 19 July 2013. It begins

The American Historical Association strongly encourages graduate programs and university libraries to adopt a policy that allows the embargoing of completed history PhD dissertations in digital form for as many as six years.

There was a lot of debate about the wisdom of dissertation embargoes before and after the statement was announced. Rauchway finished his post with a comment that all of us should think about.

When we find ourselves trying to make scholarship less readily available – however good our intentions – we should probably ask ourselves if we can solve our problems some other way.

Designing exploratory studies: measurement

On Wednesday I argued that we need carefully done exploratory studies to discover phenomena as much as we need carefully done experimental studies to test explanations for the phenomena that have been discovered.1 Andrew Gelman suggests four preliminary principles:

  1. Validity and reliability of measurements.
  2. Measuring lots of different things.
  3. Connections between quantitative and qualitative data.2
  4. Collect or construct continuous measurements where possible.

Today I’m going to focus on #1, the validity and reliability of measurements.

If there happen to be any social scientists reading this, it’s likely to come as a shock to you to learn that most ecologists and evolutionary biologists haven’t thought too carefully about the problem of measurement, or at least that’s been my experience. My ecologist and evolutionary biologist friends are probably scratching their heads. “What the heck does Holsinger mean by ‘the problem of measurement.'” I’m sure I’m going to butcher this, because what little I think I know I picked up informally second hand, but here’s how I understand it.

(more…)

Designing exploratory studies: preliminary thoughts

Andrew Gelman has a long, interesting, and important post about designing exploratory studies. It was inspired by the following comment from Ed Hagen following a blog post about a paper in Psychological Science.

Exploratory studies need to become a “thing”. Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

I think Ed’s suggestion is too narrow. Exploratory studies are essential to good science, not just good social science. We often (or at least I often) have only a vague idea about how features I’m interested in relate to one another. Take leaf mass per area (LMA)

We often (or at least I often) have only a vague idea about how features I’m interested in relate to one another. Take leaf mass per area (LMA)1 and mean annual temperature or mean annual precipitation, for example. In a worldwide dataset compiled by IJ Wright and colleagues2, tougher leaves (higher values of LMA) are associated with warmer temperatures and less rainfall.

LMA vs. mean annual temperature and log(mean annual rainfall)

LMA, mean annual temperature, and log(mean annual rainfall) in 2370 species at 163 sites (from Wright et al. 2004)

We expected similar relationships in our analysis of Protea and Pelargonium,3 but we weren’t trying to test those expectations. We were trying to determine what those relationships were. We were, in other words, exploring our data, and guess what we found. Tougher leaves are associated with less rainfall in both general and with warmer temperatures in Protea. They were, however, associated with cooler temperatures in Pelargonium, exactly the opposite of what we expected. One reason for the difference might be that Pelargonium leaves are drought deciduous, so they avoid the summer drought characteristic of the regions from which our samples were collected. That is, of course, a post hoc explanation and has to be interpreted cautiously as a hypothesis to be tested, not as an established causal explanation. But that is precisely the point. We needed to investigate the phenomena to identify a pattern. Only then could we generate a hypothesis worth testing.

I find that I am usually more interested in discovering what the phenomena are than in tying down the mechanistic explanations for them. The problem, as Ed Hagen suggests, is that studies that explicitly label themselves as exploratory play little role in science. They tend to be seen as “fishing expeditions,” not serious science. The key, as Hagen suggests, is that to be useful, exploratory studies have to be done as carefully as explicit, hypothesis-testing confirmatory studies. A corollary he didn’t mention is that science will be well-served if we observe the distinction between exploratory and confirmatory studies.4

(more…)

Follow up on dodging a bullet

Last week I mentioned that I had considered participating in Benefunder and that an article in Science made me glad I’d decided not to participate.

I was tempted because it sounded like a very promising idea. In the end, though, I just couldn’t see investing $500 in the project. It seemed too unlikely that a donor would be interested in supporting the esoteric research that I do.

Christian Braemer, CEO/co-founder of Benefunder, left a comment on that post. If you’re interested in Benefunder, I encourage to visit the original post and read the whole comment. Here’s the key part of what he has to say about the article in Science:

The worst part is the title, which implies we’ve folded and that couldn’t be further from the truth as you’ll see in a string of press releases that are about to come out regarding new partners, board members, and investors (although this story certainly isn’t going to help with the latter).

Not to be overly dramatic, but this story is a major threat to finding new ways to fund research in this country. Consider this, with our new approach, if we get just 4% of current DAF distributions, that equates to over $1B in new funding for research. Crowdfunding (which we are not) doesn’t have the right alignment of interests, expectations, or volume to be able to pull this off – at least not for the foreseeable future.

By the way, Christian posted his comment on 8 November, the day after my original post went live. His comment only went live this morning, because it’s the first time in a week I’ve had a chance to check in.

 

Oxford Handbook on the Science of Science Communication

I just ran across an old (13 October 2015) press release from the Annenberg Public Policy Center of the University of Pennsylvania announcing that Oxford University Press will publish the Oxford Handbook on the Science of Science Communication in 2017.1 The very brief press release on the Annenberg Center’s website makes it sound very interesting. I look forward to learning more about it as its release date approaches. In the meantime, here’s a very brief description lifted from the press release.

The handbook, developed for scientists, academics and students of science communication, will grapple with the failure of widely accessible scientific evidence to effectively inform public controversies on issues including climate change, the human papilloma virus vaccine, and the safety of bio-engineered foods on the market in the United States.


1OUP doesn’t seem to have an announcement of it on its website yet.

How to think about replication

Caroline Tucker reviewed a paper by Nathan Lemoine and colleagues in late September and reminded us that inferring anything from small, noisy samples is problematic.1 Ramin Skibba now describes a good example of the problems that can arise.

In 1988, Fritz Strack and his colleagues reported that people found cartoons funnier if they held a pen in their teeth than if they held it between their lips. Why? Because holding a pen between your teeth causes you to smile, while holding one between your lips causes you to pout. This report spawned a series of studies on the “facial feedback hypothesis”, the hypothesis that facial expressions influence our emotional states. It seems plausible enough, and I know that I’ve read advice along this line in various places even though I’d never heard of the “facial feedback hypothesis” until I read Skibba’s article.

Unfortunately, the hypothesis wasn’t supported in what sounds like a pretty definitive study: 17 experiments and 1900 experimental subjects. Sixteen of the studies had large enough samples to be confident that the failure to detect an effect wasn’t a result of small sample size. Strack disagrees. He argues that (a) using a video camera to record participants may have made them self-conscious and suppressed their responses and (b) the cartoons were too old or too unfamiliar to participants to evoke an appropriate response.

Let’s take Strack at his word. Let’s assume he’s right on both counts. How important do you think the facial feedback to emotions is if being recorded by a video camera or being shown the wrong cartoons causes it to disappear (or at least to be undetectable)? I don’t doubt that Strack detected the effect in his sample, but the attempt to replicate his results suggest that the effect is either very sensitive to context or very weak.

I haven’t gone back to Strack’s paper to check on the original sample size, but the problem here is precisely what you’d expect to encounter if the original conclusions were based on a study in which the sample size was small relative to the signal-to-noise ratio. To reliably detect a small effect that varies across contexts requires either (a) very large sample size (if you want your conclusions to apply to the entire population) or (b) very careful specification of the precise subpopulation to which your conclusions apply (and extreme caution in attempting to generalize beyond it).

(more…)

Please vote

I votedToday is election day in the United States. The polls opened in Connecticut 2 hours ago, and there was a long line at my polling place by the time I arrived at 6:15.

If you are a registered voter and you aren’t one of the 41 million who have already cast a ballot in early voting, please vote today. If you don’t know where to vote, the League of Women Voters Education Fund has compiled information on the location of polling places at Vote411.org. All you need to do is put in your home address, and you’ll get a street address, hours of operation, and a Google map showing where to go. In addition to the Presidential race, there are important state and local races around the country. Please take the time to make your voice heard. Representative government works only if we select representatives who reflect our values. Voting is how we express those values.