Uncommon Ground

Designing exploratory studies: measurement

On Wednesday I argued that we need carefully done exploratory studies to discover phenomena as much as we need carefully done experimental studies to test explanations for the phenomena that have been discovered.1 Andrew Gelman suggests four preliminary principles:

  1. Validity and reliability of measurements.
  2. Measuring lots of different things.
  3. Connections between quantitative and qualitative data.2
  4. Collect or construct continuous measurements where possible.

Today I’m going to focus on #1, the validity and reliability of measurements.

If there happen to be any social scientists reading this, it’s likely to come as a shock to you to learn that most ecologists and evolutionary biologists haven’t thought too carefully about the problem of measurement, or at least that’s been my experience. My ecologist and evolutionary biologist friends are probably scratching their heads. “What the heck does Holsinger mean by ‘the problem of measurement.'” I’m sure I’m going to butcher this, because what little I think I know I picked up informally second hand, but here’s how I understand it.

Look back at that post from Wednesday. I wrote

We often (or at least I often) have only a vague idea about how features I’m interested in relate to one another. Take leaf mass per area (LMA)1 and mean annual temperature or mean annual precipitation, for example.

I just took it as given that LMA and MAP are interesting variables. Yes, there’s background information suggesting why something like LMA might be interesting (because it might be related to rates of carbon acquisition and resistance to generalist herbivores) and that there are reasons it’s likely to be related to something like MAP (because MAP is related to water availability). But we leap immediately to looking at LMA and MAP as if they were really the variables that matter rather than treating them as indicators of some much more complicated underlying feature. We know, for example, there are other traits related to rates of carbon acquisition and other environmental variables related to water availability, but we tend to behave as if LMA is the trait that matters or as if MAP is the environmental variable that matters.

How would our thinking change if we were more conscious of treating LMA and MAP as indicators of something important than as something important in and of themselves?

First, we’d need to determine whether the indicators are valid. That would mean figuring out some way to justify the assertion that if we measure LMA on a series of different leaves and arrange the leaves in order of increasing (or decreasing) LMA, the order is the same as what we’d get if we ranked the leaves in terms of that “really important thing we’re interested in.” That is, roughly, what’s meant by saying that a measurement is valid. It requires some theory or pre-existing data (or some combination of the two) to justify the monotonic connection between the indicator we measure and the “really important thing we’re interested in.”

Second, we’d need to determine whether the indicators are reliable. That would mean determining experimentally whether or not the measuring the indicator provides a reliable ordering of individuals with respect to the “really important thing we’re interested in.” Don’t ask me how to do this. This is the part I haven’t figured out yet. I know that folks who work in psychometrics and other social science fields have ways of doing it in their fields. I’m sure we can find a way to do it in ecology and evolutionary biology, but I haven’t studied the measurement literature in the social sciences and I haven’t thought enough about it to know how to do it properly in ecology and evolution. I just know that figuring out how to do it is very important.

And until I figure out how to do it well, I know that remembering that (a) almost invariably what I’m measuring is an indicator of something else that is the thing that really matters and (b) I need to be as sure as I can that what I’m measuring is a valid and reliable indicator of that thing (even if I don’t have any formal way of giving myself that assurance, at least for the time being).


1At least they are needed as much in ecology and evolution. Perhaps in fields, like some areas of physics, where theory can make predictions that should hold in all places and all times exploratory studies aren’t needed. But if that’s the case they aren’t needed because the exploratory work is done with mathematical theory instead of with observation.

2Remember that Gelman was writing about exploratory studies in social sciences. I haven’t thought about this a lot yet, but I’m not sure that qualitative data plays the same role in ecology and evolution. I can’t think of any data we collect that is comparable to data social scientists collect in interviews. Maybe we need to think about whether we are missing important aspects of the phenomena we study by not collecting something analogous to interview data when we’re in the field.

Leave a Comment

Your email address will not be published. Required fields are marked *