The Joint Committee on Quantitative Assessment of Research of the International Mathematical Union, the International Council of Industrial and Applied Mathematics, and the Institutes of Statistics has just released a report on citation statistics.1 As academic readers are well aware, Thomson Scientific (formerly the Institute for Scientific Information) has for many years used its database to provide rankings of journals (impact factors) and even of scientists (see, for example, http://isihighlycited.com/). Many authors have criticized impact factors and simple citation counts. Some have proposed other indices based on the same statistics. Here's what the authors conclude.We do not dismiss citation statistics as a tool for assessing the quality of research--citation data and statistics can provide some valuable information. We recognize that assessment must be practical, and for this reason easily‐derived citation statistics almost surely will be part of the process. But citation data provide only a limited and incomplete view of research quality, and the statistics derived from citation data are sometimes poorly understood and misused. Research is too important to measure its value with only a single coarse tool.
The graph at the top of this entry is just one reason why impact factors and citation indices must be used with caution. Citation patterns differ markedly among fields, both in the average number of citations that an article will receive and in how quickly those citations accumulate. The report also points out that citation databases don't help (much) in evaluating the quality of the work cited. A paper may be cited frequently only because it's a minority opinion that everyone feels compelled to include in a "but see".2
The report concludes by discussing a paper by Goldstein and Spiegelhalter investigating the use of League Tables for assessing outcomes in healthcare and education.
1Thanks to Andrew Gelman for the link.
2A common pattern in scholarly papers is for the author to cite several sources as establishing a consensus for a particular position and to cite a dissenting opinion as "but see so-and-so for a different opinion." I do it frequently myself.
3Even the complicated ones like those at eigenfactor.org must be treated cautiously.
The report concludes by discussing a paper by Goldstein and Spiegelhalter investigating the use of League Tables for assessing outcomes in healthcare and education.
The article by Goldstein and Spiegelhalter is valuable to read today because it makes clear that the over‐reliance on simple‐minded statistics in research assessment is not an isolated problem. Governments, institutions, and individuals have struggled with similar problems in the past in other contexts, and they have found ways to better understand the statistical tools and to augment them with other means of assessment.Undoubtedly there is useful information buried in patterns of citation, but it's equally clear that the simple statistics derived from them3 must be interpreted with great caution.
1Thanks to Andrew Gelman for the link.
2A common pattern in scholarly papers is for the author to cite several sources as establishing a consensus for a particular position and to cite a dissenting opinion as "but see so-and-so for a different opinion." I do it frequently myself.
3Even the complicated ones like those at eigenfactor.org must be treated cautiously.
Leave a comment