The Poisonous Effect of ‘Rankings’ in Research

This year, we have seen cases of researchers with a high impact factor being assigned to second-tier universities, which are looking to climb a few steps in the world university ranking. This is just a small sample of a systemic corruption of academic and scientific quality indicators.

Fernando Broncano / SINC

German economist Horst Siebert called the unintended effects of incentives a ‘cobra effect’, recalling the measure of the British colonial government of India when it decided to pay citizens for the delivery of each dead cobra snake in Delhi, an industry of poisonous snake breeding.

Thus, the well-known police series The Wire alludes to the techniques of manipulating success rates by the Baltimore police and thousands of other examples that are easily found in all areas of economic and political institutions. Unfortunately, they are common practices in the management of public goods such as health and education.

It is a systemic failure of the metric society that dominates the contemporary world based on the massive use of quantitative indicators. Rankings were born as statistical devices for information on complex systems in a very large and globalized world, but soon generated what falls under the so-called “Goodhart’s Law”’ (by economist Charles Goodhart): “When a measure becomes a target, it ceases to be a good measure”.

“When a measure becomes a target, it ceases to be a good measure” (Goodhart’s Law)

In this way, we come to the perversion of the system of university institutions rankings in a world of internationalised and globalised campuses, in which attracting students is part of a growing economic bubble.

The Spanish press has reflected these days various exercises of the ‘cobra effect’, such as assigning researchers with a high impact factor to second-tier universities to climb a few steps in the world ranking. This is just a small sample of a systemic corruption of academic and scientific quality indicators. It is worth remembering, albeit briefly, how an indicator, the impact factor, created an adaptive environment of universities, research centres, scientific journals, and personal curricula.

Peer citations to gain recognition

At the dawn of the current scientific production system, at the end of World War II, sociologists and science historians developed the concept of ‘invisible college’ to capture the system of authorities of a speciality or discipline and developed the hypothesis that citations among scientists would allow studying the degree of recognition and the invisible ties in a research community. The scope of the idea was limited and relative to disciplines and research topics.

At the same time, Eugene Garfield, a chemist obsessed with information, developed the well-known indicator of ‘impact factor’ of a journal in the 1950s. This indicator reflects the average yearly number of citations of articles published in the last two years in a given journal. He founded the Institute for Scientific Information (ISI) which began to publish the Journal Citations Report (JCR), with the aim that libraries would have information about which journals to subscribe to in each scientific field. ISI was faced with bankruptcy and went through a couple of institutions until it was bought by the venture capital firm Barin Private Equity Asia, which created the service company Clarivate Analytics, which owns the largest database of scientific production and, in turn, created the Web of Science, in which the impact factor is the great indicator.

The third step in the production of this enormous ‘cobra effect’ was the liberalisation of higher education in the agreements of the General Agreement on Trade in Services (within the World Trade Organisation). These implied a global regulation to homogenise university systems, which went from being training or education to educational services.

In Europe, the Bologna system of homogenisation was one of the stages of this liberalisation. The effect was to create a global market for educational services that demanded a global information system. And so were born, first, the Shanghai ranking and, later, the Times Higher Education and others linked to various groups of communication. These rankings employ various indicators, but the journal impact factor remains central.

The convergence of the three ideas turned the rankings into evaluation systems that have generated a systemic ‘cobra effect’ in science management and higher education.

Publication for the sake of publication

This whole process stems from distortions impossible to control, despite the fact that they have been criticised by indicator specialists. An increasing number of statements from scientific institutions have advocated for the abandonment of the impact factor as a measure of personal performance and scientific quality. Although it has been progressively replaced by relativised indicators, such as quartiles, the causal basis remains the basis of the evaluation.

Much more serious has been the perversion that the global research system has generated. Firstly, it has produced the culture of publication for the sake of publication. Secondly, it has produced the concentration and oligopoly of publications in a few companies that in turn host internal practices of journals to climb up the ranking steps. In third and most disastrous place is the perversion of the universities management turned into degree service companies whose market value is the place in the ranking. The fact that this immense ‘cobra effect’ was based on controversial and debatable hypotheses has not prevented the creation of the higher education bubble as one of the basic structures of current economic globalisation and new geostrategies of global competition.

Fernando Broncano, Professor of Logic and Philosophy of Science at the Carlos III University of Madrid. He has written on the theory of rationality, political epistemology and the theory of contemporary culture.