Recent media reports announced that the University of Belgrade had fallen in the Shanghai Ranking of the world’s universities. The University of Belgrade is no longer among the top 500 higher education institutions in the world, but is now placed in the 501–600 group. For days, the news circulated through public discourse, leaving room for many questions and concerns. One question, however, stood out: why has the quality of higher education declined so dramatically?

Yet that question already contains an implicit assumption: that a fall in the ranking automatically means a fall in quality. Before accepting that assumption as fact, it is worth asking what the Shanghai Ranking actually measures, how it does so, and how much confidence can be placed in its results. In other words, before attempting to explain why quality has declined, it is first necessary to establish whether the ranking provides a reliable answer to that question at all. This leads to the central question: to what extent does the Academic Ranking of World Universities actually reflect the quality of higher education, and how do its methodology and contemporary use shape the behaviour of universities and entire national education systems?

To answer this question, we need to take a step back and examine how the idea of ranking universities emerged and how it developed over time.

From counting “great men” to a global race for prestige

The idea of ranking universities is not a product of the digital age. Its roots reach back to the beginning of the twentieth century, when eminent figures, scientists and “geniuses” educated at particular institutions were counted in Europe and the United States. The logic was simple: an individual’s success was attributed to the quality of the institution they had attended.

Nevertheless, during the first half of the twentieth century, the prevailing belief was still that qualitative assessments, accreditation and standards were far more important than numerical comparison, or rank. When an official government classification of colleges was attempted in the United States in the early 1900s, institutions that felt they had been unfairly evaluated responded with strong protests.

The turning point came in the 1950s and 1960s, at the height of the Cold War and the competition for scientific supremacy. Universities began to be viewed differently. Their performance was no longer assessed separately; instead, surveys were used to rank universities within national education systems. In other words, the introduction of reputational surveys produced the form of ranking we recognise today, in which the rise of one institution inevitably means the decline of another.

The true commercial boom of this ranking system came in the 1980s, when popular magazines began publishing annual college rankings. Ranking moved from academic circles into the media and became a product aimed at students and parents. From that point onward, rankings were no longer merely a source of information. They also became a matter of public prestige. As both students and universities increasingly recognised their publicity value, ranking developed into what Hazelkorn calls a “rankings obsession”. Rankings thus became a major driver of the race for reputation.

Globalisation, the knowledge economy and the growing international mobility of students intensified this competition and moved it from the national to the global level, while giving it increasingly broad and powerful geopolitical implications. It was at precisely this moment, in 2003, that the Academic Ranking of World Universities was created at Shanghai Jiao Tong University, becoming widely known as the Shanghai Ranking. Although it was initially designed for the university’s internal purposes, it soon became the first global university ranking.

The Chinese experiment that changed the rules of the game

The Shanghai Ranking initially began as an internal project of the Institute of Higher Education at Shanghai Jiao Tong University. Its aim was to establish where Chinese universities stood in relation to global leaders, in order to secure state support and chart a path towards creating world-class institutions. Only when the results were published online free of charge did the ranking attract global attention and acquire international significance. As early as the following year, European ministers cited the Shanghai Ranking to justify major education reforms, university mergers and the reallocation of funding. What began as an internal tool soon developed into a consultancy that today offers universities around the world a broad range of paid analytical and advisory services.

But what actually lies behind a single position in the Shanghai Ranking? To understand why ministers and university rectors rely on it so heavily, but also why experts criticise it so strongly, it is worth looking at how universities are ranked.

Selection for the Shanghai Ranking is based on clearly defined scientific parameters. Only institutions that can point to Nobel laureates, Fields Medal winners, leading highly cited researchers, and publications in major scientific journals and recognised academic databases are included. In other words, although more than 2,500 universities are assessed methodologically, the official ranking displays only the top 1,000.

Each university is then assessed in four areas:

  • Quality of education (10%). This counts how many former students who completed undergraduate, master’s or doctoral degrees at the university later received a Nobel Prize or Fields Medal. The older the degree, the less weight it carries, but degrees awarded as far back as the 1920s are still counted.

  • Quality of faculty (40%). Half of this score depends on how many university employees have received a Nobel Prize or Fields Medal. The award is attributed to the university at which the laureate works at the time the prize is awarded, rather than the institution where the research was conducted. The other half depends on the number of researchers included on international lists of highly cited scientists.

  • Research output (40%). Half of this score depends on the number of papers published in Nature and Science during the previous five years. The other half depends on the total number of papers indexed in international scientific databases during a single year, with papers in the social sciences counted twice.

  • Per capita performance (10%). Results from the previous categories are divided by the number of academic and research staff. This partially reduces the advantage enjoyed by large universities.

Finally, all results are combined. In each category, the best-performing university receives 100 points, while all others receive a score proportional to their result compared with the leader. These scores are then weighted according to the percentages above and combined into a single final number used to order universities in the ranking.

The illusion of methodology: What does the Shanghai Ranking actually measure, and what does it not?

At first glance, such a methodology appears clear and convincing. The data come from independent sources, the rules are known in advance, and the result is a single number that anyone can understand. This is precisely why the Shanghai Ranking has many supporters among policymakers and university administrations. Yet when the scoring rules are examined more closely, it becomes clear that every argument in its favour is accompanied by a serious criticism.

1. Objective data, invisible decisions

Supporters of the ranking most often emphasise its objectivity because, unlike other rankings, it does not rely on surveys or data provided by universities themselves, but exclusively on publicly available sources. Subjectivity does not disappear, however; it is merely transferred to decisions that the public cannot see. Raw data are unavailable, statistical adjustments are unexplained, and the ranking’s authors themselves decide which institution should receive credit for a particular award or publication.

The case of Albert Einstein’s Nobel Prize demonstrates how decisive this can be. Einstein and several other Nobel laureates worked at the University of Berlin before the Second World War. After the division of the city, two universities claimed its legacy. One was the Free University of Berlin, founded in 1948 in West Berlin, and the other was Humboldt University, located in the eastern part of the city. In the first ranking in 2003, the authors attributed the pre-war prizes to the Free University, helping it reach 95th place. After Humboldt University protested, the same awards were attributed to Humboldt the following year, allowing it to rise to 95th place while the Free University fell by more than one hundred positions. When the Free University also protested, the ranking’s authors simply removed both universities from the list, and neither had returned by 2007.

The quality of neither university changed during that period. What changed was only the decision about which institution should receive credit for an award granted almost a century earlier.

2. Nobel Prizes do not teach in classrooms

Nobel Prizes, Fields Medals and high citation rates may appear to supporters of the ranking as clear indicators of outstanding staff and a stimulating academic environment, but they primarily measure historical prestige rather than what students receive today. Degrees awarded in the 1920s continue to generate points, while prizes are attributed to the university where the laureate works at the time of the award, even when the relevant research was carried out much earlier and elsewhere.

Einstein conducted the research for which he was rewarded while working at the Swiss patent office, yet received the prize as a professor in Berlin. Andrew Fire’s 2006 Nobel Prize helped Stanford University move from third to second place, even though the research had been carried out at another institution.

Because only prizes in physics, chemistry, medicine, economics and mathematics are counted, the humanities, the arts and most social science disciplines remain completely invisible.

3. The number of papers is not a measure of the value of science

Even those who doubt that the ranking measures educational quality often accept that it reliably measures scientific output, but that picture is incomplete as well. Scientific databases strongly favour English-language publications and the natural sciences. Books, which are crucial in the social sciences, are generally not counted, nor are conference proceedings, which are essential in engineering and computer science.

The ranking also counts how many papers have been published rather than how influential they are, even though most published papers are hardly ever cited. This approach particularly disadvantages institutions specialising in the social sciences and humanities. Despite this, the London School of Economics and Political Science, one of the world’s most prestigious institutions in these fields, appeared only in the 201–300 group in 2006. In the same year, the ranking published by Times Higher Education Supplement placed it 17th.

The indicators also create room for manipulation. One fifth of the total score depends on the number of highly cited researchers who list a particular university as their institution. Some Saudi universities therefore began hiring distinguished foreign researchers on part-time contracts on the condition that they also listed the Saudi institution in their affiliation.

Of the 170 highly cited researchers affiliated with universities in Saudi Arabia in 2014, as many as 139, or 82 per cent, had their primary employment in another country. King Abdulaziz University alone had 122 such researchers. In 2004, Saudi Arabia had no university among the world’s top 500; ten years later, it had two in the top 200 and another two in the top 500.

The ranking was therefore improved by adding names to institutional affiliations rather than by developing domestic science. Today, the ranking’s authors consider only a researcher’s primary institution, which can be interpreted as a response to precisely such practices.

4. A simple number, a complex error

Supporters see the reduction of all results to a single number, with the best university receiving 100 points, as an advantage. Methodologists, however, warn that this simplicity hides serious mathematical weaknesses. Because the best university in each category receives 100 points again every year while the weights remain unchanged, a change in the performance of only one leading university alters the value of points for all others.

A simple example has shown that this can completely reverse the order of universities beneath it, and even cause a university that improves in every category to fall from second to last place. Since most indicators are based on absolute numbers, the ranking systematically favours large universities.

Among institutions ranked roughly between 400th and 500th place, more than 90 per cent receive no points at all for Nobel Prizes or Fields Medals. For most universities in the ranking, a complex methodology therefore effectively comes down to simply counting papers.

5. Reforms that change a position, not quality

The most persuasive argument in favour of the ranking is that it encourages governments and universities to make changes. China is the example most frequently cited. Through special state programmes for the development of leading universities, known as Projects 211 and 985, China invested substantial resources in its strongest institutions. The number of Chinese universities among the top 500 in the Shanghai Ranking rose from 14 in 2003 to 24 in 2009. In this case, the ranking served as a clear target that stimulated investment in science.

The problem is that a better ranking can also be achieved without any genuine improvement in research or teaching. A group of methodologists demonstrated this through a thought experiment. A university administration seeking to rise quickly in the ranking could close its law, humanities and most social science departments because they generate few points. The money saved could then be redirected to research groups in medicine or the natural sciences, which generate prizes, highly cited researchers and papers in prestigious journals.

The ranking would reward such a move even though the university would lose a large part of its educational mission.

The same authors showed that a similar effect can be achieved through university mergers. Since most indicators are based on the total number of prizes, researchers and publications, a larger institution automatically receives more points. The former University of Paris was divided into thirteen smaller universities in the early 1970s, none of which could individually compete with the world’s largest universities.

The authors calculated that merging four of them, focused on the natural sciences and medicine, would, using 2007 data, create an institution approximately comparable with Harvard. This could be achieved without spending a single additional euro and without producing a single new scientific paper, simply by adding together existing results. The example did not remain purely theoretical. In 2009, France merged three universities in Strasbourg and continued with similar mergers, while researchers have argued that one of the goals was precisely to improve their Shanghai Ranking position.

There is also a deeper problem. Since a university’s position depends on comparison with others, the rise of one university necessarily means the fall of another. The ranking therefore cannot demonstrate that an entire education system has improved, only who has overtaken whom. Research also shows that universities at the very top of the ranking almost never change because only the wealthiest institutions can afford the elite researchers who generate points. Instead of encouraging balanced development, the ranking therefore directs attention and money towards institutions that are already wealthy and prestigious.

6. A guide that does not show students the way

For students planning to study abroad, the ranking can serve as a useful first point of reference, but a single number for an entire university can hardly answer all the questions that matter to them. When choosing where to study, students usually need information about particular degree programmes, teaching quality, costs, accommodation and employment opportunities, none of which forms part of the Shanghai Ranking.

The same applies to employers, for whom graduates’ knowledge and skills are probably more relevant when hiring than the number of Nobel Prizes a university received several decades ago. It should also be remembered that different rankings often produce different results. Of the top 100 universities in three leading global rankings, only 35 appear in all three.

Finally, the top 100 universities account for less than one per cent of more than 15,000 higher education institutions worldwide. For a large number of students, the ranking therefore provides only limited insight into the institutions they will actually attend.

7. A measure tailored to one model

Supporters present the ranking as a neutral global measure that makes it possible to compare universities and track changes in the distribution of scientific power, but a measure cannot be neutral if its criteria are not neutral.

Analysis of the indicators shows that the ranking implicitly rewards old universities that have not changed their names throughout history, are located in stable English-speaking countries, enjoy extensive autonomy and have large budgets. That description fits the best-known American and British universities almost perfectly, and these institutions are also found at the top of the ranking.

The relationship with national wealth is also striking. No country with a university in the top 50 has a gross domestic product per capita below 36,000 dollars at purchasing power parity, while the global average is around 14,350 dollars. Systems in which a large proportion of scientific research takes place in separate research institutes, as in France, are disadvantaged because the work of those institutes does not contribute to the Shanghai Ranking score.

Why does a contested ranking still have so much power?

Although it has been subjected to serious methodological criticism in academic literature, the Shanghai Ranking continues to exert significant influence on policymakers and university administrations. Researchers explain that numbers and rankings create the appearance of neutral, “hard” truth.

Constant measurement encourages institutions to adopt external criteria as their own and voluntarily adjust their behaviour to match them. Surveys show that the great majority of universities have formal mechanisms for monitoring their ranking positions. Most university leaders also believe that rank directly affects the willingness of international partners to cooperate.

In such an environment, simply appearing in the ranking can become more important for an individual university than the fundamental methodological questions behind it.

Can higher education be viewed beyond a single ranking?

A review of the academic literature shows that the Shanghai Ranking does not provide a credible picture of the overall quality of higher education. Its real scope is more limited: it primarily reflects accumulated research prestige and institutional size in certain fields of the natural sciences.

The data on which it relies come from public sources, but decisions about how those data are interpreted and to which institutions they are attributed remain outside public view. Awards and citation counts tell us more about researchers’ past achievements than about the quality of teaching today, while counting publications disadvantages the social sciences and humanities. A single aggregate number appears simple but is sensitive to methodological decisions and favours large institutions.

The ranking has encouraged many countries to invest in science, but at the same time it has also rewarded measures that improve ranking positions without producing genuine progress. It offers students only limited insight into what actually matters when choosing where to study. As a global measure, it reflects existing differences in university wealth and tradition more than it objectively measures them.

For these reasons, Shanghai Ranking results are best interpreted with caution, as an indication of the scientific visibility of certain fields rather than as an assessment of the overall quality of an institution. Annual changes in the ranking often do not reflect genuine changes in teaching and research, so decisions should not be based on individual ranking positions.

It is more useful to rely on tools that present several separate indicators and allow users to choose what matters to them. In addition, investing in the development of the higher education system as a whole produces more in the long term than efforts to push one or two institutions up the ranking at any cost.

The news from the beginning of this article should be viewed in that light. The University of Belgrade’s fall into the 501–600 group does not, in itself, prove that the quality of higher education has dramatically declined. It primarily shows how the University currently performs according to one specific set of research indicators.

The real question of quality requires a broader and more careful answer than a single position in a ranking.