Should I vote in the QS World Rankings?

I was invited recently by email to vote in the QS world rankings of universities. Part of the QS rankings (40%) goes for something called “academic reputation” and I suppose my vote/opinion counts towards that. My main problem with rankings in general is that sometimes people use them for things that they are not supposed to be used for. For example, those in power might use the rankings to make policy decisions. Further, it is never really disclosed how the rankings are arrived at. Yes, there are some vague explanations, but it is never possible to replicate the results. Given all that, should I vote?

There have been many posts recently about world rankings. One describes some impacts of the world rankings on higher education and its quality and regulation. Another is called the ultimate absurdity of college rankings. A third by Richard Holmes gives some reasons why rankings are unreliable, especially the subject rankings.

Anyway, to be more specific about my own dilemma, should I give my opinion about other universities? I have a problem because I don’t really know enough. I have to name up to 30 (foreign) universities that produce the best research in the natural sciences, as well as 10 in my own country (Ireland). Sure, we can all name Caltech and so on at the top. After the top ten or so, I run out of names.  So what do I do? Who should I name? How do I decide if one university is better than another?  Can I compare, for example, Kyoto University with the University of New South Wales? To be honest, I know they are both very good with top class reputations, but I couldn’t rank one above the other. (Kyoto was 35 and NSW was 52 in the overall QS rankings in 2012, by the way.) To be honest, I probably wouldn’t name either of them in my 30, but that’s just because I don’t know them and I don’t know anyone there. The same goes for hundreds of other universities.

So what sort of a picture does my response, and the responses of other academics like me, actually give? The number of votes a university gets is probably proportional to how many foreigners know someone who works there. The top ten will remain the top ten, because everyone automatically assumes they are great and everyone has heard of them. But when you move down the list to around number 50, and below, how precise can we be? Some volatility is to be expected, and indeed happens. (Why volatility happens is not always clear, as discussed here)

The data for “reputation” is gathered in such a random fashion that it cannot be meaningful.

The president of University College Cork wrote an email to all staff in May 2011, and got into hot water. The full text is here QS now says that they use “sophisticated anomaly detection algorithms” among other things, to stop academics asking their friends to vote for them.

It was recently exposed that the QS rankings used a site that pays people to fill out surveys. There is an excellent article about this here by Elizabeth Redden. Indeed, I was offered 300 pounds in credits to complete my survey. After I did so, out of curiosity, I tried to buy QS reports with my credits. The website didn’t work, I got an error message and the credits were worthless.

i wonder if anyone has actually done a survey to see how many students actually use rankings when deciding on a university.


4 thoughts on “Should I vote in the QS World Rankings?

  1. Pingback: Ninth Level Ireland » Blog Archive » Should I vote in the QS World Rankings?

  2. I’m trying to come up with a comparison between football clubs and university league tables, partly because there is a fashion for using this comparison in the press and by universities themselves. It works in illustrating the folly of rankings once you accept that there is no equivalent of actual matches between pairs of universities. So, if football clubs could not be compared on the results of matches, how would we decide the best club in the world? Many football fans would abandon the question at this point as facile, but this is exactly the approach that many global university rankings make.

    So to compare clubs in the absence of match results, we could use measures like the size of the ground, the wage bill, the number of players from outside the country where the club is based, the number of transfers, the number of goals for, goals against, offside decisions, fouls committed, corner kicks, replica kits sold, fans’ reputational survey and so on. Now imagine my ranking firm applies some not-exactly transparent and irreproducible ‘regional adjustments’ to try and factor in that some countries are bigger/richer than others. Also permit me to adjust my methods and weightings for measures each year. Both changing weightings and irreproducible methods tend to be a feature of certain university rankings.

    Why would anyone take my fictional ranking of universities, sorry -football clubs, seriously?
    You might not be surprised to see Manchester United and Real Madrid at the top of the rankings. But how seriously would you take the relative ranking of say, Shamrock Rovers, A-league team Perth Glory and K-league FC Seoul?

    • Mark, thanks for your comment. I have seen this analogy before, it is often useful. I think your comment captures exactly the point I was making, and explains it better! Even if the people you surveyed were all the professional footballers in the world, how much sense do the results make.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s