Should I vote in the QS World Rankings?

I was invited recently by email to vote in the QS world rankings of universities. Part of the QS rankings (40%) goes for something called “academic reputation” and I suppose my vote/opinion counts towards that. My main problem with rankings in general is that sometimes people use them for things that they are not supposed to be used for. For example, those in power might use the rankings to make policy decisions. Further, it is never really disclosed how the rankings are arrived at. Yes, there are some vague explanations, but it is never possible to replicate the results. Given all that, should I vote?

There have been many posts recently about world rankings. One describes some impacts of the world rankings on higher education and its quality and regulation. Another is called the ultimate absurdity of college rankings. A third by Richard Holmes gives some reasons why rankings are unreliable, especially the subject rankings.

Anyway, to be more specific about my own dilemma, should I give my opinion about other universities? I have a problem because I don’t really know enough. I have to name up to 30 (foreign) universities that produce the best research in the natural sciences, as well as 10 in my own country (Ireland). Sure, we can all name Caltech and so on at the top. After the top ten or so, I run out of names.  So what do I do? Who should I name? How do I decide if one university is better than another?  Can I compare, for example, Kyoto University with the University of New South Wales? To be honest, I know they are both very good with top class reputations, but I couldn’t rank one above the other. (Kyoto was 35 and NSW was 52 in the overall QS rankings in 2012, by the way.) To be honest, I probably wouldn’t name either of them in my 30, but that’s just because I don’t know them and I don’t know anyone there. The same goes for hundreds of other universities.

So what sort of a picture does my response, and the responses of other academics like me, actually give? The number of votes a university gets is probably proportional to how many foreigners know someone who works there. The top ten will remain the top ten, because everyone automatically assumes they are great and everyone has heard of them. But when you move down the list to around number 50, and below, how precise can we be? Some volatility is to be expected, and indeed happens. (Why volatility happens is not always clear, as discussed here)

The data for “reputation” is gathered in such a random fashion that it cannot be meaningful.

The president of University College Cork wrote an email to all staff in May 2011, and got into hot water. The full text is here QS now says that they use “sophisticated anomaly detection algorithms” among other things, to stop academics asking their friends to vote for them.

It was recently exposed that the QS rankings used a site that pays people to fill out surveys. There is an excellent article about this here by Elizabeth Redden. Indeed, I was offered 300 pounds in credits to complete my survey. After I did so, out of curiosity, I tried to buy QS reports with my credits. The website didn’t work, I got an error message and the credits were worthless.

i wonder if anyone has actually done a survey to see how many students actually use rankings when deciding on a university.

Citation indicator in world rankings

An interesting post by Richard Holmes about the THE university world rankings, and why the citation part of the rankings is not yet reliable. I just noticed that he withdrew the post and replaced it with an apology.

The world top 20 in the THE citation indicator rankings are

1.   Moscow (State) Engineering Physics Institute (MEPhI)
1.   Rice University
3.   University of California Santa Cruz
3.   MIT
5.   Princeton
6.   Caltech
7.   University of California Santa Barbara
7.   Stanford
9.   University of California Berkeley
10.  Harvard
11.  Royal Holloway London
12.  Chicago
13.  Northwestern
14.  Tokyo Metropolitan University
14.  University of Colorado Boulder
16.  University of Washington Seattle
16.  Duke
18.  University of California San Diego
18.  University of Pennsylvania
18.  Cambridge

The list looks slightly odd to me. I would like to know how these numbers are calculated, but I can’t find the information anywhere. It is apparently calculated by Thomson-Reuters, who use their large database of citation data to compute this number.  I would like to calculate it for my own university UCD, but I don’t know how. In 2011 UCD got 80.5 and in 2012 UCD got 74.9.  What happened to us?  The formula for calculating the number seems to be a secret, so we can’t replicate the calculation. Just accept it. We got worse between 2011 and 2012.

On a different matter, there is an interesting post by Phil Davis on the relation between impact factors and citations. There is none. [Essentially none, in my opinion.]  I made an earlier post about this, and gave links to other writings.