I was invited recently by email to vote in the QS world rankings of universities. Part of the QS rankings (40%) goes for something called “academic reputation” and I suppose my vote/opinion counts towards that. My main problem with rankings in general is that sometimes people use them for things that they are not supposed to be used for. For example, those in power might use the rankings to make policy decisions. Further, it is never really disclosed how the rankings are arrived at. Yes, there are some vague explanations, but it is never possible to replicate the results. Given all that, should I vote?
There have been many posts recently about world rankings. One describes some impacts of the world rankings on higher education and its quality and regulation. Another is called the ultimate absurdity of college rankings. A third by Richard Holmes gives some reasons why rankings are unreliable, especially the subject rankings.
Anyway, to be more specific about my own dilemma, should I give my opinion about other universities? I have a problem because I don’t really know enough. I have to name up to 30 (foreign) universities that produce the best research in the natural sciences, as well as 10 in my own country (Ireland). Sure, we can all name Caltech and so on at the top. After the top ten or so, I run out of names. So what do I do? Who should I name? How do I decide if one university is better than another? Can I compare, for example, Kyoto University with the University of New South Wales? To be honest, I know they are both very good with top class reputations, but I couldn’t rank one above the other. (Kyoto was 35 and NSW was 52 in the overall QS rankings in 2012, by the way.) To be honest, I probably wouldn’t name either of them in my 30, but that’s just because I don’t know them and I don’t know anyone there. The same goes for hundreds of other universities.
So what sort of a picture does my response, and the responses of other academics like me, actually give? The number of votes a university gets is probably proportional to how many foreigners know someone who works there. The top ten will remain the top ten, because everyone automatically assumes they are great and everyone has heard of them. But when you move down the list to around number 50, and below, how precise can we be? Some volatility is to be expected, and indeed happens. (Why volatility happens is not always clear, as discussed here)
The data for “reputation” is gathered in such a random fashion that it cannot be meaningful.
The president of University College Cork wrote an email to all staff in May 2011, and got into hot water. The full text is here. QS now says that they use “sophisticated anomaly detection algorithms” among other things, to stop academics asking their friends to vote for them.
It was recently exposed that the QS rankings used a site that pays people to fill out surveys. There is an excellent article about this here by Elizabeth Redden. Indeed, I was offered 300 pounds in credits to complete my survey. After I did so, out of curiosity, I tried to buy QS reports with my credits. The website didn’t work, I got an error message and the credits were worthless.
i wonder if anyone has actually done a survey to see how many students actually use rankings when deciding on a university.