Impact Factor – what it should and shouldn’t be used for

There was a good editorial in Nature Materials that clarified a few things for me about the impact factor.  They made the point that the impact factor of a journal in conjunction with the median does tell you something about the journal.  It does not tell you something about an individual person, or an individual paper. It should not be used for grant-giving, tenure, appointment or promotion.

There was another editorial in Nature on this topic in 2005.

And again in 2003.  This one comments on the fact that most people just copy references from another paper. I have definitely observed this. The rich get richer, and the poor get poorer, when it comes to citations. You have to get a paper in the loop, and then sit back and watch the citations pile up.

Unlike impact factor, citations do tell you something about an individual paper, after a suitable period of time has elapsed. Some people say that the only way to tell if a paper is a good paper is to read it yourself. I disagree. First of all, that doesn’t work if the field is not my field and I am not qualified to judge. Secondly, my opinion is just one person’s opinion, whereas if I look at the number of citations, I am getting the opinion of all the other researchers in that field in the whole world (in some sense). It would of course be better to pick up the phone and ask all the other people in that field individually what their opinion is, but that is not practical. I think the number of citations is a compromise, it’s not perfect because there are different reasons a paper might be cited, but it’s better than nothing.

There’s a related blog here, about the REF in the UK. The author makes the point that averaging the h-index over a department seems to be a reasonable measure. Another thing I learned here is that the impact factor will not be used in the REF in 2014.

One of the comments makes the interesting point that once we start using a metric to make our decisions, this metric ceases to have any value because people will start playing games to manipulate the metric. One way around this is to keep changing the metric.

Eggs in one basket in the university sector

I wrote an earlier post about the idea of a country putting all its research funding (and perhaps other resources too) into a few top places. I think this is a very interesting topic for debate.

Recently there was another article on the matter in The Guardian. The former president of DCU posted about it.

One of the arguments in favour of funding only the top universities is that “the money is going there anyway.” The Guardian cites that 75% is going to the top 30 institutions, and things like that. Therefore it is seemingly correct to say that most of the funding goes to the top universities. This is probably because most of the top researchers are at the top universities. But the word “most” is the key point to me. The point is that there are some excellent researchers at universities that are not ranked highly. Those researchers account for the other 25% of the funding. If you take away that 25% from them, you are saying to them that they either need to move to a top ranked university, or stop getting funded. Presumably this will result in the movement of those people to the top ranked universities. This cements the two-tier world, cements the position of the top ranked universities, and makes it impossible for other universities to move up there. In the long term we will get a polarized situation where only the top ranked places do funded research. We will move from a 75-25 split to a 100-0 split.

Next question: is that good or bad? It depends on your point of view, and what you are trying to achieve.

Next consider a similar scenario where a country puts all its funding into a few selected areas/subjects, instead of a few selected universities. The same story plays out. Excellent researchers in the non-chosen areas do not get funded. They have a choice, either move research area, move country, or stop getting funded.

Next question: is that good or bad? It depends on your point of view, and what you are trying to achieve.

There is also an indirect way of putting all eggs in one basket – which could be happening right now in the UK. It is interesting to observe what is happening there – there was an article in the Guardian the other day about it. The upshot of raising the limit on fees to 9,000 pounds is that the top universities are thriving, and weaker ones are possibly struggling. Applications have dropped, so the weaker universities have to admit students with bad grades. Furthermore, they have to pass these students all the way through, because they need the money. Talk about buying a degree. It will take some years to see how this plays out.

They say all things are cyclical. I think this will probably happen here – we will put all our eggs in one basket for a while, then diversify, then go back, then diversify again, etc. As new people come into power, they need to make changes. Nobody gets noticed and rewarded for saying it ain’t broke, so I’m not going to fix it.

Taiwan Research Rankings

Through ninth level Ireland I saw a post by Richard Holmes on the Taiwan rankings. These are university rankings just for research, and just for science and engineering.

Here is how they compute their rankings, which are based on the Thomson-Reuters (formerly ISI) databases.

  • 25%  to research productivity (number of articles over the last 11 years, number of articles in the current year),
  • 35% to research impact (number of citations over the last 11 years, number of citations in the current year, average number of citations over the last 11 years)
  • 40% to research excellence (h-index over the last 2 years, number of highly cited papers, number of articles in the current year in highly cited journals).

Some of the measures seem to be *absolute* numbers, like the total number of articles over the last 11 years, and not relative numbers. This  favours larger universities. Also, arts and humanities are not counted.

I looked up the Irish universities.

235 Trinity College Dublin

277 University College Dublin

311 Queen’s University Belfast

398 University College Cork

No others in top 500.

I find it interesting that University of London, Royal Holloway comes outside the top 500, and on the other hand is number 11 in the world for research impact according to the THE world rankings. Why is there such a big difference?

Balance between basic and applied research

This is the (slightly expanded) text of a speech I gave at the NDRC Long Debate, 25 October 2012.

 

What is the right balance between basic and applied research for a small island nation?

Basic research is research that is not directed towards any particular application or development. There is an old joke: Basic research is like shooting an arrow into the air and, where it lands, you paint a target.

Applied research is defined by strategic goals set by industrial or political leaders, or driven by market needs.

One can also say that applied research is where a practical application is a number of months away. Basic research is when a practical application is not yet visible.

I first would like to convince you that we should have some basic research. That is, we should not have 100% applied and 0% basic. I think most people would already go along with that, but let me make a few points in favour of basic research.

Funding of basic research is mostly done by funding a PhD student, who is then trained as a researcher.

One important result of funding PhD students is the creation of human and intellectual capital. Building research capacity and generating employees capable of inquiry is important.  We cannot predict what will be the leading technology in 10 years time, but we will need a constant supply of researchers strong in the fundamentals to stay in the game.

It used to be thought that basic research was followed by applications. In other words, a basic discovery was made, such as the invention of the laser, and later on applications were found. Certainly, this happens. Example: the CD.  First the laser was invented, for no reason. Then Philips wanted to make a disc, and someone knew about lasers, and said hey, maybe we could use this thing called a laser?  Then they wanted to correct errors due to scratches. Someone said hey, there’s this maths from the 19th century that can be used to construct these codes.

However we now understand that, very often, basic and applied research progress together, and in fact they feed off each other. For example, an applied cryptographer might design a cipher system that appears secure. The theoretical cryptographer might do an analysis and pinpoint a flaw, and then suggest a correction. The applied cryptographer might point out that then the system is too slow, but suggest another change.  And so on, back and forth, until a final product is reached. Basic and applied research happen together.

There are many examples of basic research discovering something useless, and for it later to find an application. The laser, the structure of DNA, the internet, the transistor, the sex life of the screwworm, are some. A more recent one is graphene, discovered by basic research in 2004, where we still do not know how it will revolutionize the world.

Did you know… Einstein’s theory of relativity is needed to make GPS clocks accurate.

Other arguments:  Basic research can have wider ramifications than targeted applied research. Research aimed at Alzheimers could improve Alzheimers, however a discovery in fundamental drug research could improve several conditions all at the same time.

Basic research is recognized as a longterm generator of innovation.

A preference for applied research favours certain subjects while neglecting others. Subjects like mathematics, and in particular the arts and humanities, become disadvantaged and will suffer.

I could go on, but I hope you accept the case that at least some basic research should be funded by the state.

By the way I am not saying that all basic researchers and scientists should be given funding. I am saying that our excellent scientists should.  It’s important to let scientists decide which science is worthy of being funded, through a peer review process.

Now the next question is, how much of the state-sponsored research should be basic?  And should it be directed?

The international norm seems to be around 50-50 between basic and applied research.

The Irish government funds research through agencies such as Enterprise Ireland, Science Foundation Ireland, the Irish Research Council (now just one), the Health Research Board, Teagasc, the Marine Institute.

This year has seen a shift towards funding research in certain areas of science, and funding only research that has `impact’ on the economy.

The Research Prioritisation Steering Group recommended 14 areas that the government should direct research funding towards.  The chair of the RPSG, Jim O’Hara, suggested that about 80% of funding go to these priority areas, with the other 20% going towards more fundamental research (The Irish Times, 2 March 2012).  Interestingly, the RPSG itself could not agree on the percentage, I have been told by a member.

Subsequent to the RPSG recommendations, the new SFI policy is not an 80-20 split but a 100-0 split. This puts all research funding into the priority areas and areas with impact on the economy, and absolutely nothing into research in other areas.

We don’t yet know if the other agencies such as the IRC will put all their money into the new priority areas. If this happens, it is demoralizing for researchers in other areas. This has a knock-on effects in the university environment, where such researchers are also teachers. Many young people are motivated by a great teacher.  Research-informed teaching is a crucial part of our university education. Dropping some subjects will affect future generations.

Another problem is that directing research funding into certain areas is a high-risk strategy. Many Silicon Valley companies are known for not directing the research and just letting it happen. How do you get the most out of really bright people? You put them in a room and say “do what you want. We would prefer if it helps the company.” Google and Microsoft research work this way. Researchers will create, when given the chance. You just don’t know exactly what they will create.

The Economist wrote in 2010 about this strategy of Picking Winners. It is back in fashion among governments around the world, however it mixes science policy with industrial policy. It fails more often than it succeeds, according to the Economist. Other arguments against picking winners (research prioritization) are

– You can’t predict what will be growth areas over the next ten years.
– We lose expertise in the non-priority areas, top people will move to another country, nobody in those areas wants to come here.
– Companies in non-priority areas will not come to Ireland.
– If there is a breakthrough development in a non-priority area, we will not be able to capitalise because we have lost our expertise.
I did some research while preparing for tonight, and I looked at countries of a similar size to Ireland.

Typically the leading countries in science spend about 3% of GDP on R&D. In 2009 in Ireland it was 1.77%.

The Swedish Research Council was founded in 2001, the same year as SFI. They put about 3.6% of GDP into research. In their proposal for 2009-2013 they recommended an increase of 270 million in free basic research, and 200 million in basic research in areas of high priority. (So increasing the split in favour of basic.)

I found a 2007 report from Austria (2.75% of GDP into R&D), with an analysis on the competitiveness of their research. They looked at other countries and said:

“With very few exceptions, all leading scientific nations, in particular smaller ones such as Switzerland (3%), Israel (4.5%), Sweden (3.6%), Denmark (3%), Finland (3.8%) or Holland (1.8%), are world-leading not only overall but also in all individual scientific disciplines.

This provides a strong indication for a wide-ranging effort to attain international quality in all areas of science and argues (also for countries with smaller economies) against too strong a focus on particular disciplines. It appears to be the case that excellence in individual disciplines or fields of research is hardly possible without excellence in most disciplines. The jump to a world-leading position in basic research cannot be attained in a matter of a few years. Many of today’s top nations have invested in the necessary resources, structures and incentives over a period of decades.”

Israel is slightly bigger than Ireland, and puts 4.5% of GDP into R&D (highest in the world). Michael Sela former Director of Weizman institute said:

“Israel has managed to establish an excellent scientific base that makes it an important exporter of high technologies, most notably software and biotechnology. Israel has succeeded in doing so because of the way it has been concentrating resources in science and technology on basic research as well as its application in order to strengthen its economic base.”

“For a small country, we cannot be excellent at everything. But we have no right to make compromises in our effort to strive for excellence. Such a vision includes virtually complete academic freedom and the gathering of first-rate minds, which are then left alone to ripen at will.”

“But no matter what a national science policy looks like, it must not put any constraints on the scientists but rather give them the necessary freedom and means for them to carry out their work. As they are the specialists, they know better than any politician or executive which avenues to explore and which discoveries could be developed into new products.”

“At the Weizmann Institute, our philosophy is ‘Research for its own sake’, but whenever results from this free research have potential in the marketplace, we aim to pursue this energetically, preferably by our own industry.”

What do I propose for Ireland?

A combination of funding to priority areas and funding to small basic research projects across all areas.

Some money, reaching critical mass, should be given to excellent researchers, from any subject. About 25K per year will support one PhD student. One million per year for four years will support forty different research projects. Some of those will pay off. (we can’t predict which ones.)

Partnerships between a basic researcher and an applied researcher/developer should be supported. These two people could meet and discuss back and forth how to translate an idea towards the marketplace.  I have the impression that businesses, if asked what is the role of a university, say that it is NOT the patenting and spin-off, but the informal contacts, publications, conferences and joint R&D.

All this abstract policy talk is well and good, but it’s also about people. We have in Ireland, right now, some excellent people, in whatever subject and for whatever reason. Do we want to support the research of those people? If not, they need to know.

University Funding

It’s a thorny question: how should universities be funded?

Another way of asking the question is: who should pay?

In order to answer this we might ask: who benefits? (cui bono) The argument is that whoever benefits should pay.

Following this line of thought, some would argue, it follows that the students should pay. Because, studies show that the average salary of a university graduate is higher than the average salary of a non-graduate.  So it seems that yes, the students do benefit (in purely monetary terms anyway). This means that students should pay, and that’s why we have fees.

But doesn’t society as a whole benefit as well? One could argue that, for example, health care has improved and everyone in our society is living longer because of medical schools in universities. So it also seems that society as a whole benefits. This means that society should pay, and that’s why the government gives money collected in taxes to the universities.

Developing this point, the DCU students will vote on five different ways to fund universities. These are

– means tested tuition fees,

– a graduate tax,

– a student loan scheme,

– a student contribution fee of 3,000 euro,

– a fully free system funded by conventional taxation.

In the first option, the student pays if they come from a family with ‘high’ income, and they don’t pay if they come from a family with ‘low’ income. In the second option graduates pay after they graduate. This option was supported today by the ESRI. They propose that if the income of a graduate reaches a certain level, they pay an extra tax for a number of years. This is called the Income Contingent Loan scheme. (Another variation is to pay this graduate tax for their lifetime.) The third option, a student loan, usually goes along with fairly high tuition fees, and students need a loan to pay them. They get the loan from the government, at a good interest rate, and repay it after graduation. The fourth option is where all students pay something, while they are students. The last option is where all of society pays. Well, those who pay taxes anyway.

I am ignoring the issue of precisely how much money is paid in each of these. These details need to be worked out, and I haven’t done it.

In some countries there is a combination of the state paying and the students paying. Here is the government contribution to UCD for the last five years:

2008   186,406,034 euro

2009   175,580,384 euro

2010   152,618,765 euro

2011   140,918,644 euro

2012   121,731,537 euro

[Added: over the same period, full-time student numbers increased nationally by 17%.]

This shows that the government/taxpayers contribution is decreasing every year. The rate of decrease is quite alarming. Normally this decrease would be compensated for by an increase in fees. In other words, we trade society paying for the students paying. In Ireland the universities cannot do this because the universities are not allowed by law to charge tuition fees to undergraduates. There is a fee, called the student contribution charge, whose rate is set by the universities subject to a legal maximum. Currently the maximum fee allowed is 2,250 euro. Some or all of this fee can be covered by a grant if the student comes from a family with low income.

To give a full answer, one also has to answer the following question: what is the purpose of a university? A question for another day.

Citation indicator in world rankings

An interesting post by Richard Holmes about the THE university world rankings, and why the citation part of the rankings is not yet reliable. I just noticed that he withdrew the post and replaced it with an apology.

The world top 20 in the THE citation indicator rankings are

1.   Moscow (State) Engineering Physics Institute (MEPhI)
1.   Rice University
3.   University of California Santa Cruz
3.   MIT
5.   Princeton
6.   Caltech
7.   University of California Santa Barbara
7.   Stanford
9.   University of California Berkeley
10.  Harvard
11.  Royal Holloway London
12.  Chicago
13.  Northwestern
14.  Tokyo Metropolitan University
14.  University of Colorado Boulder
16.  University of Washington Seattle
16.  Duke
18.  University of California San Diego
18.  University of Pennsylvania
18.  Cambridge

The list looks slightly odd to me. I would like to know how these numbers are calculated, but I can’t find the information anywhere. It is apparently calculated by Thomson-Reuters, who use their large database of citation data to compute this number.  I would like to calculate it for my own university UCD, but I don’t know how. In 2011 UCD got 80.5 and in 2012 UCD got 74.9.  What happened to us?  The formula for calculating the number seems to be a secret, so we can’t replicate the calculation. Just accept it. We got worse between 2011 and 2012.

On a different matter, there is an interesting post by Phil Davis on the relation between impact factors and citations. There is none. [Essentially none, in my opinion.]  I made an earlier post about this, and gave links to other writings.

In science we trust, just have a little faith

Since the US election result there have been many articles about how Nate Silver got it right. It has been triumphed as a victory for science and Bayesian statistics. There’s only one problem – it’s not quite science! Or at least, we don’t know if it’s science or not.

Before I go on, I am the first to say congratulations to Nate Silver and Sam Wang and all others who use polls and statistical modelling to give a prediction (with probability) of an election outcome.  I think it’s great.

The reason it’s not science is that we don’t know how Nate Silver makes his predictions. The methods he uses are proprietary and are kept secret. Everybody seems to be assuming that he used some kind of statistical modelling methods to predict the outcome. This may well be true, he probably did, but we don’t know. For this reason it cannot be called science. The scientific method contains experiments where all the methods and conditions are made public, so that other scientists can attempt to replicate the experiment if they want to. In the case of computer usage, the source code is made public. The scientific community doesn’t accept what other scientists say unless it has been confirmed by other scientists. The cold fusion debacle is a good example.

In this case, we cannot replicate Nate Silver’s predictions because he keeps the model secret. True, there is an explanation of sorts here. He assigns weightings to various things. We don’t know the precise weights though. Perhaps he assigns weightings based on the star signs of the election candidates. Maybe he’s just really good at guessing. Who knows. Some people can guess at how he might have done it, and those guesses may well be right, but we don’t know. We just have to have faith.