There was a good editorial in Nature Materials that clarified a few things for me about the impact factor. They made the point that the impact factor of a journal in conjunction with the median does tell you something about the journal. It does not tell you something about an individual person, or an individual paper. It should not be used for grant-giving, tenure, appointment or promotion.
There was another editorial in Nature on this topic in 2005.
And again in 2003. This one comments on the fact that most people just copy references from another paper. I have definitely observed this. The rich get richer, and the poor get poorer, when it comes to citations. You have to get a paper in the loop, and then sit back and watch the citations pile up.
Unlike impact factor, citations do tell you something about an individual paper, after a suitable period of time has elapsed. Some people say that the only way to tell if a paper is a good paper is to read it yourself. I disagree. First of all, that doesn’t work if the field is not my field and I am not qualified to judge. Secondly, my opinion is just one person’s opinion, whereas if I look at the number of citations, I am getting the opinion of all the other researchers in that field in the whole world (in some sense). It would of course be better to pick up the phone and ask all the other people in that field individually what their opinion is, but that is not practical. I think the number of citations is a compromise, it’s not perfect because there are different reasons a paper might be cited, but it’s better than nothing.
There’s a related blog here, about the REF in the UK. The author makes the point that averaging the h-index over a department seems to be a reasonable measure. Another thing I learned here is that the impact factor will not be used in the REF in 2014.
One of the comments makes the interesting point that once we start using a metric to make our decisions, this metric ceases to have any value because people will start playing games to manipulate the metric. One way around this is to keep changing the metric.