Replicating Austerity Results

An interesting story came to light this week when a student tried to replicate the results in a paper. He found that he wasn’t able to.

I wrote something before about the fundamental principle of the scientific method – that the results of an experiment should be capable of replication and verification by others. If you can’t do that, it’s not science. What happened here is that a graduate class was given an assignment: pick a paper and replicate the results. An excellent assignment for graduate students, in my opinion, and a valuable contribution to the scientific community. This particular student was unable to replicate the results in his chosen paper. It turns out that he found an error. The authors made a mistake.

First and foremost, this shows the scientific method and the scientific community at work. If the system works, papers with errors should be found out, and the errors corrected, and science as a whole moves on. Collectively we learn something. It’s a good example of that.

The authors are 100% responsible for the error. It was a bad error, and it’s not good news for the reputations of the authors. It makes them look bad. Other papers by these authors will now be questioned.

A second point is that scientific papers that are published are supposed to be checked by reviewers/referees. That’s part of the process. Why didn’t the peer review process catch the error in this case? Because it wasn’t peer reviewed. The paper was presented at a conference that does not referee the papers.

This student picked a paper that is being quoted a lot by politicians and economists that are in favour of “austerity measures.” Apparently it has been used as an argument in favour of austerity. One cannot blame the authors of the paper for that (unless there are things going on behind the scenes that I am not aware of, as has been alleged on reddit for example). If anyone is going to use the findings of a paper, they are responsible for checking the paper first. If you don’t check the results, especially the results of a paper that has not been peer-reviewed, and there is a mistake, you cannot blame the authors for your consequent errors. You are responsible for your own papers.

The paper in question is available here. The research was funded by the U.S. National Science Foundation.


One thought on “Replicating Austerity Results

  1. The episode also illustrates the value of making source code and data freely available online. If any halfway interested researcher can rerun your code in 30 minutes, mistakes like this would be spotted much sooner. In this case, the spreadsheet containing the coding error was offline.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s