Following the news that the first Planethunters paper has been submitted, we thought we’d write a little bit about what peer review is and what it does. This is relevant not just in charting out the future of our first paper, but also in the wider discussion of scientific results in the media. This post is an adapted version of a post I wrote for the first Galaxy Zoo paper that was submitted in 2008.
What’s scientific publishing all about? How does it work? If you’ve followed the blog and the forum, you have a pretty good idea of the first part of the scientific process: discovery! We figure out something new about the universe and how it works.
This is one of the amazing and unique things about science. Good scientists spend most of their time arguing against the effects they see in their own data, to avoid falling into traps of seeing only what they expect to see. To see how unique and amazing this is, try to imagine a politician arguing against a piece of legislation s/he is sponsoring! This process of double, triple, and quadruple-checking one’s own work is a very important part of science.
Once we were convinced that we really understood what is going on, we could then write up our conclusions in the form of a scientific paper. Over the past few weeks, the Planethunters team, together with colleagues from the Kepler team, wrote up a paper describing the first results. The paper was passed back and forth between people who made edits and comments and the paper thus passed the first through the first check — our own examination of our results.
e next step in scientific research is to submit the paper to a journal. This has now happened, and the paper Fischer et al. (2011) (where “et al.” means “and the rest,” including YOU!!) has been submitted to the top UK journal Monthly Notices of the Royal Astronomical Society (MNRAS).
The editor of this journal will now select an anonymous referee who can comment on the scientific and technical merits of the paper. The referee is another astronomer or cosmologist whom the editor can ask for an expert assessment of the work. He or she will have a few weeks to read it, think about it, and then make a number of recommendations to the editor of the journal. There are three options. The referee can reject the paper outright. This generally happens very rarely, except in highly competitive top journals like Nature and Science. They can support publication of the paper, asking for only a few minor modifications. This also happens quite rarely, though! The most common outcome is for her to write a “referee report,” suggesting a number of modifications and ask for clarifications. The referee might have questions about some part of the analysis, suggest some alternative thoughts and ideas, or criticise the methodology. Sometimes referees can be hostile to a paper; but often, they are genuinely helpful and constructive.
After receiving the report, we get a few weeks to digest it and modify the paper according to the referee’s comments, and argue against the points raised that we disagree with. This process may repeat itself a number of times if the referee isn’t happy with our modifications, and so it can often take weeks and months for a paper to get to a decision by the editor (acceptance or rejection). If a referee is being particularly unreasonable, we can write to the editor requesting a new referee. In extreme circumstances, we could even choose to submit the paper to a different journal and hope for a more reasonable referee.
The whole process is generally known as peer review since the referee is a peer — a fellow scientist and expert in the field. If the paper is accepted, it will appear both in the online and print version of the journal after another few weeks or months. A paper accepted in such a journal is then considered peer-reviewed.
It’s important to note that something said in a “peer-reviewed” paper isn’t necessarily true. The point of peer-review is to weed out obviously flawed paper whose logic has holes or whose data don’t support the conclusion. Knowing that a paper has been peer-reviewed should give you extra confidence that its results are believable – that means that an expert in the field has read through the paper and thinks its conclusions are believable.It’s really just the first step of proper “peer-review,” because the process continues.
As the community of astrophysicists digests the paper, they too pass judgement on whether they consider the paper important and whether they believe the conclusion. Thus, in the years after publication, other astrophysicists might deem Fischer et al. (2011) a key paper and cite it in the future, commenting on it positively. Or they might disagree with it, but that would still be a sign that it was important enough to comment on. Or it might just fade into obscurity if astronomers don’t consider it important. That’s the historical legacy of a paper – and that’s the ultimate peer-review.