ads

How to review a paper



As junior scientists develop their expertise and make names for themselves, they are increasingly likely to receive invitations to review research manuscripts. It’s an important skill and service to the scientific community, but the learning curve can be particularly steep. Writing a good review requires expertise in the field, an intimate knowledge of research methods, a critical mind, the ability to give fair and constructive feedback, and sensitivity to the feelings of authors on the receiving end. As a range of institutions and organizations around the world celebrate the essential role of peer review in upholding the quality of published research this week, Science Careers shares collected insights and advice about how to review papers from researchers across the spectrum. The responses have been edited for clarity and brevity.


someone correcting a paper

What do you consider when deciding whether to accept an invitation to review a paper?
I consider four factors: whether I'm sufficiently knowledgeable about the topic to offer an intelligent assessment, how interesting I find the research topic, whether I’m free of any conflict of interest, and whether I have the time. If the answer to all four questions is yes, then I’ll usually agree to review.
- Chris Chambers, professor of cognitive neuroscience at Cardiff University in the United Kingdom

I am very open-minded when it comes to accepting invitations to review. I see it as a tit-for-tat duty: Since I am an active researcher and I submit papers, hoping for really helpful, constructive comments, it just makes sense that I do the same for others. So accepting an invitation for me is the default, unless a paper is really far from my expertise or my workload doesn’t allow it. The only other factor I pay attention to is the scientific integrity of the journal. I would not want to review for a journal that does not offer an unbiased review process.
- Eva Selenko, senior lecturer in work psychology at Loughborough University in the United Kingdom

I'm more prone to agree to do a review if it involves a system or method in which I have a particular expertise. And I'm not going to take on a paper to review unless I have the time. For every manuscript of my own that I submit to a journal, I review at least a few papers, so I give back to the system plenty. I've heard from some reviewers that they're more likely to accept an invitation to review from a more prestigious journal and don't feel as bad about rejecting invitations from more specialized journals. That makes things a lot harder for editors of the less prestigious journals, and that's why I am more inclined to take on reviews from them. If I've never heard of the authors, and particularly if they're from a less developed nation, then I'm also more likely to accept the invitation. I do this because editors might have a harder time landing reviewers for these papers too, and because people who aren't deeply connected into our research community also deserve quality feedback. Finally, I am more inclined to review for journals with double-blind reviewing practices and journals that are run by academic societies, because those are both things that I want to support and encourage.
- Terry McGlynn, professor of biology at California State University, Dominguez Hills

I usually consider first the relevance to my own expertise. I will turn down requests if the paper is too far removed from my own research areas, since I may not be able to provide an informed review. Having said that, I tend to define my expertise fairly broadly for reviewing purposes. I also consider the journal. I am more willing to review for journals that I read or publish in. Before I became an editor, I used to be fairly eclectic in the journals I reviewed for, but now I tend to be more discerning, since my editing duties take up much of my reviewing time.
- John P. Walsh, professor of public policy at the Georgia Institute of Technology in Atlanta

Once you’ve agreed to complete a review, how do you approach the paper?
Unless it’s for a journal I know well, the first thing I do is check what format the journal prefers the review to be in. Some journals have structured review criteria; others just ask for general and specific comments. Knowing this in advance helps save time later.

I almost never print out papers for review; I prefer to work with the electronic version. I always read the paper sequentially, from start to finish, making comments on the PDF as I go along. I look for specific indicators of research quality, asking myself questions such as: Are the background literature and study rationale clearly articulated? Do the hypotheses follow logically from previous work? Are the methods robust and well controlled? Are the reported analyses appropriate? (I usually pay close attention to the use—and misuse—of frequentist statistics.) Is the presentation of results clear and accessible? To what extent does the Discussion place the findings in a wider context and achieve a balance between interpretation and useful speculation versus tedious waffling?
- Chambers

I subconsciously follow a checklist. First, is it well written? That usually becomes apparent by the Methods section. (Then, throughout, if what I am reading is only partly comprehensible, I do not spend a lot of energy trying to make sense of it, but in my review I will relay the ambiguities to the author.) I should also have a good idea of the hypothesis and context within the first few pages, and it matters whether the hypothesis makes sense or is interesting. Then I read the Methods section very carefully. I do not focus so much on the statistics—a quality journal should have professional statistics review for any accepted manuscript—but I consider all the other logistics of study design where it’s easy to hide a fatal flaw. Mostly I am concerned with credibility: Could this methodology have answered their question? Then I look at how convincing the results are and how careful the description is. Sloppiness anywhere makes me worry. The parts of the Discussion I focus on most are context and whether the authors make claims that overreach the data. This is done all the time, to varying degrees. I want statements of fact, not opinion or speculation, backed up by data.
- Michael Callaham, emergency care physician and researcher at the University of California, San Francisco

Most journals don't have special instructions, so I just read the paper, usually starting with the Abstract, looking at the figures, and then reading the paper in a linear fashion. I read the digital version with an open word processing file, keeping a list of “major items” and “minor items” and making notes as I go. There are a few aspects that I make sure to address, though I cover a lot more ground as well. First, I consider how the question being addressed fits into the current status of our knowledge. Second, I ponder how well the work that was conducted actually addresses the central question posed in the paper. (In my field, authors are under pressure to broadly sell their work, and it's my job as a reviewer to address the validity of such claims.) Third, I make sure that the design of the methods and analyses are appropriate.
- McGlynn

First, I read a printed version to get an overall impression. What is the paper about? How is it structured? I also pay attention to the schemes and figures; if they are well designed and organized, then in most cases the entire paper has also been carefully thought out.

When diving in deeper, first I try to assess whether all the important papers are cited in the references, as that also often correlates with the quality of the manuscript itself. Then, right in the Introduction, you can often recognize whether the authors considered the full context of their topic. After that, I check whether all the experiments and data make sense, paying particular attention to whether the authors carefully designed and performed the experiments and whether they analyzed and interpreted the results in a comprehensible way. It is also very important that the authors guide you through the whole article and explain every table, every figure, and every scheme.

As I go along, I use a highlighter and other pens, so the manuscript is usually colorful after I read it. Besides that, I make notes on an extra sheet.
- Melanie Kim Müller, doctoral candidate in organic chemistry at the Technical University of Kaiserslautern in Germany

I first familiarize myself with the manuscript and read relevant snippets of the literature to make sure that the manuscript is coherent with the larger scientific domain. Then I scrutinize it section by section, noting if there are any missing links in the story and if certain points are under- or overrepresented. I also scout for inconsistencies in the portrayal of facts and observations, assess whether the exact technical specifications of the study materials and equipment are described, consider the adequacy of the sample size and the quality of the figures, and assess whether the findings in the main manuscript are aptly supplemented by the supplementary section and whether the authors have followed the journal’s submission guidelines.
- Chaitanya Giri, postdoctoral research fellow at the Earth-Life Science Institute in Tokyo

I print out the paper, as I find it easier to make comments on the printed pages than on an electronic reader. I read the manuscript very carefully the first time, trying to follow the authors’ argument and predict what the next step could be. At this first stage, I try to be as open-minded as I can. I don’t have a formalized checklist, but there are a number of questions that I generally use. Does the theoretical argument make sense? Does it contribute to our knowledge, or is it old wine in new bottles? Is there an angle the authors have overlooked? This often requires doing some background reading, sometimes including some of the cited literature, about the theory presented in the manuscript.

I then delve into the Methods and Results sections. Are the methods suitable to investigate the research question and test the hypotheses? Would there have been a better way to test these hypotheses or to analyze these results? Is the statistical analysis sound and justified? Could I replicate the results using the information in the Methods and the description of the analysis? I even selectively check individual numbers to see whether they are statistically plausible. I also carefully look at the explanation of the results and whether the conclusions the authors draw are justified and connected with the broader argument made in the paper. If there are any aspects of the manuscript that I am not familiar with, I try to read up on those topics or consult other colleagues.
- Selenko

I spend a fair amount of time looking at the figures. In addition to considering their overall quality, sometimes figures raise questions about the methods used to collect or analyze the data, or they fail to support a finding reported in the paper and warrant further clarification. I also want to know whether the authors’ conclusions are adequately supported by the results. Conclusions that are overstated or out of sync with the findings will adversely impact my review and recommendations.
- Dana Boatman-Reich, professor of neurology and otolaryngology at Johns Hopkins University School of Medicine in Baltimore, Maryland

I generally read on the computer and start with the Abstract to get an initial impression. Then I read the paper as a whole, thoroughly and from beginning to end, taking notes as I read. For me, the first question is this: Is the research sound? And secondly, how can it be improved? Basically, I am looking to see if the research question is well motivated; if the data are sound; if the analyses are technically correct; and, most importantly, if the findings support the claims made in the paper.
- Walsh

The main aspects I consider are the novelty of the article and its impact on the field. I always ask myself what makes this paper relevant and what new advance or contribution the paper represents. Then I follow a routine that will help me evaluate this. First, I check the authors’ publication records in PubMed to get a feel for their expertise in the field. I also consider whether the article contains a good Introduction and description of the state of the art, as that indirectly shows whether the authors have a good knowledge of the field. Second, I pay attention to the results and whether they have been compared with other similar published studies. Third, I consider whether the results or the proposed methodology have some potential broader applicability or relevance, because in my opinion this is important. Finally, I evaluate whether the methodology used is appropriate. If the authors have presented a new tool or software, I will test it in detail.
- Fátima Al-Shahrour, head of the Translational Bioinformatics Unit in the clinical research program at the Spanish National Cancer Research Centre in Madrid

How do you go about drafting the review? Do you sign it?
Using a copy of the manuscript that I first marked up with any questions that I had, I write a brief summary of what the paper is about and what I feel about its solidity. Then I run through the specific points I raised in my summary in more detail, in the order they appeared in the paper, providing page and paragraph numbers for most. Finally comes a list of really minor stuff, which I try to keep to a minimum. I then typically go through my first draft looking at the marked-up manuscript again to make sure I didn’t leave out anything important. If I feel there is some good material in the paper but it needs a lot of work, I will write a pretty long and specific review pointing out what the authors need to do. If the paper has horrendous difficulties or a confused concept, I will specify that but will not do a lot of work to try to suggest fixes for every flaw.

I never use value judgments or value-laden adjectives. Nothing is “lousy” or “stupid,” and nobody is “incompetent.” However, as an author your data might be incomplete, or you may have overlooked a huge contradiction in your results, or you may have made major errors in the study design. That’s what I communicate, with a way to fix it if a feasible one comes to mind. Hopefully, this will be used to make the manuscript better rather than to shame anyone. Overall, I want to achieve an evaluation of the study that is fair, objective, and complete enough to convince both the editor and the authors that I know something about what I’m talking about. I also try to cite a specific factual reason or some evidence for any major criticisms or suggestions that I make. After all, even though you were selected as an expert, for each review the editor has to decide how much they believe in your assessment.
- Callaham

I use annotations that I made in the PDF to start writing my review; that way I never forget to mention something that occurred to me while reading the paper. Unless the journal uses a structured review format, I usually begin my review with a general statement of my understanding of the paper and what it claims, followed by a paragraph offering an overall assessment. Then I make specific comments on each section, listing the major questions or concerns. Depending on how much time I have, I sometimes also end with a section of minor comments. I may, for example, highlight an obvious typo or grammatical error, though I don’t pay a lot of attention to these, as it is the authors’ and copyeditors’ responsibility to ensure clear writing.

I try to be as constructive as possible. A review is primarily for the benefit of the editor, to help them reach a decision about whether to publish or not, but I try to make my reviews useful for the authors as well. I always write my reviews as though I am talking to the scientists in person. I try hard to avoid rude or disparaging remarks. The review process is brutal enough scientifically without reviewers making it worse.

Since obtaining tenure, I always sign my reviews. I believe it improves the transparency of the review process, and it also helps me police the quality of my own assessments by making me personally accountable.
- Chambers

I want to help the authors improve their manuscript and to assist the editor in the decision process by providing a neutral and balanced review of the manuscript’s strengths and weaknesses and how to potentially improve it. After I have finished reading the manuscript, I let it sink in for a day or so and then I try to decide which aspects really matter. This helps me to distinguish between major and minor issues and also to group them thematically as I draft my review. My reviews usually start out with a short summary and a highlight of the strengths of the manuscript before briefly listing the weaknesses that I believe should be addressed. I try to link any criticism I have either to a page number or a quotation from the manuscript to ensure that my argument is understood. I also selectively refer to others’ work or statistical tests to substantiate why I think something should be done differently.

I try to be constructive by suggesting ways to improve the problematic aspects, if that is possible, and also try to hit a calm and friendly but also neutral and objective tone. This is not always easy, especially if I discover what I think is a serious flaw in the manuscript. However, I know that being on the receiving end of a review is quite stressful, and a critique of something that is close to one’s heart can easily be perceived as unjust. I try to write my reviews in a tone and form that I could put my name to, even though reviews in my field are usually double-blind and not signed.
- Selenko

I'm aiming to provide a comprehensive interpretation of the quality of the paper that will be of use to both the editor and the authors. I think a lot of reviewers approach a paper with the philosophy that they are there to identify flaws. But I only mention flaws if they matter, and I will make sure the review is constructive. If I'm pointing out a problem or concern, I substantiate it enough so that the authors can’t say, “Well, that's not correct” or “That's not fair.” I work to be conversational and factual, and I clearly distinguish statements of fact from my own opinions.

I used to sign most of my reviews, but I don't do that anymore. If you make a practice of signing reviews, then over the years, many of your colleagues will have received reviews with your name on them. Even if you are focused on writing quality reviews and being fair and collegial, it's inevitable that some colleagues will be less than appreciative about the content of the reviews. And if you identify a paper that you think has a substantial error that is not easily fixed, then the authors of this paper will find it hard to not hold a grudge. I've known too many junior scientists who have been burned from signing their reviews early on in their careers. So now, I only sign my reviews so as to be fully transparent on the rare occasions when I suggest that the authors cite papers of mine, which I only do when my work will remedy factual errors or correct the claim that something has never been addressed before.
- McGlynn

My review begins with a paragraph summarizing the paper. Then I have bullet points for major comments and for minor comments. Major comments may include suggesting a missing control that could make or break the authors’ conclusions or an important experiment that would help the story, though I try not to recommend extremely difficult experiments that would be beyond the scope of the paper or take forever. Minor comments may include flagging the mislabeling of a figure in the text or a misspelling that changes the meaning of a common term. Overall, I try to make comments that would make the paper stronger. My tone is very formal, scientific, and in third person. I'm critiquing the work, not the authors. If there is a major flaw or concern, I try to be honest and back it up with evidence.
- Sara Wong, doctoral candidate in cellular and molecular biology at the University of Michigan, Ann Arbor

I start by making a bullet point list of the main strengths and weaknesses of the paper and then flesh out the review with details. I often refer back to my annotated version of the online paper. I usually differentiate between major and minor criticisms and word them as directly and concisely as possible. When I recommend revisions, I try to give clear, detailed feedback to guide the authors. Even if a manuscript is rejected for publication, most authors can benefit from suggestions. I try to stick to the facts, so my writing tone tends toward neutral. Before submitting a review, I ask myself whether I would be comfortable if my identity as a reviewer was known to the authors. Passing this “identity test” helps ensure that my review is sufficiently balanced and fair.
- Boatman-Reich

My reviews tend to take the form of a summary of the arguments in the paper, followed by a summary of my reactions and then a series of the specific points that I wanted to raise. Mostly, I am trying to identify the authors’ claims in the paper that I did not find convincing and guide them to ways that these points can be strengthened (or, perhaps, dropped as beyond the scope of what this study can support). If I find the paper especially interesting (and even if I am going to recommend rejection), I tend to give a more detailed review because I want to encourage the authors to develop the paper (or, maybe, to do a new paper along the lines suggested in the review). My tone is one of trying to be constructive and helpful even though, of course, the authors might not agree with that characterization.
- Walsh

I try to act as a neutral, curious reader who wants to understand every detail. If there are things I struggle with, I will suggest that the authors revise parts of their paper to make it more solid or broadly accessible. I want to give them honest feedback of the same type that I hope to receive when I submit a paper.
- Müller

I start with a brief summary of the results and conclusions as a way to show that I have understood the paper and have a general opinion. I always comment on the form of the paper, highlighting whether it is well written, has correct grammar, and follows a correct structure. Then, I divide the review in two sections with bullet points, first listing the most critical aspects that the authors must address to better demonstrate the quality and novelty of the paper and then more minor points such as misspelling and figure format. When you deliver criticism, your comments should be honest but always respectful and accompanied with suggestions to improve the manuscript.
- Al-Shahrour

When, and how, do you decide on your recommendation?
I make a decision after drafting my review. I usually sit on the review for a day and then reread it to be sure it is balanced and fair before deciding anything.
- Boatman-Reich

I usually don’t decide on a recommendation until I’ve read the entire paper, although for poor quality papers, it isn’t always necessary to read everything.
- Chambers

I only make a recommendation to accept, revise, or reject if the journal specifically requests one. The decision is made by the editor, and my job as a reviewer is to provide a nuanced and detailed report on the paper to support the editor.
- McGlynn

The decision comes along during reading and making notes. If there are serious mistakes or missing parts, then I do not recommend publication. I usually write down all the things that I noticed, good and bad, so my decision does not influence the content and length of my review.
- Müller

In my experience, most papers go through several rounds of revisions before I would recommend them for publication. Generally, if I can see originality and novelty in a manuscript and the study was carried out in a solid way, then I give a recommendation for “revise and resubmit,” highlighting the need for the analysis strategy, for example, to be further developed. However, if the mechanism being tested does not really provide new knowledge, or if the method and study design are of insufficient quality, then my hopes for a manuscript are rather low. The length and content of my reviews generally do not relate to the outcome of my decisions. I usually write rather lengthy reviews at the first round of the revision process, and these tend to get shorter as the manuscript then improves in quality.
- Selenko

Publication is not a binary recommendation. The fact that only 5% of a journal’s readers might ever look at a paper, for example, can’t be used as criteria for rejection, if in fact it is a seminal paper that will impact that field. And we never know what findings will amount to in a few years; many breakthrough studies were not recognized as such for many years. So I can only rate what priority I believe the paper should receive for publication today.
- Callaham

If the research presented in the paper has serious flaws, I am inclined to recommend rejection, unless the shortcoming can be remedied with a reasonable amount of revising. Also, I take the point of view that if the author cannot convincingly explain her study and findings to an informed reader, then the paper has not met the burden for acceptance in the journal.
- Walsh

My recommendations are inversely proportional to the length of my reviews. Short reviews translate into strong recommendations and vice versa.
- Giri

How long does it take you to review a paper?
This varies widely, from a few minutes if there is clearly a major problem with the paper to half a day if the paper is really interesting but there are aspects that I don't understand. Occasionally, there are difficulties with a potentially publishable article that I think I can't properly assess in half a day, in which case I will return the paper to the journal with an explanation and a suggestion for an expert who might be closer to that aspect of the research.
- Nicola Spaldin, professor of materials theory at the Swiss Federal Institute of Technology in Zurich

It usually takes me a few hours.  Most of the time is spent closely reading the paper and taking notes. Once I have the notes, writing the review itself generally takes less than an hour.
- Walsh

It can take me quite a long time to write a good review, sometimes a full day of work and sometimes even longer. The detailed reading and the sense-making process, in particular, takes a long time. Also, sometimes I notice that something is not quite right but can’t quite put my finger on it until I have properly digested the manuscript.
- Selenko

A few hours. I like to use two sittings, even when I am pretty sure of my conclusions. Waiting another day always seems to improve the review.
- Callaham

Normally, a peer review takes me 1 or 2 days, including reading the supporting information.
- Müller

I almost always do it in one sitting, anything from 1 to 5 hours depending on the length of the paper.
- Chambers

In my experience, the submission deadline for reviews usually ranges between 3 working days to up to 3 weeks. As a rule of thumb, I roughly devote 20% of my reviewing time to a first, overall-impression browsing of the paper; 40% to a second reading that includes writing up suggestions and comments; 30% to a third reading that includes checking the compliance of the authors to the journal guidelines and the proper use of subject-typical jargon; and 10% to the last goof-proof browsing of my review. Altogether, it usually takes me more than a day.
- Giri

What further advice do you have for researchers who are new to the peer-review process?
Many reviewers are not polite enough. It's OK for a paper to say something that you don't agree with. Sometimes I will say in a review something like, “I disagree with the authors about this interpretation, but it is scientifically valid and an appropriate use of journal space for them to make this argument.” If you have any questions during the review process, don't hesitate to contact the editor who asked you to review the paper. Also, if you don't accept a review invitation, give her a few names for suggested reviewers, especially senior Ph.D. students and postdocs. In my experience, they are unlikely to write a poor quality review; they might be more likely to accept the invitation, as senior scientists are typically overwhelmed with review requests; and the opportunity to review a manuscript can help support their professional development.
- McGlynn

The paper reviewing process can help you form your own scientific opinion and develop critical thinking skills. It will also provide you with an overview of the new advances in the field and help you when writing and submitting your own articles. So although peer reviewing definitely takes some effort, in the end it will be worth it. Also, the journal has invited you to review an article based on your expertise, but there will be many things you don’t know. So if you have not fully understood something in the paper, do not hesitate to ask for clarification. It will help you make the right decision.

- Al-Shahrour

Remember that a review is not about whether one likes a certain piece of work, but whether the research is valid and tells us something new. Another common mistake is writing an unfocused review that is lost in the details. You can better highlight the major issues that need to be dealt with by restructuring the review, summarizing the important issues upfront, or adding asterisks. I would really encourage other scientists to take up peer-review opportunities whenever possible. Reviewing is a great learning experience and an exciting thing to do. One gets to know super fresh research firsthand and gain insight into other authors’ argument structure. I also think it is our duty as researchers to write good reviews. After all, we are all in it together. The soundness of the entire peer-review process depends on the quality of the reviews that we write.
- Selenko

As a junior researcher, it may feel a little weird or daunting to critique someone's completed work. Just pretend that it's your own research and figure out what experiments you would do and how you would interpret the data.
- Wong

Bear in mind that one of the most dangerous traps a reviewer can fall into is failing to recognize and acknowledge their own bias. To me, it is biased to reach a verdict on a paper based on how groundbreaking or novel the results are, for example. Such judgments have no place in the assessment of scientific quality, and they encourage publication bias from journals as well as bad practices from authors to produce attractive results by cherry picking. Also, I wouldn’t advise early-career researchers to sign their reviews, at least not until they either have a permanent position or otherwise feel stable in their careers. Although I believe that all established professors should be required to sign, the fact is that some authors can hold grudges against reviewers. We like to think of scientists as objective truth-seekers, but we are all too human and academia is intensely political, and a powerful author who receives a critical review from a more junior scientist could be in a position to do great harm to the reviewer's career prospects.
- Chambers

It is necessary to maintain decorum: One should review the paper justly and entirely on its merit, even if it comes from a competing research group. Finally, there are occasions where you get extremely exciting papers that you might be tempted to share with your colleagues, but you have to resist the urge and maintain strict confidentiality.
- Giri

At least early on, it is a good idea to be open to review invitations so that you can see what unfinished papers look like and get familiar with the review process. Many journals send the decision letters to the reviewers. Reading these can give you insights into how the other reviewers viewed the paper, and into how editors evaluate reviews and make decisions about rejection versus acceptance or revise and resubmit.
- Walsh

At the start of my career, I wasted quite a lot of energy feeling guilty about being behind in my reviewing. New requests and reminders from editors kept piling up at a faster rate than I could complete the reviews and the problem seemed intractable. I solved it by making the decision to review one journal article per week, putting a slot in my calendar for it, and promptly declining subsequent requests after the weekly slot is filled—or offering the next available opening to the editor. And now I am in the happy situation of only experiencing late-review guilt on Friday afternoons, when I still have some time ahead of me to complete the week's review.
- Spaldin



source by:

http://www.sciencemag.org/careers/2016/09/how-review-paper

Sign up here with your email address to receive updates from this blog in your inbox.

0 Response to "How to review a paper"