Behind the scenes of constructive reviewing

What does it mean to write a constructive review for a qualitative research paper? In the first of the Inside the Ethnography Atelier Workshop series, Lisa Cohen took us for a deep dive into qualitative research reviewing. In brief, her recommendations are to stay open to different ways of conducting research, to develop an acute eye for what can make a paper stronger, and to always keep a constructive mindset. Whether you are a junior scholar trying to learn how to review a paper or a seasoned scholar interested in making the reviewing process more helpful for authors, here are a few tips and thoughts on how to build and write constructive reviews.

Now let’s imagine you just received a reviewing assignment. The very first step - even before your start reviewing - is to ask yourself whether you should accept the review. Here are a few decision tools that might help – if you feel like you know something about the theory, methods or questions that are raised in the paper; if you feel you can be objective about the paper; if you feel you will learn something; and/or if you have the resources and capacity to review, this might be a good opportunity. Lisa suggests reviewing more than you would like, but not as much as she does (over 20 papers per year!). It’s hard to say how long to spend on a review - perhaps you will need time for ideas to come together; perhaps not. Still hesitant? To cite a few benefits of reviewing: you learn about writing, you influence the field by offering guidance to authors, you contribute an important service, and you get to read new and exciting work. In other words, reviewing is a great way to help move the field forward, and it also makes you grow as a scholar.

Let’s say you’ve decided to accept the assignment. What next? Before you start reading the article, you might remind yourself of some specificities of qualitative research. First, figuring out the paper’s story is typically the hardest part in a qualitative research paper. This means that as a reviewer, a big part of your role might be about helping the author find what ties the findings, methods and theory together. Second, qualitative research has fewer conventions in the way it should be conducted - it involves a variety of approaches. The credibility of a qualitative research paper stems from the trustworthiness and rigor authors convey in their work; as a reviewer, your goal will be to ponder your review accordingly. Finally, the literature review in qualitative work seeks to establish a literature rather than develop hypotheses. You may want to have that in mind while reviewing the theory section. Taking these specificities as premises will help you ensure the review is constructive.

Now to the review itself. Where to start? With a pen or keyboard in hand, read the abstract. Then read the data section. If you are like Lisa, you might regret it if you start elsewhere. It is rare to bring in completely new data, as data often anchors the paper; hence starting there will give you the paper’s core. Consider the text, tables and quotes, then think about what you would expect to see in the paper’s introduction. Once you’ve reflected on the data, move to the introduction and theory sections. Think about what sort of data would fit the theory the authors are articulating. Are there any mismatches? Does the research question match the data? Do the methods match the theory? One of your main challenges as a reviewer is to think about what would be needed for findings, theory and methods to align. For other important issues you might want to keep an eye out for while reviewing, check out the common issues Lisa listed as typical issues in qualitative papers (see below).

As you write up the review itself, you then have several tasks. First of all, start by restating the paper’s goal. Summarizing may help you check what you’ve understood from the paper; importantly, it will also help the author(s) realize what an outsider might see and how different it might be from their initial intent. As an author, this is precious! Second, identify the strengths of the paper. Write these strengths as an explicit paragraph in your review, not just a broad sentence, this helps set the tone and build a relationship between the authors and yourself. Remember this is a conversation, not a mere critique. Start your sentence saying “it’s great that you …” or “there are many things I like…”. Third, provide constructive criticism. If something is unclear, write questions rather than statements - asking for example “what is the research question?” if the research question is unclear to you. Offer options, whenever possible, as to what you think the authors could do. Be specific about the issues you see. Explain why and how you have a problem with something. Contextualize your comments - explain that you are someone who does X research more than Y, so that the authors know how to interpret your suggestions. Make suggestions, but always make sure the authors know that all choices are ultimately theirs.

As you finalize your review, a few things to remember. First, aim for balance across sections. You might even make it explicit to the authors that you are presenting your comments in what you see as their order of importance. Don’t aim for comprehensiveness, rather try and cover what is most critical. This may mean you will not cover every single part. Second, keep in mind that your role is to help not only the author, but also the editor. Number your paragraphs so that the editor can refer to your comments in their letter. Write as good a review as you can so that the editor can draw from it in their decision. Use the notes to the editor to share additional thoughts. Perhaps the paper isn’t there yet, but you see great potential in the data; let the editor know. To sum up, both the review’s content and form can matter to make it a constructive one.

When you pick up your next review, the insights above may help you craft your review constructively, step by step. As you reflect on the story, on its importance, and on the authors’ approach, Lisa’s questions and suggestions can help you delimit the most crucial points you think might guide the authors towards an improved version of their work and ideas. Remember that no research paper is written alone; it is the result of a conversation that occurs – in part – during the review process. Lisa has been recognized as an incredibly constructive reviewer both for qualitative and quantitative reviewing in Organization/Management and Sociology Journals. Both as reviewers and as authors, we have much to learn from her experience and insights.

A common set of issues and questions to have in mind, based on Lisa’s recommendations:

During and after the first reading:

1. Is this important work? Is it interesting? Will others beyond the specific domain of the paper care about this piece of research?

2. Is this piece of research new? Are the authors forgetting old papers that share a similar argument? Does this all seem too obvious?

3. Do I understand the paper as a whole? Did the authors define the terms? Did they avoid jargon?

When considering the data:

1. Are the findings as convincing as needed? Did the authors show their data, rather than say what was in the data? Should some findings be in the methods section?

2. As a reader, can you distinguish the authors’ interpretation from what participants said? Is it clear what participants said? Does the analysis come too soon? Is the analysis sufficient or are the findings too descriptive?

3. Did the authors claim more than can be claimed with regards to the data they show? Do the quotes from participants match the concepts the authors use?

When considering the literature and contribution:

1. Did the author(s) go back beyond a decade in their review of the literature? Is some classic work missing?

2. Is the review of the literature too narrow? Is the literature review too broad, and could the author(s) focus more specifically on what the reader(s) need to know?

3. Are there too many concepts? Are the authors trying to do too much in one paper? How would the paper be without some of the concepts that are included? Is there a theoretical contribution? Does understanding the phenomenon matter and do the authors make a case for it?

When considering the methods section:

1. Is it clear who the authors talked to? Is it clear that the authors are using all of the data? Could some tables be helpful to include more data?

2. What is the unit of analysis? Are the authors explaining clearly how they analyzed the data, rather than providing an abstract description of the analysis?

3. As authors, we might forget that there is more than one way of collecting and analyzing qualitative data - as reviewers, can you clearly understand the different phases the authors went through in their methods and what came out of each step?

Figures and tables:

1. Are the figures too complex? Are they necessary to show how things fit together? Do the figures match the data?

2. Are there tables to show more data than is present in the body of text? Or to show patterns present in the data?

3. As a reviewer can you see what the authors’ claims are based upon? Some tables may ultimately become an appendix, but tables might help reviewers assess whether the claims the authors make align with the data.

Some references about the variety of approaches in qualitative research:

Pratt, M. G., Kaplan, S., & Whittington, R. (2020). Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research. Administrative Science Quarterly, 65, 1–19. https://doi.org/10.1177/0001839219887663

Grodal, S., Anteby, M., & Holm, A. L. (2020). Achieving Rigor in Qualitative Analysis: The Role of Active Categorization in Theory Building. Academy of Management Review, Online First. https://doi.org/10.5465/amr.2018.0482