How to critique a research article: a summary and critical appraisal

How should you read a research article before critiquing it?

Don’t read from the first line till the last. That is the slowest way to understand a paper.

S. Keshav wrote a short note called “How to Read a Paper” (ACM SIGCOMM Computer Communication Review, 2007). He suggests up to three passes, and it works outside computer science also. For a critique I would adjust it this way.

1. First pass (10 minutes). Read the title, abstract, headings, figures and tables, and the conclusion. After this you should be able to say in one sentence what the authors claim, and what kind of article it is. A review or a short communication is judged in a different way from an original study (see types of research articles).
2. Second pass (an hour or two). Now read the methods closely, then the results tables. Wherever you can’t follow a step, put a mark there.
3. Third pass. Read the discussion side by side with the results. Most problems come out here only. Many times the discussion says what the authors hoped for, and the tables say something smaller.

Look at the journal also. Suppose the paper was accepted in nine days by a journal you have never heard of. Then that becomes part of your appraisal (the checks are there in our predatory journal guide).

How do you write a summary of a research paper?

In the summary you report what the paper did and what it found. Use your own words and don’t judge anything yet. Mostly I begin with the aim and then the design and method. After that the sample and setting (who, how many, where, when). Then put the main findings with the key numbers. The authors’ own conclusion comes last.

For your own literature review notes, three or four sentences will do. In a coursework critique I would keep the summary to roughly one quarter or one third of the full length. Rest of the space should go to the appraisal. If the assignment wants APA style, give the full reference first (the format is there in our APA 7 guide).

Many scholars just reword the abstract. That is the authors’ sales pitch. Write from the methods and results. Many times you will notice that the abstract claim slightly more than what the results show.

What questions should you ask in a critical appraisal?

Take the paper one section at a time. I would ask these questions of almost any empirical article.

In the Indian social science and management papers that I read, the sample row gives the most trouble. Many papers write only “Data were collected from 400 respondents”, and you can’t tell who they are. Compare “Questionnaires were circulated in the college’s WhatsApp groups”, where at least the convenience sample is visible. What each sampling method can support is explained in our guide to sampling techniques.

With measures, check whether the scale actually worked in this sample. Suppose a scale was validated on American undergraduates. That does not make it automatically valid for farmers in Vidarbha (see reliability and validity).

Which critical appraisal checklist should you use?

If your field is health, nursing, psychology or public health, use a checklist that was made for that study design. Don’t prepare one of your own. The CASP checklists and the JBI critical appraisal tools are the two free sets that people use the most. Choose the one which matches the study’s design.

Many people get this point wrong. CONSORT, STROBE, COREQ and PRISMA are reporting guidelines. They give a list of what a paper should report, so that readers are able to judge it. Whether the study was done well, they do not tell you. The STROBE website itself says its checklist “is not an instrument to evaluate the quality of observational research”. A trial may tick every CONSORT item and still carry a serious flaw. The flaw is just reported clearly.

Reporting guidelines are still useful. If a STROBE item is missing (say, how missing data were handled), you can write that the paper doesn’t let you judge that point. Only don’t call this “quality assessment using STROBE”.

Reporting guidelines are listed by the EQUATOR Network. Risk-of-bias tools are listed by its sister project, the LATITUDES Network. You will need those tools for a formal systematic review, and our systematic review guide matches each tool to a design. Outside health research there is no standard checklist, so use the questions above.

What is the format of an article critique? (with example)

If your instructor has given a format, follow that. Otherwise this outline works for coursework, journal-club handouts and most “review this article” assignments.

1. 1

   Reference and context

   Give the full reference and the article type, and say why the paper is relevant for your topic.

2. 2

   Neutral summary

   Write the aim, design, sample, main findings and the authors’ conclusion. Keep your judgement out of this part.

3. 3

   Strengths

   Two or three specific strengths. Link each one to some part of the paper.

4. 4

   Weaknesses

   The problems which affect the conclusions most, with evidence, most important first.

5. 5

   Overall judgement

   Say how much the conclusions can be trusted and what that means for your own study.

A short article critique example

Summary. This article studies women in self-help groups (SHGs) in rural Odisha, and asks if financial-literacy training is associated with their mobile banking use. The authors surveyed 280 members from one district. They recruited them through a partner NGO and used a Technology Acceptance Model questionnaire translated into Odia. With structural equation modelling, they report that members who got the training used mobile banking more often. So they conclude training “leads to” adoption. They even want it compulsory for all SHGs.

Strengths. The question is relevant for policy. Rural women are also under-represented in a literature that is mostly urban. The questionnaire is printed in an appendix, and the authors report reliability from their own sample.

Weaknesses. The biggest problem is the causal claim. One cross-sectional survey can show that training and use go together. It cannot show that the training causes the use. Women already comfortable with phones may be the ones attending training. Second problem is that every respondent came through the NGO which runs the training. So the results cannot be generalised to Odisha, and much less to all SHGs. Third, the Odia translation gets one line only, with nothing on back-translation or a pilot. Ethics approval is mentioned. Funding and competing interests are not declared though. Here that is important, because a co-author is employed by the same NGO.

Overall judgement. As a descriptive study it is useful. Training and use go together in this data, that much I accept. It does not support the causal conclusion or the national recommendation. In my review I would cite it as evidence of a link among NGO-connected SHGs, and note the need for a comparison group or longitudinal design.

I put the causal claim first since that one damages the conclusion the most. Under every weakness I wrote why it matters.

How do you write a critique that is critical but not negative?

In a “critical” review you are judging the paper with care. In a balanced critique you find strengths and also weaknesses. If you list only the faults, or give only praise, you have done half the work.

• Link each criticism to a particular page, table or sentence of the article.
• For each weakness, write what it does to the conclusions. Some flaws change nothing in them, so those go under minor points.
• Keep in mind what the authors could not reasonably do (on a UGC fellowship, a national random sample is rarely feasible).
• Give a possible fix, for example a comparison group, a longitudinal design or a sensitivity analysis.
• Write about the paper and not about the people. Say “The sampling limits generalisation”. Don’t write “the authors were careless”.

Department journal clubs vary from polite praise to very sharp criticism. Go with two strengths and two weaknesses, with page numbers, and you can hold your own.

How do article critiques feed into your literature review?

For your thesis you will not write a full critique of each paper. Still, any paper you depend on should get a short appraisal note. Your review chapter has to tell how strong the evidence is, and not only what the evidence says. Fill one set of rows like this for each article.

Mostly scholars leave the last two rows empty. But those rows save you in the DC or RAC meeting, when somebody asks why you trusted a particular study. Our literature review guide explains how these notes turn into a synthesis matrix, and after that into a chapter arranged by theme.