Step 1: Are the results of the study valid?
To detect whether or not the results are valid you must look for any biases in the study which may have lead to false outcomes. A systematic approach to assessment of the study design is needed.
Step 1a: Was there randomization?
Randomization improves study design by assuring (most of the time) that both known and unknown factors which may impact the outcome are equally distributed. Unrandomized studies often lead to the wrong conclusion that a treatment has a beneficial effect. For example, a study of antidepressants on smoking cessation may select for individuals already motivated to stop smoking unless treatment and control groups are randomized.
Step 1b: Was there excessive loss to follow-up?
Patients who are lost to follow-up tend to have different outcomes than patients who complete a study and can bias a study either for or against a treatment. You can decide if there was excessive loss to follow-up by reassessing the conclusions. Assume that all lost subjects had a negative outcome in a positive study or that they had a positive outcome in a negative study. If the conclusions change there was an excessive loss to follow-up.
Step 1c: Was there an intention to treat analysis?
In an intention to treat analysis patients remain in the original group to which they were randomized regardless of what treatment they received. This will prevent unknown factors that influence treatment from affecting the reported outcome. For example, if patients left the study group for the control group because of worsening clinical status in a randomized trial of a new therapy for heart failure, the study may have a positive outcome not because the new therapy was effective but because the patients with worse disease left the treatment group.
Step 1d: Was the study blinded?
The expectations of patients or providers can influence outcomes if the study is not blinded. If patient, care provider, or both were unblinded, then those evaluating outcomes should not be aware of assigned treatment groups.
Step 1e: Were treatment groups similar other than the intervention?
Look for tables showing baseline patient characteristics and treatments other than the intervention. Significant differences between the two groups (even if by chance) make it difficult to attribute the results to the study intervention. For example, some trials of new therapies for Rheumatoid Arthritis allow steroid use which can significantly alter answers on quality of life questionnaires.
Step 2: What are the results?
Step 2a: Absolute risk reduction (ARR)
ARR is the absolute difference between the proportion who suffered an event in the control group and the proportion in the treatment group. If X = the proportion of patients in the control group who had an event (such as death) and Y = the proportion of patients in the treatment group who had an event, then ARR= X-Y.
Step 2b: Relative risk (RR)
RR is the ratio of events among patients that receive the treatment to the events in control patients. RR = Y/X
Step 2c: Relative risk reduction (RRR)
RRR is the most common way to report treatment differences. It is the percentage that a treatment reduces risk of an event relative to the risk in the control patients. RRR = [1 - (Y/X)] x 100%. If the risk of death in a population receiving a new treatment is 10% and the risk of death in the control population is 20% then the new treatment reduces the risk of death by 50% compared with the baseline risk (RRR = 50%).
Step 2d: Number needed to treat (NNT)
NNT helps to determine if the benefits of therapy are worth the costs and potential harm of the therapy. NNT can serve as a more practical way of thinking about results. NNT = 1/ARR. An equivalent RRR but quite different NNT is obtained from the following examples. Consider that the risk of death for two different treatment populations is 25% and 0.5% compared with baseline risks of 50% and 1% respectively. The RRR for either treatment is 50% but the NNT for the first treatment is 4 (4 patients would need to be treated to prevent one death) and the NNT for the second treatment is 200 (200 patients would need to be treated to prevent one death). Although the RRR is equal, the first treatment would have a larger influence on your clinical practice especially if treatment is costly or not well tolerated.
Step 2e: Confidence interval (CI) and p-value
CIs define the range in which the true effect lies. We arbitrarily use 95% CIs to define statistical significance. The true result will lie beyond the CIs 5% of the time (2.5% above the upper boundary and 2.5% below the lower boundary of the 95% CI). The p-value gives the probability that the result is due to chance alone. P-values less than 0.05 are considered statistically significant.
Step 3: Will the results help in caring for my patients?
After examining the results you must decide whether or not the results apply to your patients. Ask whether your patient would be included in the study by looking at the inclusion and exclusion criteria as well as the features of reported subgroups. Also, look to see if the outcome studied is clinically relevant has no clinical significance. Ask whether treatment adversely affects other outcomes not studied in the trial. Finally, the costs, benefits, and risks must be assessed individually before deciding on treatment for your patients based on the study results.
References
1. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature, II: how to use an article about therapy or prevention, A: are the results of the study valid? JAMA. 1993;270:2598-2601.
2. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature, II: how to use an article about therapy or prevention, B: what were the results and will they help me in caring for my patients? 1994;271:59-62.
3. Dawson-Saunders B, Trapp RG. Basic and Clinical Biostatistics.
Norwalk, CT: Appleton and Lang , 1994.