Reports in both the media and medical literature concerning harm from an exposure are eye-catching for both patients and clinicians. When one suspects an exposure or intervention is harmful, it is important to interpret all of the information that is available on that particular topic. This tutorial will help you to systematically review studies on harm. It is largely based upon the User's Guide from JAMA (1). Descriptive studies (case series and case reports) will not be considered here.
Step 1: Assess the study design
A poor study design can completely negate the study's validity.
Step 1a: What was the study hypothesis? did it make sense?
Step 1b: What was the study design (randomized controlled trial, cohort, case control)?
Randomized controlled trials (RCTs) may offer the best information. They represent a true prospective experiment in which patients are randomly assigned to be exposed or not exposed (or in some cases to be exposed to something different than the proposed causal agent). Both groups are then assessed for a positive or negative outcome. RCTs are rarely done to study exposures because of the ethical implications. However, much useful data about harm has originated from RCTs that were done to study interventions. Cohort studies involve following 2 groups of patients over time (the exposed and the unexposed) and observing outcomes. They provide a nice substitute for RCTs which are not logistically or ethically feasible in many situations. They are also more desirable when bad outcomes are infrequent. When harmful outcomes are rare, it would take a lot longer and/or require too many resources to acquire the number of subjects required to show significant trends. Case-control studies involve 2 groups, one with the outcome of interest and one without the outcome, chosen by the investigators who assess retrospectively for presence or absence of exposure to the causal agent. When the outcome of interest is rare or takes a long time to develop, this study design is especially useful.
Step 1c: Were there comparison groups?
The presence of comparison groups greatly influences the credibility of the results. With the above 3 study designs there is the capacity for comparison groups.
Step 1d: How similar were the comparison groups?
Again, this greatly influences the credibility of the results. Ideally the comparison groups would be similar in every regard apart from the exposure in RCTs and cohort studies or the outcome for case-control studies. RCTs are theoretically optimal for assuring that groups are similar with regard to all determinants of outcome (apart from the exposure) both known and unknown. Subjects in a cohort study either select themselves or are selected by the investigators. Therefore the subjects would unlikely be similar to the unexposed group. Some of the different characteristics between groups may be confounding variables. For this reason it is extremely important that the investigators document carefully the characteristics of both groups and either demonstrate their similarity or statistically control for their differences. Even if the authors control for the confounding variables or demonstrate their equivalence across both groups, there may be an imbalance of other prognostic factors (confounding variables) that the authors are not aware of which have determined the outcome. One of the benefits of the case-control study design is that the investigators can choose the groups to be similar for all variables except for outcome of interest. If there are differences between the 2 groups, the investigators must adjust for them.
Step 1e: Was the exposure clearly defined and able to be reliably evaluated?
Step 1f: Was the outcome clearly defined and able to be reliably evaluated? Was it a clinically important outcome?
Step 1g: Were the exposures and outcomes measured in the same way in the groups being compared?
Case-control studies are susceptible to recall bias and investigator bias during the assessment for the exposure. Investigators may use strategies to minimize these biases (such as blinding interviewers and subjects to the hypothesis of the study). Exposure opportunity should be the same in both cases and controls or else the true risk might be different than the study reports--higher if controls had greater opportunity and lower if controls had less opportunity for exposure. Assessment of the outcome is a key issue in RCTs and cohort studies, which may be subject to surveillance bias. Surveillance bias might lead to a spurious report of increased risk from an exposure.
Step 1h: How good was the follow-up? Was it appropriately long? Was there excessive loss to follow-up?
This really only applies to RCTs and cohort studies. Significant loss to follow-up is a major threat to the validity of the study (see tutorial 1). Patients who are lost to follow-up may have very different outcomes from those who are not. The longer the follow-up period, the more opportunity there is for loss to follow-up. On the other hand, a longer follow-up period provides for positive outcomes for analysis.
Step 2: Assess the results
Step 2a: How strong was the association between the exposure and the outcome?
The strength of inference will always be less for cohort and case-control studies because of the potential for confounding variables for which statistical adjustment was not made. Relative risk (RR) is the most common way of expressing the relationship between exposure and outcome in RCTs and cohort studies. The odds ratio (OR) is the substitute measure for RR in case-control studies. When the outcome of interest is rare in the population from which the sample is drawn the OR closely approximates the RR.
Step 2b: Was there a temporal relationship between the exposure and outcome?
In order to postulate a causal relationship, the exposure must precede the outcome.
Step 2c: Was there a dose-response gradient?
We will more likely believe that an exposure increases the risk of a bad outcome if a rise in adverse outcomes occurs with increasing quantity and duration of the exposure.
Step 2d: How precise was the estimate of risk?
One can assess the precision of the estimate by looking at the confidence interval (CI). In a "positive" study (the investigators report an increased risk of adverse outcome from an exposure) the lower end of the CI shows the minimum estimate of the RR or OR. If it is less than 1, the result is not very precise and you should question the study results being concluded as "positive." In a "negative" study (the investigators report no change in outcome whether or not exposure occurred), the higher end of the CI shows the maximum estimate of the RR or OR. If it is greater than 1, the result is not very precise and you should consider that nothing can be concluded from the study.
Step 2e: Did the authors adjust for differences between the comparison groups?
Step 2f: Was the statistical analysis appropriate?
Step 3: Do you agree with the authors' conclusions?
Step 4: How applicable are the results?
Step 4a: Did the risk apply to a narrow or broad population?
Look at patient characteristics such as age, gender, race, morbidity, geography, type of healthcare received, etc. The study may have shown risk for only certain groups in certain settings.
Step 4b: What was the magnitude of risk? (ie, the clinical importance)
The RR and OR, even if impressively high, do not equate to clinical importance. They do not measure how frequent the exposure actually is, how often a problem occurs, or how severe the problem is. One can make an estimate of the magnitude of risk using a calculation analogous to the number needed to treat (NNT) discussed in tutorial 1. First the absolute risk (AR) increase is calculated. The reciprocal of the AR increase is the number of people who must be exposed to the harmful agent to result in one adverse outcome (=number needed to harm).
Step 4c: Should clinical practice be altered because of the results of this study?
Consider the the strength of the association and the study design. With a strong study design (eg a well constructed RCT ) a small increase in risk may represent a true harmful effect. We should require a much greater increase in risk to consider a true harmful effect in a weaker study design (eg cohort and case-control studies). Small increases in risk in these kinds of studies are likely to result from subtle flaws in study design. Consider also the magnitude of risk discussed above. Even after you have considered all of the aspects discussed in this tutorial the question of changing clinical practice can still be difficult to answer. One must consider what the adverse consequences of reducing or eliminating the exposure will be.
Appendix
A variable is said to be confounded with another variable if it is impossible to determine which variable is responsible for the observed effect.
Recall bias: Subjects in the group with a positive outcome may be more likely to remember an event (exposure) than the group with a negative outcome
Interviewer bias: An interviewer may question more carefully in the subject with a positive outcome as compared to the subject with a negative outcome.
Surveillance bias: If investigators are looking for an association between an exposure and harm, the study itself may lead to an increase in screening (or surveillance). Increased surveillance may lead to the detection of disease that would have otherwise gone unnoticed.
Relative risk (RR): the incidence (or risk) of the adverse outcome in the exposed group divided by the risk of the adverse outcome in the unexposed group. Values >1 represent an increased risk associated with the exposure whereas values <1 represent a decreased risk associated with the exposure.
Odds ratio (OR): the odds of a case patient being exposed divided by the odds of a control patient being exposed.When the outcome of interest is rare in the population from which the sample is drawn the OR closely approximates the RR.
Using a 2x2 table RRs and ORs can be calculated from the following equations.
RR=[a/(a+b)]/[c/(c+d)]
OR=(a/c)/(b/d)
| Patient | Adverse Outcome (case) | No Adverse Outcome (control) |
| Exposed | a | b |
| Not Exposed | c | d |
References
1. Levine M, Walter S, Lee H, et al. Users' guide to the medical literature IV. How to use an article about harm. JAMA. 1994;271(20):1615-1619.
2. Dawson-Sanders B, Trapp RG. Basic & Clinical Biostatistics.
Appleton& Lange. 1994. Norwalk, CT.