Article: A Study to Develop Clinical Decision Rules for the Use of Radiography in Acute Ankle Injuries
Authors: Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR
Publication: Ann Emerg Med. 1992;21(4):384-390
Step 1: Evaluate the Study Design
Step 1a: Will serve a purpose if it is valid? Does the rule make clinical sense?
The purpose of the study was to design a clinical decision rule (CDR) "...that will predict fractures in patients with ankle injuries, thereby assisting clinicians in being more selective in their use of radiography." The authors further specify that they wanted the rule to have 100% sensitivity (100% negative predictive value). The CDR for midfoot injuries, which the authors also developed, will not be discussed here.
The rule stated would be as follows. A patient presenting after an acute blunt ankle injury who lacks any of 4 characteristics (age > 55 years, inability to bear weight for 4 steps both immediately after the injury and in the ED, bone tenderness at the posterior edge or tip of either malleolus) can safely forgoe radiography because the likelihood of a clinically significant fracture approaches zero.
The rule definitely serves a purpose. The decision to selectively order radiographs in acute ankle injuries is complex (one must consider patient's age, mental status, mechanism of injury, presence of bruising, swelling, location of tenderness, ability to perform passive and active range of motion, and strength in many different movements). As the authors mention, many unnecessary radiographs are ordered for ankle injuries. Only roughly 15% of patients who come to the ED after an acute ankle injury actually have a fracture. Unnecessary radiographs are expensive, slow down the flow of an ED (thus impairing the care of other patients), and expose the patient to unnecessary radiation. The authors also document that their sample of clinicians were poor at predicting the probability of ankle fracture without use of any rules. A caveat should be mentioned when considering this CDR. If truly valid, the rule does not predict who may or may not have a clinically significant injury. Damage to tendons and ligaments may be more devastating in some circumstances than fractures.
The rule also makes clinical sense (see step 1c below) in most circumstances. Obviously the rule does not make sense if the patient is not ambulatory at baseline or has altered mental status. The rule suggests a simple course of action and is therefore likely to be used if valid.
Step 1b: Is the outcome of interest clearly defined and clinically important?
The outcome of interest, a clinically significant ankle fracture, is clearly defined as any fracture that is not an avulsion fracture of 3 mm or less. Caveat #1: The authors do not mention it, but they mean a radiographically apparent ankle fracture. Rarely (especially in children) a clinically important ankle fracture will not be present on an initial radiograph. Most clinicians would agree that ankle fractures are clinically important. They generally require immobilization (splinting or casting), sometimes require surgery, and may result in significant morbidity and rarely mortality (eg PE, fat embolism). Caveat #2: There are other clinically important outcomes(severe soft tissue injuries) that are not included in the study (see above).
Step 1c: Are the predictors clearly defined, sensible, reliable, and reproducible as determined by different clinicians.
32 predictors (including the 4 eventually chosen for the rule) were selected for the initial analysis. The authors do not report what these 32 predictors were. 17 were listed in table 2 and chosen for further analysis after an initial pilot study. Although they are not explicitly defined, they were simple enough as to not require explicit defining.
The 17 predictors individually seem sensible choices. What seems insensible is the abundance of them. It seems unrealistic and impractical that 17 (let alone 32 in the pilot study) variables could be be accurately assessed and documented in one clinical encounter. The 4 predictors chosen for the rule seem sensible. Most clinicians are clear on what an acute blunt ankle injury is, determination of age, how to interpret weight bearing status, and the anatomical location of malleoli. It makes sense that these predictors would be associated with ankle fractures. What seems confusing to some extent is the predictor "unable to bear weight (both immediately and in the ED-4 steps)." It makes sense that one would not need a radiograph in someone who could do both. It also makes sense that a radiograph would not necessarily be required in someone who could not bear weight immediately after the injury, but could in the ED. But what about the person who could bear weight immediately after the injury but now in the ED cannot? According to the rule, this patient would not need a radiograph. The specification of 4 steps also seems somewhat restrictive. What if the patient can ambulate for 4 steps but not 5? How was this number determined?
The investigators did measure interobserver reliability of the 32 predictors on 100 patients in the pilot study. The authors used this information in choosing which predictors (pertaining to the physical examination)to exclude from the decision rule. (Only predictors with high interobserver agreement (k>0.6) were included.)
Step 1d: How did the authors construct the list of potential predictors? Did the authors include all elements that might have predictive power in the rule?
"The [initial 32] clinical variables were chosen for the study...based on [the investigators] clinical experience and on previous studies." The "previous studies" are not specifically referenced here. The 32 predictors chosen for the pilot study on 155 patients were then narrowed down to 17 based on significant association with malleolar fractures on radiographs by univariate techniques. Two of the predictors (tenderness over the posterior aspect or tip of the lateral or medial malleolus) were included in the final decision rule based on both statistically significant associations (determined by recursive partitioning analysis) with radiographic fractures and high interobserver reliability. The other 2 predictors (age >55 and inability to bear weight immediately after the injury) were included in the final decision rule based on statistically significant associations with radiographic fractures and the assumption that they would have high interobserver reliability. They did not include all elements that might have predictive power in the rule. They specifically mention that ecchymosis, range of motion, anterior drawer sign, swelling of either medial or anterior aspect of the ankle, and bone tenderness over certain anatomic areas were not included in the rule because of low interobserver reliability. The authors also state that different number and variation of predictors could have been used to create a CDR with 100% sensitivity. The number and combination of predictors in the final rule were chosen because of better specificity and simplicity in addition to100%sensitivity. This decision seems reasonable, but one must remember that there are many historical factors, signs and symptoms not included in the rule that in some circumstances may suggest fracture. Certain mechanisms of injury and diseases of bone or pathologic states, although not included in the rule, may predict higher likelihood of fracture.
Step 1e: Were those who were assessing predictors blinded to outcomes (and vice versa)?
yes
Step 1f: Was the study site described?
The 2 study sites were described as tertiary university (teaching?) hospitals and the clinical settings were described as EDs. The authors do not explicitly mention that all of the patients meeting the inclusion criteria and lacking the exclusion criteria were consecutively entered into the study so that we cannot be certain that there was not a selection bias.
Step 1g: Were the data collected prospectively?
Yes.
Step 2: Evaluate the Results
Step 2a: Does the rule have predictive power? Look at the results and statistical methods used to derive the rule. Were all of the relevant results reported and described well? Was the statistical analysis appropriate?
The rule does have good predictive power (100% negative predictive value). The specificity of the rule was low (40%). An impressive proportion of radiographs (36%) would have been omitted had the rule actually been employed. Most of the relevant results were reported in a series of figures and tables. However, confidence intervals (CI) for sensitivity and specificity of the rule, were not reported. The proportion of the study population with each predictor is also not reported. The authors described both of their attempts to derive the CDR, initially using multivariate analysis and ultimately with recursive partitioning. The rule derived by multivariate analysis was abandoned due to lower sensitivity and difficulty of use. Both methods would have been appropriate.
Step 2b: Was there an adequate sample size, including an adequate number of positive outcomes?
The sample size was impressive (750 with ankle or midfoot pain, 689 with ankle pain). There were 102/689 (14.8%) clinically significant fractures. No talus fractures occurred which may or may not affect the predictive power of the rule concerning talus fractures. This suggests a larger sample size would have been better. There were over 10 positive outcomes per predictor (25.5). If the sample of 100 used to determine interobserver reliability of the predictors had been larger the 95% CI around the kappa for predictive physical examination findings (0.57-0.88) used in the rule would have been smaller.
Step 2c: Were important predictors present in a significant proportion of the study population?
These data was not reported.
Step 2d: Were important patient characteristics reported?
Age and gender were reported. Other important characteristics (eg mechanism of injury) were also reported.
Step 2e: Was interobserver reliability of the rule assessed and reported?
Interobserver reliability for predictors was tested on 100 patients and was very good (k=0.72) for the combination of physical examination variables in the rule. However, interobserver reliability for the outcome of interest (radiograph with clinically important fracture) was not measured. Only 1 radiologist read all of the radiographs.
Step 3: Has the rule's validity been tested?
Yes, to date 9 validation studies are reported in the English literature.
1) Stiell IG, Greenberg GH, McKnight D, et al. Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation. JAMA. 1993;269(9):1127-1132.
2) Pigman EC, Klug RK, Sanford S, et al. Evaluation of the Ottawa clinical decision rules for the use of radiography in acute ankle and midfoot injuries in the emergency department: an independent site assessment. Ann Emerg Med. 1994;24(1):41-45.
3) Kerr L, Kelly A, Grant J, et al. Failed validation of a clinical decision rule for the use of radiography in acute ankle injury. N Z Med J. 1994;107:294-295
4) Solomito AL, Singal BM, Radack M. Ankle radiography in the emergency department: a prospective validation of Ottawa ankle rules. Acad emerg Med. 1994;1:A64.
5) Lucchesi GM, Jackson RE, Peacock WF, et al. Sensitivity of the Ottawa rules. Ann Emerg Med. 1995;26(1):1-7.
6) McBride KL. Validation of the Ottawa ankle rules. experience in a community hospital. Can Fam Physician. 1997;43:459-465.
7) Auleley G, Kerboull L, Durieux P, et al. Validation of the Ottawa ankle rules in France: a study in the surgical emergency department of a teaching hospital. Ann Emerg Med. 1998;32:14-18.
8) Leddy JJ, Smolinski RJ, Lawrence J, et al. Prospective evaluation of the Ottawa ankle rules in a university sports medical center. with a modification to increase specificity for identifying malleolar fractures. Am J Sports Med. 1998;26(2):158-165.
9) Perry S, Raby N, Grant PT. Prospective survey to verify the Ottawa ankle rules. J Accid Emerg Med. 1999;16:258-260.
The 8th study was excluded from the analysis below because the investigators used a modified version of the CDR.
|
study--> appraisal steps |
1) | 2) | 3) | 4) | 5) | 6) | 7) | 9) |
| 3a: rule applied to broad or narrow pt sample | narrow (n=1485); same population and clinicians used in derivation and validation studies | narrow (n=110); university and community hospital w/ different clinicians and triage nurses | broad;(n=350);community and tertiary care settings in New Zealand w/ wide range of clinicians | broad;(n=631); community setting in 2 cities (Ohio) w/ wide range of clinicians | broad (n=484); community setting in Michigan w/ wide range of clinicians | broad(n=296); community setting in Canada w/ family practitioners | broad (n=416); tertiary hospital in France w/ variety of MDs | broad (n=800); tertiary center in UK w/ variety of MDs |
| 3b: pts chosen in unbiased manner | yes | yes | no | unclear, convenience sample | yes | no | yes | yes |
| 3c: different clinicians involved | no | yes | yes | yes | yes | yes | yes | yes |
| 3d: blinded assessment of predictors and outcomes | yes | yes | no | no (not mentioned) | outcome assessed by blinded radiologist. Not clear that clinicians blind to outcome when assessing predictors | no | yes | no; study investigators acting as part of clinician population were not blinded |
| 3e: prospective or retrospective | prospective | prospective | prospective | prospective | prospective | prospective | prospective | prospective |
| 3f: rule consistent and valid when applied | yes; 100% sensitivity (95% CI 93-100%); 49% specificity (95% CI 44-54%); would have reduced radiographs by 34% | yes; 100% sensitivity (95% CI 92-100%; 19% specificity (95% CI 6-24%); would have reduced radiographs by 19% | 93% sensitivity; 11% specificity; 95% CI not reported | yes; 100% sensitivity (95% CI 94-100%);19% specificity (95% CI 16-23%); would have reduced radiographs by 17% | 95% sensitivity(95% CI 93-97%); 16% specificity(95% CI 12-19%) | 97% sensitivity (95% CI 91-100%); 30% specificity (95% CI 24-36%); would have reduced radiographs by 26% | yes; 98% sensitivity (95% CI 88-100%); 45% specificity (95% CI 39-50%) | 94% sensitivity; 46% specificity; no CI reported |
| 3g: excessive loss to fu | 175/1660 (10.5%) excluded b/cause incomplete data collection | 16/126 (12.6%) excluded b/cause incomplete data collection | 0 | 64/820(7.8%) excluded b/cause incomplete data collection | 42/484 (8.6%)pts charts not complete yet included in study | 22/318 (7%) excluded because incomplete data collection | 0 | 0 |
| comments | done by original investigators; refined rule in phase 1 of study (age>55 now omitted from rule with similar sensitivity and specificity) | used refined rule; compared application of rule by attending MDs and triage nurses; triage nurses fared slightly worse | used original rule; many design flaws | used original and refined rule (results same) | used refined rule; clinicians were not aware that they were using 3 predictors (they initially collected data on a data form w/ ~30 predictors) | used refined rule; had 37 pts younger than 16 in study; not all pts had radiography | used refined rule | used original rule; 26% did not have radiographs |
Study 7 was the best of all the studies and did show the rule to be valid and sensitive.
Step 4: Did the rule have an impact? Has an impact analysis been performed?
4 impact analyses have been performed.
1) Stiell IG, McKnight, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271(11):827-832.
2) Stiell IG, Wells G, Laupacis A, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. BMJ. 1995;311:594-597.
3) Auleley G, Ravaud P, Giraudeau B, et al. Implementation of the Ottawa ankle rules in France: a multicenter randomized controlled trial. JAMA. 1997;277:1935-1939.
4) Cameron C, Naylor CD. No impact from active dissemination of the Ottawa ankle rules: further evidence of the need for local implementation of practice guidelines. CMAJ. 1999;160:1165-1168.
|
Impact analysis--> Appraisal |
1) | 2) | 3) | 4) |
| 4a: did clinicians use the rule? | yes, 96% compliance | yes, 95% compliance | yes, ~93% compliance | no |
| 4b: Did the rule decrease the number of radiographs? Were there optimal care outcomes? | yes, 28% relative reduction in radiography at intervention hospital; yes | yes; ~22% relative reduction in radiography; yes | yes; ~22% reduction in radiography in intervention period; ~15% in postintervention period | no |
Step 5: Grade the rule
This rule could be graded a 1 even though the results of some of the validation and impact studies have been mixed.