If an issue has inconsistent results, this refers to wide variability of treatment effects across individual studies. When variability in treatment effects is seen, it is important to try to identify the explanations for these inconsistent results; many times this variability may be explained by differences in populations, differences in the intervention, difference in outcome measures, or differences in study methodology. When no explanation for the inconsistency in results is identified, this may lower our confidence in the estimate of effect across the body of evidence.
More often than not, direct comparisons of interventions (e.g., how does fecal immunochemical testing compare with colonoscopy in reducing CRC mortality?) are unavailable, introducing uncertainty of comparative effectiveness. In addition, the existing body of evidence may differ with respect to the population, intervention, or outcome as it relates to a specific clinical question.
When studies include few patients or few events, the estimates of effect have wide confidence intervals that include benefit and no effect, or even potentially harm. In a systematic review on the use of thiopurines vs. placebo for induction of remission in adults with moderate to severe Crohn’s disease, the pooled estimate from five RCTs showed a relative risk of 0.87 (95% confidence interval, 0.71-1.06) for the outcome of failure of remission, with the lower boundary of the 95% confidence interval suggesting a close to 30% benefit, but the upper boundary failing to show an effect.6 Although the pooled estimate suggests that thiopurines are beneficial, our confidence in the result is reduced because of imprecision.
Publication bias is another factor that can result in a rating down of the quality of the evidence. Often this is attributed to a lack of reporting of small studies and/or studies showing no benefit. For example, a meta-analysis of 27 publications on estimates of cancer risk in Barrett’s esophagus showed an inverse relationship between study size and cancer risk, suggesting publication bias and an overestimation of the risk of developing esophageal cancer (0.5% per year).7 A subsequent meta-analysis of 57 studies (11,434 patients and 58,547 patient-years of follow-up evaluation) showed a lower pooled annual risk of 0.3% per year.8
Although evidence from observational studies starts out as low-quality evidence, rating up the quality of evidence may be appropriate in specific circumstances. For example, average-risk screening colonoscopy reduces the risk of CRC mortality but the risk of colonoscopy-related splenic rupture is less clear because only case reports are available. However, as the incidence of splenic rupture in the population not screened with colonoscopy approaches zero, the resulting estimate of relative effect for splenic rupture as a result of colonoscopy is likely large, increasing our confidence in the estimate of the adverse event (Table 1).
Outcomes critical for decision making determine the overall quality of evidence
Because competing management strategies will have both beneficial outcomes as well as undesirable effects, basing the overall quality of evidence for a recommendation solely on the beneficial outcomes would be inappropriate. For example, although there is higher-quality evidence from a meta-analysis of natalizumab in reducing the risk of failure to achieve remission in Crohn’s disease,9 the overall quality of evidence should be based on the lower quality of evidence for harm (progressive multifocal leukoencephalopathy), as long as the occurrence of progressive multifocal leukoencephalopathy was considered critical for decision making by the guideline panel.
Making recommendations
Although the overall quality of evidence across outcomes is the defining starting point for guideline construction, additional factors, such as the balance between desirable and undesirable effects and patients’ values and preferences, need explicit considerations and may change the strength, and even the direction, of recommendations. Finally, the uncertainty of whether the recommended course of action represents a wise use of resources may need consideration depending on the perspective of the guideline panel (Table 1).
Conclusions
The GRADE process is an internationally recognized method by which clinical practice guidelines may be developed taking into account not only the quality of the evidence surrounding a specific PICO question, but also the relative benefits and harms associated with competing management strategies, potential ambiguity in patient preferences for treatments and outcomes, and the health care resources necessary to implement interventions or strategies.
The ultimate goal is the creation of clinical practice guidelines that are based on the best existing evidence, are transparent in their construction, provide clear guidance to practicing physicians, and form the basis of metrics upon which we agree to be judged. The latter is what will be required of us going forward: documentation that we are delivering high-quality health care that is proven to improve the outcomes of patients. When we achieve these goals we should be rewarded appropriately. The AGA GRADE process will ensure that gastroenterologists remain in charge of our own destiny, and that we define the optimal manner in which to care for our patients.
