To some such pronouncements are not guidelines, only the bottom line of a review. But in policy terms it matters little whether physicians prescribe a drug because of a guideline or because of the advice they read in ACP Journal Club or a POEM. The outcome for the patient is the same. Reviews need a bottom line, but summarizing the results of a study (eg, drug A worked better than drug B) differs from advising physicians what to do (eg, prescribe drug A). The latter is a statement of policy rather than science and should be based on broader considerations than one study.28
EBM faults guidelines that omit evidence-based methods, such as those issued by advocacy groups that reflect personal opinions and selective use of studies more than systematic reviews.32,33 Yet the recommendations in EBM journals and POEMs differ little in appearance: They provide little documentation of how conclusions were reached, feature select evidence (the study under review), rely on authors’ opinions, and provide few details on rationale. EBM journals should extract themselves from this inconsistency by sharpening the distinction between summarizing evidence and setting policy and eschewing the latter unless it emanates from evidence-based methods.
How this is handled in POEMs will reflect on family medicine. The prominence the specialty has given POEMs (promotion in JFP, family practice literature,34-35 the Internet,36-37 and newsletters19) signals the way family physicians think studies should be reviewed. It is important to get this right. If POEMs are meant to be critical appraisals and 17% contain no critique, calling them critical appraisals casts doubts on the specialty’s understanding of the term and perpetuates confusion about definitions. Conversely, by instituting greater scrutiny—defining the core criteria that must be discussed to qualify a study commentary as a critical appraisal and systematizing their use in POEMs—the specialty would set a new standard for EBM. If POEMs are not meant to be critical appraisals, it is important to clarify the distinction in terms, especially for family physicians who have grown accustomed to POEMs, know little about alternatives, and have come to believe that POEMs, critical appraisals, and EBM are essentially the same.
Conclusions
Advocates of EBM should be systematic in their application of critical appraisal. Critical appraisals do not deserve the name if they accept studies on face value. The criteria for determining which studies are rated good or bad should be explicit and consistent. But the scrutiny of evidence should not be taken to extremes, to the point that studies are rejected for being imperfect when there is little likelihood that the findings are wrong. By making the perfect the enemy of the good, excesses in critical appraisal do injustice to the goal of helping patients and imply existence of a level of certainty that science cannot provide.