But a closer look reveals inconsistencies in how these criteria are applied. The heterogeneity is apparent in any random set of POEMs: Some authors list strengths and weaknesses by name with no elaboration, some give more details, some address only one criterion (eg, allocation concealment), while others state simply that the study was well designed. Some, like The Lancet review, say nothing about quality, reporting only the design and results. Eight (17%) of the 48 POEMs published in the first 6 months of 2000 included no critical appraisal (or a vague remark).
Similar omissions plague ACP Journal Club and the BMJ journals. Although the reviews describe concealment of allocation and blinding, and the commentary sections sometimes address design flaws at length, the degree to which this occurs, if at all, is variable. An ACP Journal Club review22 of the United Kingdom Prospective Diabetes Study, the landmark trial of intensive glycemic control, mentioned no concerns about its external validity (for contrast see the report by the American Academy of Family Physicians and American Diabetes Association23). Calling such synopses critical appraisals obfuscates the meaning of the term.
Some say that even brief remarks are critical appraisals. But good appraisals consistently and objectively rate studies using uniform criteria.24 POEMs and the EBM journals do not currently meet this standard; more systematic procedures are needed to ensure that narratives routinely discuss core elements of internal and external validity and that manuscripts lacking these elements are returned. The current conditions for preparing reviews make this difficult, however. With their modest budgets3 journals rely on hundreds of volunteer contributors, each with a different writing style and level of expertise. Onerous procedures for critical appraisal might discourage participation. Because journal space is limited, narratives are kept short (700 words for POEMs,25 425 words for the BMJ journals and ACP Journal Club20) to review more studies. Each issue of JFP contains 8 POEMs, and Evidence-Based Medicine has 24 reviews. Longer appraisals would reduce the number and currency of reviews and would be less concise for busy physicians.
The disadvantage of short reviews and rapid turnover is a greater risk of inaccuracies and imbalance. Careful analysis of a study requires more time to do research and more space to explain the results than journals currently provide. Authors have only weeks to prepare manuscripts, which leaves little time to verify that descriptions are accurate and give proportionate emphasis to the issues that matter most. For many studies a few hundred words provide inadequate room to fully explain the design, results, and limitations. The risk of mistakes is heightened with less time, analysis, expert review, and space.
This tradeoff between quantity and quality begs the question of what is more important to readers and to patient care: the number of studies that physicians know about or the accuracy with which they are described. POEMs, which are designed to change practice,26 can do harm if physicians acting on inaccurate or incomplete information make choices that compromise outcomes. The Lancet review is a shot across the bow. The 40,000 annual deaths from breast cancer13 remind us that for certain topics a mistaken inference can cost thousands of lives. If inconsistencies in appraisal make this happen often enough, efforts to synopsize evidence can do more harm than good. Also, it is incongruent for programs espousing EBM, discourages a discipline that accepts evidence on face value, to report studies with little or no discussion of validity.
Setting policy in critical appraisals
It is also antithetical for EBM to support evidence-based practice guidelines27 and to publish clinical advice that is not derived from these methods. Critical appraisals that conclude by suggesting how physicians should modify patient care cross the line from science to policy. In the first half of 2000 73% of the POEMs advised physicians (with varied explicitness) to use tests (6), drugs (14), or other treatments (5) and to withhold others (10). Such advice is common fare in the medical literature, but EBM ascribes to a higher standard. Because imprudent practice policies can do harm or compromise effectiveness, EBM holds that guidelines should be drafted with care using evidence-based methods.28 This entails reviewing not one study but all relevant evidence, with systematic grading of studies and explicit linkage between the recommendations and the quality of the data.27-30 This process typically involves an expert panel and months or years of deliberations.
In contrast, practice recommendations in EBM journals and POEMs reflect what individual authors think of a study. They lack the time, funding, and journal space for a systematic literature review. Thus, the authors and their readers cannot be sure that the conclusions reflect the evidence as a whole without undue influence from the reviewed study. Rules of evidence and grades for recommendations are rarely provided. Unlike guideline panels, authors seldom vet their recommendations with experts, societies, and agencies, which often uncover flawed inferences. The thinking process behind recommendations is necessarily telescoped. While the United States Preventive Services Task Force spent 2 years deciding whether pregnant women should be screened for bacterial vaginosis, a POEM31 produced its recommendations within weeks.