Specification Curve Analyses Results
Each analysis (or specification) yielded a hazard ratio for red meat exposure and death.
- The median HR was 0.94 (IQR, 0.83-1.05) for the effect of red meat on all-cause mortality — ie, not significant.
- The range of hazard ratios was large. They went from 0.51 — a 49% reduced risk for early mortality — to 1.75: a 75% increase in early mortality.
- Among all analyses, 36% yielded hazard ratios above 1.0 and 64% less than 1.0.
- As for statistical significance, defined as P ≤.05, only 4% (or 48 specifications) met this threshold. Zeraatkar reminded me that this is what you’d expect if unprocessed red meat has no effect on longevity.
- Of the 48 analyses deemed statistically significant, 40 indicated that red meat consumption reduced early death and eight indicated that eating red meat led to higher mortality.
- Nearly half the analyses yielded unexciting point estimates, with hazard ratios between 0.90 and 1.10.
Paradigm Changing
As a user of evidence, I find this a potentially paradigm-changing study. Observational studies far outnumber randomized trials. For many medical questions, observational data are all we have.
Now think about every observational study published. The authors tell you — post hoc — which method they used to analyze the data. The key point is that it is one method.
Dr. Zeraatkar and colleagues have shown that there are thousands of plausible ways to analyze the data, and this can lead to very different findings. In the specific question of red meat and mortality, their many analyses yielded a null result.
Now imagine other cases where the researchers did many analyses of a dataset and chose to publish only the significant ones. Observational studies are rarely preregistered, so a reader cannot know how a result would vary depending on analytic choices. A specification curve analysis of a dataset provides a much broader picture. In the case of red meat, you see some significant results, but the vast majority hover around null.
What about the difficulty in analyzing a dataset 1000 different ways? Dr. Zeraatkar told me that it is harder than just choosing one method, but it’s not impossible.
The main barrier to adopting this multiverse approach to data, she noted, was not the extra work but the entrenched belief among researchers that there is a best way to analyze data.
I hope you read this paper and think about it every time you read an observational study that finds a positive or negative association between two things. Ask: What if the researchers were as careful as Dr. Zeraatkar and colleagues and did multiple different analyses? Would the finding hold up to a series of plausible analytic choices?
Nutritional epidemiology would benefit greatly from this approach. But so would any observational study of an exposure and outcome. I suspect that the number of “positive” associations would diminish. And that would not be a bad thing.
Dr. Mandrola, a clinical electrophysiologist at Baptist Medical Associates, Louisville, Kentucky, disclosed no relevant financial relationships.
A version of this article appeared on Medscape.com.