Flip-Flop Medicine

“Contradicted and Initially Stronger Effects in Highly Cited Clinical Research” by John P. A. Ioannidis, in The Journal of the American Medical Association (July 13, 2005), 515 N. State St., Chicago, Ill. 60610.

One day it’s horrible for your health to let a drop of alcohol pass your lips; the next, you’re told that a glass of red wine is just what the doctor ordered. So it’s gone lately, with one highly publicized medical study after another contradicted or reversed.

This is no laughing matter. In 1991, the Nurses’ Health Study found that women receiving hormone therapy (estrogen and progestin) enjoyed a big (44 percent) reduction in the risk of coronary artery disease, and millions of women were encouraged to begin the therapy to counteract the effects of menopause. But in 2002, the Women’s Health Initiative produced a radically different conclusion: Hormone therapy increases the risk of coronary events in postmenopausal women by 29 percent. A subsequent study confirmed that result.

The explanation, according to Ioannidis, who teaches at the University of Ioannina School of Medicine in Greece and Tufts-New England Medical Center, is that the first study was not based on a random sample of the population. A “randomized” sample reflects various factors, known or unknown, that might be involved in the body’s reaction to the thing being studied. (Why aren’t all studies randomized? Cost is not the only explanation; ethical and other considerations are sometimes involved.)

But randomization alone does not assure valid results. Ioannidis isolated 45 widely cited clinical studies from the medical literature between 1990 and 2003. Six of the original 45 articles were based on non-randomized trials, and five of the six were later challenged—a very high error rate.

The other 39 studies were all based on random samples, yet nine were nevertheless challenged. The reason? For the most part, the sample sizes were smaller than in subsequent studies.

“The examination of contradictions and refutations offers a fascinating look at the process of science” as new studies appear over the years, writes Ioannidis. In an age of instant publicity, however, a surprising study can make global headlines before that process has a chance to run its course.