The news headlines have been full of accusations of bad science, fraud, and scandal in the world of medical research lately. In fact, one researcher published a paper showing that, statistically, there is a greater than 50-percent chance that any given scientific study is wrong.

Yet we increasingly rely on scientific studies to advance our knowledge in the fields of both conventional medicine and alternative health. Documented and reproducible scientific studies are there to help us sort the wheat from the chaff in the oftentimes contradictory world of health claims, and they can provide us with more reliable data than anecdote or folklore.

Fudging the Figures

In 2005 epidemiologist John Ioannidis’ research led him to the astonishing conclusion that “most claimed research findings are false.” Researcher bias, “manipulation in the analysis or reporting of findings,” whether intentional or subconscious, plays a key role in skewing the validity of study results. Ioannidis statistically analyzed the damaging effects of the following factors on the reliability of study results: small sample sizes or the study of tiny, isolated effects; sloppy study designs and definitions; and the presence of a profit motive or the protection of prestige. Ioannidis noted, too, that “the hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true,” because competing teams of scientists rush to publish their results. Alone or in combination, these factors can destroy the credibility of a scientific study.

Ioannides’ statistical study is not an isolated blast of the trumpet. In June 2005 Nature published an article with the provocative title, “Scientists behaving badly.” More than 3,400 US scientists, predominantly in the fields of biology, medicine, and chemistry, were asked whether they had participated in a range of 16 bad behaviours, including “falsifying or ‘cooking’ research data,” “not properly disclosing involvement in firms whose products are based on one’s own research,” and “changing the design, methodology or results of a study in response to pressure from funding sources.” Many of these inappropriate behaviours suggest researcher bias and diminish the likelihood that their studies’ findings are true.

While less than two percent of respondents reported engaging in practices such as plagiarism or outright falsification of data, overall, 33 percent of the scientists admitted that they had engaged in at least one of the bad behaviours in the last three years. More than a quarter of the respondents (27.5 percent) admitted to “inadequate record keeping related to research projects,” and almost 16 percent said they had modified a study because of pressure from funders. Ioannidies found that both these factors–too much flexibility in recordkeeping and the influence of financial interests–play a statistically significant role in reducing the verity of research results.

Faking It

What about real-life headlines? Perhaps the most well-known recent example was the scandal over falsified stem cell research. This is certainly a hot–even hyped–area of research, promising treatment for now-incurable conditions such as Parkinson’s disease and spinal cord injuries. In 2004 South Korean researcher Hwang Woo Suk published what appeared to be groundbreaking research on the cloning of human embryonic stem cells in the journal Science. It was later discovered that Hwang obtained cell donations from his own research team, and the research was faked with doctored photographs. Science has since retracted the study.

Either sloppy record-keeping or the need to justify spending hundreds of millions of dollars may have influenced analysists at the World Bank to publish what appeared to be manipulated statistical data. Canadian researcher Amir Attaran’s article in the Lancet (November 2004) alleged that the World Bank invented epidemiological statistics to support claims its malaria-control program made immense advances in reducing malaria deaths in India, in some areas by as much as 98 percent.

The World Bank refused to share its too-good-to-be-true data with Attaran, but he was able to obtain statistics from Indian health authorities. The raw data showed that the incidence of malaria deaths had actually increased, not decreased.

How to Lie with Statistics

Susan Safyan is an editor at alive magazine who doesn’t believe everything she reads.

Source: alive #287, September 2006