Why A Journalist Scammed The Media Into Spreading Bad Chocolate Science

Earlier this spring, headlines around the world trumpeted an exciting bit of news that seemed too good to be true: “Eating chocolate … can even help you LOSE weight!” as Britain’s Daily Mail put it.

From India to Australia and Texas to Germany, news organizations shared findings published in the International Archives of Medicine in late March.

The problem? The study they were based on was pure junk. And the person behind it, John Bohannon, would be the first to tell you that.

In fact, that’s exactly what he did Wednesday, in a story for io9 that’s gone viral. Bohannon, a science journalist who also has a Ph.D., lays out how he carried out an elaborate hoax to expose just how easily bad nutrition science gets disseminated in the mainstream media.

Diet science, Bohannon stresses, is still science – and reporters need to know how to cover it. “You have to know how to read a scientific paper — and actually bother to do it,” he writes. “For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.”

To be clear, the study involved was real — a randomized controlled trial. Bohannon and his partners, a German television reporter and his collaborator, really did recruit 16 people for a study on dieting. And they found that the ones who followed a low-carb diet and also ate a 1.5-ounce bar of dark chocolate daily lost weight faster than the control group that was dieting alone.

So what’s the problem, you ask? There are plenty.

For starters, as Bohannon explains in great detail, the study design itself was flawed — it had too few subjects, and the research measured too many factors, making it likelier that some random factor would appear to have statistical significance. (Bohannon does an excellent job of explaining the specifics himself.)

Then there’s the journal that published it — a so-called pay-for-play publication, which failed to carry out peer review of the findings. As Bohannon himself exposed in another sting for the journal Science a couple of years ago, there are lots of these publications that will publish bad research for a fee.

And finally, none of the reporters who covered it asked an outside expert to weigh in on the research – standard operating procedure in good science journalism. If they had, an astute scientist would have spotted the problems with the study design immediately. Indeed, just this week, our sister blog, Shots, ran a great post by Tara Haelle explaining how this process of tapping independent experts helped steer her from reporting on a flawed study suggesting that a childhood vaccine prevents leukemia.

Bohannon’s full story is long, but worth the read as an explanation of the pitfalls that plague science communication – especially nutrition information – in today’s media climate.

And the problem is massive, says Gary Schwitzer, the publisher of Health News Review. For the past nine years, the site has been dedicated to critiquing the media’s coverage of health in an effort to improve it.

“He’s really only scratching the surface of a much broader, much deeper problem,” Schwitzer says. “We have examples of journalists reporting on a study that was never done. We have news releases from medical journals, academic institutions and industry that mislead journalists, who then mislead the public.” And the pressure to publish or perish, he says, can lead well-intentioned scientists to frame their work in ways that aren’t completely accurate or balanced or supported by the facts.

“We are really mired in a mess, the boundaries of which few people really have a sense for,” says Schwitzer.

Schwitzer says it’s appropriate that Bohannon chose chocolate for his diet study stunt. From claims that chocolate can boost memory to press releases that oversell its benefit to heart health, the sweet stuff is the subject of many of the news stories Schwitzer’s site criticizes. (To be clear, there is good research suggesting benefits from eating chocolate — but Bohannon’s study isn’t it.)

But does Bohannon’s stunt reveal the credibility crisis in nutrition science journalism or add to it?

“I think it’s a little of both,” Schwitzer says.

Schwitzer is a fan of Bohannon’s message – but he worries that it will get lost in our rapid churn, 24-hour news cycle, and the journalists who need to be schooled in proper nutrition science reporting will have already moved on to the next thing.

And there are others who aren’t taking kindly to Bohannon’s work. On Twitter last night, some scientists raised concerns about the ethics of running a study with human subjects with the ultimate intent to deceive, and whether participants could give informed consent if they didn’t know that aim.

Source: npr

Advertisement