Dr. John Bohannon, deliberate purveyor of junk science. | Image Credit: National Institutes of Health.
Earlier this year, a number of media outlets trumpeted the latest breakthrough in dietary science: “Eat dark chocolate daily to lose weight!” But what CBS News, Bild (Europe’s largest daily), Huffington Post, Shape magazine, Britain’s Daily Mail, and many others didn’t know, or didn’t bother to find out, is that the “new scientific finding” they were reporting on was all part of a dexterous ruse. It was the final stage in an experiment to expose the rigor, or lack thereof, in how complex nutritional science gets interpreted (or not) and subsequently broadcast by the media.
The fiendish mind behind the bunk study, John Bohannon, does, in fact, hold a PhD, but he studied bacterial biology. In revealing the plot this week on io9, he wrote that the motivation was to “make reporters and readers alike more skeptical.” His piece is almost 3,000 words, but it is worth the time.
While I have not earned a doctorate, nor am I a dietitian, I come from an empirical discipline, too, so I will try to briefly unpack the study before considering the larger problems exposed, and created, by Bohannon’s hoax.
The study used to promote the bunk result was real. In this case, “real” means that it was designed and performed: it involved participants, a physician, and a statistician. Beyond that, however, the entire thing was flawed. By design. Containing only 16 subjects, the participant pool was tiny. Investigators usually try to recruit large numbers of participants and to have an even spread across gender and age. These typical considerations were ignored. Subjects were divided into three test conditions: a low carb diet, a low carb diet plus 42 grams (1.5 ounces) of dark chocolate per day, and a control group, who received no alteration in their diet. 18 physiological factors were measured, which is entirely too many. Measuring so many dependent variables around one locus of interest (dark chocolate) is a bit like buying lottery tickets: buying more tickets increases your odds of scratching off a winner. Similarly, the more things you measure, to more likely something variable is going to change by some significant amount. This, and methods similar to it, are called “p-hacking,” and are a major problem in science.
When this sham study was completed, the deliberate mistakes continued to pile up. Some of the factors they measured, like body mass index and absolute body weight, dropped at a higher rate in the chocolate group than the others. But the result wasn’t enough. Bohannon and his team were interested in observing the media’s interaction with and reaction to this junk science, so they needed someone to publish the thing. Unfortunately, there are quite literally thousands of questionable outlets for questionable science. For 600 euros, the study was published in the International Archives of Medicine, a pay-to-publish open-access journal that recently changed hands. From submission to credit card number to publication, the process took only two weeks. There was no peer review, which is a hallmark of genuine academic journals.
Then Bohannon baited the hook: He did the journos’ work for them by crafting a press release about the study on the website of the freshly-minted and totally fictitious Institute of Diet and Health, written by a one Herr Doktor Professor Johannes Bohannon. There is no such scientist, obviously. Even more obvious, the IDH website still bears a WordPress logo and has just enough “content” to scroll across one time. Despite these red flags, Johannes’ press release, with a grabbing lede, “punchy quotes,” and a loaded nut graf, was apparently just too good to be ignored. Outlets that ran the story copied his quotes word for word. The few that reached out to “fact check” were more concerned about the spelling of his name than asking for the number of participants, p-values, or bothering to consult a third-party expert.
And so, in late March, millions of people all of the world in 20 countries and speaking 6 languages were presented with loud headlines declaring the latest and greatest news in dieting.
This whole “experiment” presents a whole range of other concerns, though. In a Web-connected world in which the 24-hour news beast must be fed constantly, is there any hope for Bohannon’s admission and other coverage about it to right the wrong? I didn’t hear anything about this debacle until after it was done; I came across a piece on NPR’s All Things Considered last week. But there are millions of people who have read about this junk science and thought it was true. They may never see a retraction or correction. Has another chocolate-diet monster been unleashed into the world? What’s more, there are dozens of ostensible “science journalists” who fell for the thing hook, line, and sinker!
I see a gamut of problems highlighted by Bohannon’s little jaunt, only some of which I will point to here: Scientists feel a very real pressure to “publish-or-perish,” which can promote even genuine academics to resort to p-hacking and other questionable methods of framing facts and findings. What’s more, “open access” pay-to-publish journals are hugely problematic in their own way. There are ethical concerns with Bohannon’s junk study, too; participants were not able to provide informed consent because they genuinely thought they were signing up for a an experiment meant to investigate chocolate’s role in weight loss. They were never fully debriefed. Simultaneously, a number of journalists and publications have been utterly embarrassed, which is a poor teaching mechanism and is unlikely to result in any real change in the nature of their scientific reporting. Additionally, science is hard. Quick, accurate reporting is hard, too. As I have written before, eating and nutrition are complex systems, yet there is a stubbornly tenacious tendency to draw arrow-straight lines between an ingredient and a reaction, whether it’s Chinese Restaurant Syndrome or accelerated weight loss.
Depending on who you ask, chocolate might be good for you or bad for you. Salt might be bad for you or good for you. Fat might be bad for you or good for you. Protein might be bad for you or good for you! What’s more, it’s hard to even know where to look or who to ask in the first place. (However, I do strongly suggest that you cut a wide path around the Food Babe.) Historically and contemporaneously, diet guidance is positively absurd (see: Atkins). We ought to look askance at anybody who identifies as a “nutritionist.” The term is not granted via certification and is subject to no legal regulation. Registered Dietitians aren’t necessarily any better prepared to field questions, either, since they tend to specialize in the medical extremes of helping morbidly obese people control their weight or people with eating disorders to achieve a healthy weight. Physicians and dietitians are not trained to offer advice to bros looking to “beef up” or to me, who is trying to shed a bit of stubborn belly fat.
Dietary science is exactly that: a science. And that’s hard. Being a scientific journalist is also difficult. It requires a background and education that adequately equips one to carefully read dense papers, to say nothing of the patience and wherewithal required to pursue such an undertaking when you’re working on a deadline in the publishing industry, which leans heavily on gossip and word of mouth.
Whether John Bohannon did a good journalistic thing or a bad scientific thing will take some time to discern. But he has promoted a productive and collaborative discussion, and that’s something that The Sexy Politico places enormous value on. For now, if you’d like to collaboratively discuss this with me, I’ll be at my desk with a Vosges bar to share.
