Combating bad science in healthcare

Tangentially apropos of my prior post riff on "big data." During his TED talk on "evidence based medicine," Dr. Ben Goldacre alludes to "the first published clinical trial":

Daniel 1: 11-16
11: Then said Daniel to Melzar, whom the prince of the eunuchs had set over Daniel, Hananiah, Mishael, and Azariah, 12: Prove thy servants, I beseech thee, ten days; and let them give us pulse to eat, and water to drink. 13: Then let our countenances be looked upon before thee, and the countenance of the children that eat of the portion of the king's meat: and as thou seest, deal with thy servants.

14: So he consented to them in this matter, and proved them ten days. 15: And at the end of ten days their countenances appeared fairer and fatter in flesh than all the children which did eat the portion of the king's meat. 16: Thus Melzar took away the portion of their meat, and the wine that they should drink; and gave them pulse.

Nice presentation. Good stuff on the gaming that results in "publication bias."

"We need to have all of the data on a particular treatment to know whether or not it really is effective. And there are two different ways that you can spot whether some data has gone missing in action. You can use statistics, or you can use stories. I personally prefer statistics, so that's what I'm going to do first.

This is something called funnel plot. And a funnel plot is a very clever way of spotting if small negative trials have disappeared, have gone missing in action. So this is a graph of all of the trials that have been done on a particular treatment. And as you go up towards the top of the graph, what you see is each dot is a trial. And as you go up, those are the bigger trials, so they've got less error in them. So they're less likely to be randomly false positives, randomly false negatives. So they all cluster together. The big trials are closer to the true answer. Then as you go further down at the bottom, what you can see is, over on this side, the spurious false negatives, and over on this side, the spurious false positives. If there is publication bias, if small negative trials have gone missing in action, you can see it on one of these graphs. So you can see here that the small negative trials that should be on the bottom left have disappeared. This is a graph demonstrating the presence of publication bias in studies of publication bias..."

Original TED site link for this talk here.

Also on the subject of medical science:

Lies, Damned Lies, and Medical Science
Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice? Dr. John Ioannidis has spent his career challenging his peers by exposing their bad science.

...[Ioannidis is] what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem...

From Science Based Medicine:

What Should We Do in the Absence of Evidence?
Posted by Mark Crislip

...My topic is “What Should We Do in the Absence of Evidence?” It is often said that only 15% of medical practice is evidenced based. It is a myth, and an outdated myth at that, based on a survey from 1961. Like all myths it has staying power and people like to repeat the factoid without bothering to see if it is actually true. It isn’t.

The real data suggests we do a pretty good job at following evidence as long as we are not TV doctors:

Thus, published results show an average of 37.02% of interventions are supported by RCT (median = 38%). They show an average of 76% of interventions are supported by some form of compelling evidence (median = 78%).

There is variability depending on the specialty and how you judge the evidence. Anesthesiologists do the best, but then they have the least to do.

There is always evidence. There are more than 23,000,000 citations on Pubmed and 799,000,000 hits searching ‘medicine’ on Google. It is not a lack of evidence that is the problem. It is more likely that the practitioner doesn’t know the evidence or the evidence may suck...

And, this just in...

An Award-Winning Cancer Researcher Says U.S. Science Has Never Been More Imperiled

WASHINGTON -- Around 6:30 p.m. on Wednesday, 150 or so people gathered in an otherwise empty National Press Club in downtown D.C.

Hours earlier, in the room down the hall, Sen. Ted Cruz (R-Texas) had drawn throngs of press during an appearance before the Hispanic Chamber of Commerce. Now, none remained. Instead, attendees still in their work attire sat around tables sipping wine and eating moderately moist chicken dinners, waiting to hear from the guest of the night, a doctor from the Boston Children's Hospital whom few in D.C. -- outside those walls -- knew of.

Dr. Frederick Alt, a 66-year-old Harvard professor of genetics, is responsible for some of the most consequential breakthroughs in cancer research. Sen. Ed Markey (D-Mass.), the keynote speaker who preceded Alt onstage, described him as a "luminary in the constellation of cancer fighters."

That night, Alt was receiving the Szent-Györgyi Prize from the National Foundation for Cancer Research for a twofold breakthrough. Decades ago, Alt upended conventional wisdom of human genome behavior when he discovered that cancer cells had the capacity to genetically amplify themselves, allowing them to spread, become more dangerous and resist treatments. From there, he discovered how chromosomes recognize the "machinery" that keeps their genomes stable -- a machinery that cancer cells lack. That led to a better understanding of how to protect DNA from the sort of critical damage caused by many cancers. It was research that another speaker called "foundational."

Now on the downward arc of his four-decade career, Alt could have been excused if, on Wednesday, he enjoyed a glass or two of wine and took his award in stride: the compliments of an organization whose mission aligns with his research. But the undertone of the evening's affair was more political, and more dour, than that.

Alt isn't just invested in understanding the genetics of cancer. He's preoccupied by the idea that the next generation of American scientists might not be there to take the baton he's passing.

"I think now is the worst I have ever seen it," he told The Huffington Post of the funding climate, hours before the award ceremony. "The biggest worry is that science as we’ve known it for the past many decades, where we are the leaders -- it is going to disappear if this keeps going.”...

__

UPDATE

Open Science
Published by Steven Novella under General Science

here is a movement to open access to scientific information, and with the relatively new resources provided by the internet and social media, we may be heading rapidly in that direction. However, I don’t think this will be an easy transition and we should consider the possible unintended consequences...

I agree that it is time to reconsider the entire infrastructure of how scientific research is documented and reported. While scientists and scientific institutions have been utilizing internet and social media resources, it has been ad hoc without a coherent widely accepted plan. That may not be a bad thing, to let systems develop organically, allow experimentation and see what shakes out. The experiment may have gone on long enough, however, to step back, see what we learned, and think about how to craft an optimal infrastructure...

In our “scientific utopia” scientific research would be completely transparent, high quality, free from distorting biases, and easy to search and access. The recommendations for how to get there, however, may have some unintended consequences. For me the big one is the watering down of peer review as a barrier to access to scientific data...

The two main negative consequences to consider are: overwhelming scientists with information, and overwhelming the public with information. A recent study shows this is already happening. The authors found that the number and lifetime of citations to scientific research is decreasing, suggesting that there are simply too many published studies for scientists to keep up with it all.

Lowering barriers to publication may also create an incentive for academics to publish more lower quality studies, which may be counterproductive to the advancement of science. Already we have a situation where a flood of preliminary studies are mostly wrong. More such studies may not be a good thing.

The media is also having an increasingly hard time sorting through which scientific studies are significant and which aren’t. I actually think that many reporters, especially those who are not specialist trained as science journalists, don’t care. If they had a larger body of speculative studies to sort through, looking for sexy headlines, that would exacerbate the problem of misinforming the public about the status of scientific questions, further confusing their understanding of science, and perhaps reducing their trust in science as the promises of sexy headlines never materialize or contradict each other.

Right now, at least, there is the barrier of peer-review (which is imperfect and incomplete). Without that barrier we will have the wild west, with all its good and bad aspects...

___

More to come...