Medical Science: Time to Remove the Blinders

Time to take the blinders offAround the globe, medicines regulators approve new drugs after reviewing data from clinical trials. It is these trial data that underlie assuring determinations, like those made by the US Food and Drug Administration (FDA), that a drug has been proven to be “safe and effective.” Typically such data are drawn from large, randomized controlled trials—the “gold standard” in evidence-based medicine. Because clinical trials are so central to the process by which a drug gets to the market, our supply of medicines appears to be based on a deep reliance on and trust in evidence.

But it is not the evidence we are trusting in. Indeed, it cannot be the evidence, for the very fact that neither you nor your doctor nor the most well-respected academic scientists can access the detailed data that the FDA gets. Regulators have historically treated drug trial data as secret—and the pharmaceutical industry, which sponsors the clinical trials that regulators evaluate, are under no legal obligation to share, either.

Regulation with data secrecy is the system that brought us Vioxx, Celebrex, Avandia, Lipitor, Paxil, and other stories of the past two decades.  If any lesson is to be drawn from these events, it is that regulators can err. As Ben Goldacre has noted in Bad Pharma, many of these problems were first detected by academics—not regulators, which often showed enormous reluctance to act—suggesting that the more eyes on the data, the sooner we develop more complete understandings about the effect of medicines, and the better off we all are.

Some may take issue with that view, and see the status quo as fundamentally acceptable so long as we can trust (or regain trust) that regulators’ evaluations are based on a close scrutiny of the evidence. This logic would suggest that past regulatory failures point to a need to strengthen regulators so they can do their job better. Clinical trial data secrecy is not the problem; it’s just a matter of resources.

One problem with the lets-strengthen-regulators view is that even if regulators were perfect, the fact is that they do not regulate the practice of medicine. As David Healy has written in Pharmageddon, when regulators approve a new drug for marketing, what occurs at this “magic moment” is the transformation of what was previously just a chemical into a legal medicine. But regulators do not and cannot dictate the beliefs that medical societies, doctors, or the general public hold about those medicines.

For example, professional guideline committees regularly form their own conclusions about the safety and effectiveness of medicines, and often draw on the results of studies that were not part of the marketing authorization application to regulators. In addition, some may want to know how the drug works for populations and diseases that have not been evaluated and approved by regulators. Here, even for those who wish to consult regulators, they will find regulators have nothing to say about such “off-label” use.

A more fundamental problem with the lets-strengthen-regulators view is that even if regulatory assessments improve and become more trustworthy in the future, many would still prefer to live in a world that actively supports the possibility of independent analysis.  Indeed, so important and valued is third-party independent evaluation of medicines that an entire enterprise has grown up around this activity: Evidence Based Medicine (EBM). And within EBM, systematic reviews—summaries of the evidence to establish how well a treatment works—are one of the most trusted tools.

For the most part, systematic reviewers do not meddle in regulatory affairs, and regulators rarely conduct systematic reviews. But what bears consideration is the fact that systematic reviewers and regulators share a common fundamental desire: to properly—and independently—evaluate the safety and effectiveness of medicines. Yet they go about their jobs in vastly different ways.

Consider the FDA. To evaluate medicines, the FDA requires manufacturers to submit massive dossiers containing detailed clinical study reports (CSRs) of randomized controlled trials that may run many hundreds or thousands of pages. In addition, the FDA may request electronic patient-level datasets to re-run statistical analyses, and make site visits during the trial to help ensure the integrity of trial data collection, even request original case report forms to ensure that investigators who collected the data inputted it correctly into the electronic patient-level database.

By contrast, systematic reviewers who wish to form an unbiased assessment of the safety and effectiveness of a medicine traditionally turn to peer-reviewed journal publications of clinical trials, reports that often are no more than 10 pages long each. In some cases, the totality of the “evidence” they may find for a particular trial is a 400-word long conference abstract. Worse yet, for many trials there is no public record in any form, as if the trial never took place.

While systematic reviewers have historically prided themselves on the most evidence-based approach to evidence synthesis, the FDA’s focus on thorough and detailed records suggests it is unlikely to trust the conclusions of even the highest quality systematic reviews that remain based on journal publications.

The case of the anti-influenza drug Tamiflu is illustrative. Tamiflu was approved in 1999 by the US Food and Drug Administration. Soon after, Tamiflu’s manufacturer Roche produced promotional material claiming Tamiflu reduces the risk of secondary complications by 45%. If true, this would be a major public health benefit. But the FDA—which had only concluded that Tamiflu reduces the time to first alleviation of influenza symptoms by about a day—wrote Roche warning them to cease making this “misleading” claim that is “not supported by substantial evidence.”

So far so good. But in 2003, a year or so before avian influenza H5N1 threw the world into a frenzy over the possibility of a pandemic, Roche pooled data from ten of its clinical trials—all trials it had previously submitted to the FDA—and concluded, once again, that Tamiflu reduces the risk of secondary complications of influenza. This time, however, the FDA did not act. Nor could it, because Roche’s new claim was published in the peer-reviewed Archives of Internal Medicine, and regulators do not regulate scientific journals.

Roche’s timing could not have been better. As concern grew over a future pandemic of influenza, governments began to prepare, and a key strategy was to stockpile Tamiflu (oseltamivir). Roche’s article in Archives of Internal Medicine was cited in the United States 2004 draft national pandemic plan: “The impacts of oseltamivir therapy on lower respiratory tract complications (LRTCs) of influenza and on influenza hospitalizations were calculated in a pooled analysis of 10 randomized placebo-controlled studies that included 3,591 adults and adolescents. Overall, 4.6 percent of oseltamivir treated persons had an LRTC of influenza infection compared with 10.3 percent of persons who received placebo – a 55 percent reduction (P<0.001).” Unsurprisingly, the Cochrane systematic review at the time (2006) came to the same conclusion because it trusted Roche’s journal article on its face value as well.

The US government was thus able to support its stockpiling decision based on claims the manufacturer made in a peer-reviewed journal article that the FDA had previously ruled to be misleading when printed in promotional material.

In the end, do Roche’s clinical trials prove that Tamiflu reduces the risk of serious complications of influenza as Roche contends? Or is the FDA right? Without access to the detailed clinical trial data, there is no way an independent third-party can know—and neither the FDA nor Roche are releasing the complete trial data (although Roche promised it would disclose its full study reports over three years ago).

Fortunately, there are increasingly loud calls, particularly from Europe, demanding a shift from a world built on blind trust to one where “trust and verify” is possible. Europeans are at the forefront of multiple active campaigns calling for access to data such as the AllTrials petition, the Berlin Declaration, and the BMJ Open Data Campaign. These campaigns are calling for a new era of transparency and accountability in medicine and public health policy.

Perhaps most impressive is a declaration last year from European regulators, clearly stating that “clinical trial data should not be considered commercial confidential information”. Signaling a real change, they have already put their words into action. Since November 2010, the European Medicines Agency (EMA, the EU counterpart to the FDA and Health Canada) has already released more than 1.6 million pages of clinical trial data, free of charge, to anyone who has asked for it. In addition, the EMA is currently drafting a policy on the prospective release of clinical trial data, a policy it intends to put into force in January 2014.

A Cochrane group I am a part of is among those that have applied for and received data under the EMA’s new policy. Changing our methods, we endeavoured to review clinical study reports instead of journal publications to form our conclusions. The result? A realization that almost all of the published Tamiflu analyses were faulty, based on comparing two unbalanced groups. We also found serious adverse events that the original study investigators classified as possibly due to Tamiflu—but went unnoted in the corresponding journal publication in the Journal of the American Medical Association. And as for Tamiflu’s supposed public health benefits? We found no good evidence that Tamiflu reduces the risk of hospitalizations, complications, or even interrupts transmission of the virus. While these are essentially the same conclusions the FDA drew when the blockbuster was approved 14 years ago, there’s no guarantee our conclusions—or the FDA’s—are correct, as it is unclear that any party except Roche has all the data.

The access-to-data landscape is changing, but it remains a volatile and contested terrain.  Earlier this year, AbbVie, InterMune UK and others sued EMA over its data releases. Details of the three cases remain scant—including what, precisely, was even requested—but the US and European pharmaceutical industry trade groups PhRMA and EFPIA have reportedly submitted supportive pleas, and the General Court of the European Union has ordered EMA not to provide the contested documents until a final ruling is given.

The status quo of evidence in medicine thus largely remains a system based on secrecy, with differential levels of access to data. How these disparities in access to data went unnoticed for so long is a good question to ponder. If we believe that claims—whether they be of the sponsor or of the regulator—should be open to independent evaluation, then we must work to change the norms of medical science from one of trusting in secrecy, to one of trusting in access to data.

About the Contributor

Peter Doshi is a postdoctoral fellow in comparative effectiveness research at Johns Hopkins University School of Medicine in Baltimore.  He is currently serving on advisory committees for the European Medicines Agency’s forthcoming policy on prospective release of clinical trial data. Peter can be reached at