Non-publication of large randomized clinical trials: cross sectional analysis

Source: British Medical Journal

Full Text

 

Objective To estimate the frequency with which results of large randomized clinical trials registered with ClinicalTrials.gov are not available to the public.

Design Cross sectional analysis

Setting Trials with at least 500 participants that were prospectively registered with ClinicalTrials.gov and completed prior to January 2009.

Data sources PubMed, Google Scholar, and Embase were searched to identify published manuscripts containing trial results. The final literature search occurred in November 2012. Registry entries for unpublished trials were reviewed to determine whether results for these studies were available in the ClinicalTrials.gov results database.

Main outcome measures The frequency of non-publication of trial results and, among unpublished studies, the frequency with which results are unavailable in the ClinicalTrials.gov database.

Results Of 585 registered trials, 171 (29%) remained unpublished. These 171 unpublished trials had an estimated total enrollment of 299 763 study participants. The median time between study completion and the final literature search was 60 months for unpublished trials. Non-publication was more common among trials that received industry funding (150/468, 32%) than those that did not (21/117, 18%), P=0.003. Of the 171 unpublished trials, 133 (78%) had no results available in ClinicalTrials.gov.

Conclusions Among this group of large clinical trials, non-publication of results was common and the availability of results in the ClinicalTrials.gov database was limited. A substantial number of study participants were exposed to the risks of trial participation without the societal benefits that accompany the dissemination of trial results.

Judgements about the quality of evidence

Judgements about the quality of evidence

A short glossary is included at the bottom of this page. For a more complete glossary click here: Glossary of terms.

In making health care treatment and delivery decisions, policymakers, patients and clinicians must trade off the benefits and downsides of alternative strategies. Decision-makers will be influenced not only by the best estimates of the expected advantages and disadvantages, but also by their confidence in these estimates; i.e. the quality of the evidence. The GRADE system, which we have used, provides a structured and transparent system for making judgements about the quality of evidence.¹

Using the GRADE system, we have made separate ratings of evidence quality for each important outcome. Like early systems of grading the quality of evidence, the GRADE system begins with the study design. Randomised trials provide, in general, stronger evidence than observational studies. Therefore, randomised trials without important limitations constitute high quality evidence. Observational studies without special strengths generally provide low quality evidence. However, there are a number of factors that can reduce or increase our confidence in estimates of effect.

The GRADE system considers five factors that can lower the quality of the evidence:

1. Study limitations
2. Inconsistent results across studies
3. Indirectness of the evidence
4. Imprecision
5. Publication bias

and three that can increase the quality of evidence:

1. Large estimates of treatment effect
2. A dose-response gradient
3. Plausible confounding that would increase confidence in an estimate

Factors that can lower the quality of evidence

1. Our confidence in estimates of effects decreases if studies suffer from major limitations that may bias their estimates of the treatment effect. These limitations include, for example, lack of allocation concealment; lack of blinding, particularly if outcomes are subjective and their assessment highly susceptible to bias; a large loss to follow-up; failure to adhere to an intention-to-treat analysis; or failure to report outcomes (typically those for which no effect was observed).
2. Widely differing estimates of effects across studies for which there are no compelling explanations reduces our confidence in knowing what the true effect is. Variability may arise from differences in populations (e.g., drugs may have larger relative effects in sicker populations); from differences in interventions (e.g., larger effects with higher drug doses); or outcomes (e.g., diminishing treatment effect with time). When variability exists but investigators fail to identify a plausible explanation, the quality of evidence decreases.
3. Decision-makers must consider two types of indirectness that can lower the quality of evidence. The first occurs when considering, for instance, use of one of two active drugs, A and B. Although randomized comparisons of A and B may be unavailable, randomized trials may have compared A to placebo and B to placebo. Such trials allow indirect comparisons of the magnitude of effect of A and B. Such evidence is of lower quality than head-to-head comparisons of A and B would provide. The second type of indirectness includes differences between the population, intervention, comparator to the intervention, and outcome of interest, and those included in the relevant studies. Most important here is consideration of whether important outcomes are measured directly or surrogate outcomes are used, such as a biochemical or process measure that may or may not accurately reflect what can be expected in terms of an important outcome, such as mortality or morbidity.
4. When studies include relatively few patients and few events and thus have wide confidence intervals (or a large p-value), we are less confident in an estimate.
5. The quality of evidence will be reduced if there is a high likelihood that some studies have not been reported (typically those that show no effect). The risk of such a “publication bias” is greater when published evidence is limited to a small number of trials, all of which are sponsored by people with a vested interest in the results, such as the pharmaceutical industry.

Factors that can increase the quality of evidence

1. Even well-done observational studies generally yield only low-quality evidence, because of the many potential confounders that either are not known or are not measured. However, occasionally they may provide moderate or even high quality evidence. The larger the magnitude of effect, the less likely it is that this could be explained by confounders and the evidence is, thus, stronger. For example, a meta-analysis of observational studies found that bicycle helmets reduce the odds of head injuries in cyclists involved in a crash by about two-thirds, an effect that could not easily be explained by confounders, given the design of the studies.
2. The presence of a dose-response gradient can increase our confidence in estimates of effects, for example if larger effects are associated with larger doses, as might be expected.
3. When an effect is found, if all plausible confounding would decrease the magnitude of effect, this increases the quality of the evidence, since we can be more confident that an effect is at least as large as the estimate and may be even larger. Conversely, particularly for questions of safety, if little or no effect is found and all plausible biases would lead towards overestimating an effect, we can be more confident that there is unlikely to be an important effect.
   GRADE provides a clearly articulated and comprehensive methodology for rating and summarising the quality of evidence supporting treatment and health care delivery recommendations. Although judgements will always be required for every step, the systematic and transparent GRADE approach allows scrutiny of and debate about those judgements.

[1] Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, and the GRADE Working Group. What is ‘quality of evidence’ and why is it important to clinicians? BMJ 2008; 336:995-8. 

Short glossary (for a more complete glossary click here: Glossary of terms)

Allocation concealment
The process used to ensure that the person deciding to enter a participant into a randomised controlled trial does
not know the comparison group into which that individual will be allocated.

Blinding
The process of preventing those involved in a trial from knowing to which comparison group a particular participant
belongs.

Bias
A systematic error or deviation in results or inferences from the truth.

Confidence interval
A measure of the uncertainty around the main finding of a statistical analysis.

Confounder
A factor that is associated with both an intervention (or exposure) and the outcome of interest. For example, if people in the experimental group of a controlled trial are younger than those in the control group, it will be difficult to decide whether a lower risk of death in one group is due to the intervention or the difference in ages. Age is then said to be a confounder, or a confounding variable.  Randomisation is used to minimise imbalances in confounding variables between experimental and control groups. Confounding is a major concern in non-randomised studies.

Intention-to-treat
A strategy for analysing data from a randomised controlled trial. All participants are included in the arm to which they were allocated, whether or not they received (or completed) the intervention given to that arm.

Loss to follow-up
The loss of participants during the course of a study.

Observational study
A study in which the investigators do not seek to intervene and simply observe the course of events. Changes or differences in one characteristic (e.g. whether or not people received the intervention of interest) are studied in relation to changes or differences in other characteristic(s) (e.g. whether or not they died), without action by the investigator.

Randomised trial
An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials one intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).

Related articles

• No magic answer for Achilles tendinopathy.
• Confirmation bias
• "A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null"
• Fox News Fact Check: Are studies on maternal employment all tinged with political bias?
• The mystery of the missing experiments
• Do Improved Social Signals Cause Improved Rankings? | SEOmoz
• Just How Effective Are Sales Incentive Programmes? Victor Tardieu Examines Full Circle Motivation's Case Studies And The Evidence From The IRF ROI Measures
• Too Impressed by their own Cleverness?
• Can Science Be Trusted?
• 'Evidence-based' only proves it worked in the past

Death to Elsevier! : Pharyngula

Death to Elsevier! : Pharyngula:

'via Blog this'

From scienceblogs.com deals with the boicot to Elsevier as well. I wonder who is behind Elsevier, few years ago, Rafael Bravo, a relevant family doctor from Spain, wrote in an spanish forum about Elsevier was selling its bonus which was in the military industry. Elsevier has the control of the major primary care journals in spanish, like The Lancet issue in Spanish, and other well recognized journals like Atencion Primaria, one of the few journals write in spanish and you can find in Medline.
Who are behind this Journals ? But is not only an Elselvier issue, the question is who is behind each publication we read. We now about the publication`bias, but what about the authors, or even the Universities where more recognized researches are writing. Everybody knows that "public or perish" is a dogma in USA, but can you research and publish whatever ? Of course not, Universities have "sponsors" as well (Rockeffeller Foundation, Kellogs Foundation, and so on...). Do you think this people is free to publish everything they know ?  I don`t think so. But the same people which invest in Wall Street, do the same in weapons, publishing editorials, big pharma, and so on.....so you can imagine a free science ? Hard to believe, we are eating only the information they fed us, and mass media share this to regular people all over the world.
Something is bad, and governments have to regulate this, but what kind of regulation you could expect from Obama Administration wich government is deeping the George Bush legacy ? The war in Irak has finished, but they were more rude with Wikileaks or Megaupload, talking about terrorism just for a bussiness where people just put their files as a back up (it was my case), and now the government try to destroy all this information ?
To destroy files is not like destroy books in the Nazi`s Germany ? I think so. But I`m sorry the topic was about Elsevier. 

Missing clinical trial data

EDITORIAL            

                                         Image via Wikipedia

BMJ 2012; 344 doi: 10.1136/bmj.d8158 (Published 3 January 2012)

Cite this as: BMJ 2012;344:d8158

• Article
• Related content
• Article metrics

1. Richard Lehman, senior research fellow1, 
2. Elizabeth Loder, clinical epidemiology editor2

Author Affiliations

1. eloder@bmj.com

A threat to the integrity of evidence based medicine

Clinical medicine involves making decisions under uncertainty. Clinical research aims to reduce this uncertainty, usually by performing experiments on groups of people who consent to run the risks of such trials in the belief that the resulting knowledge will benefit others. Most clinicians assume that the complex regulatory systems that govern human research ensure that this knowledge is relevant, reliable, and properly disseminated. It generally comes as a shock to clinicians, and certainly to the public, to learn that this is far from the case.

The linked cluster of papers on unpublished evidence should reinforce this sense of shock. These articles confirm the fact that a large proportion of evidence from human trials is unreported, and much of what is reported is done so inadequately. We are not dealing here with trial design, hidden bias, or problems of data analysis—we are talking simply about the absence of the data. And this is no academic matter, because missing data about harm in trials can harm patients, and incomplete data about benefit can lead to futile costs to health systems. Moreover, researchers or others who deliberately conceal trial results have breached their ethical duty to trial participants.

The linked articles look closely at the extent, causes, and consequences of unpublished evidence from clinical trials. Hart and colleagues incorporated unpublished evidence into existing meta-analyses of nine drugs approved by the US Food and Drug Administration in 2001 and 2002.1 These reanalyses produced identical estimates of drug efficacy in just three of 41 cases (7%); in the remaining cases, estimates of drug efficacy were evenly split between more (19/41) and less (19/41). It is sometimes assumed that incorporation of missing data will reduce estimates of drug benefits, but this study shows that “publication bias” can cut both ways. Each increment of data can change the overall picture, but in most cases with no certainty that the picture is complete.

A fundamental step towards tackling this problem was taken in 2005, when, as Chan describes in the Research Methods and Reporting section, prior registration of all trials became a condition for later publication.2 Chan details the ways in which authors of systematic reviews can search for unpublished evidence, and he strikes an optimistic note when he states that “Key stakeholders—including medical journal editors, legislators, and funding agencies—provide enforcement mechanisms that have greatly improved adherence to registration practices.”

However, two studies we publish give little cause for optimism that this adherence extends to timely sharing of trial results. A survey of publicly funded research in the United States between 2005 and 2008 by Ross and colleagues shows that registration is not followed by reporting of summary results within 30 months of completion in more than half of trials.3 Even at three years, one third remain unpublished. The US Food and Drug Administration Amendments Act of 2007 made publication of a results summary on ClinicalTrials.gov within 12 months mandatory for all eligible trials in the US “initiated or ongoing as of September 2007”—Prayle and colleagues examine the extent to which this has happened.4 The tally stands at 22%. When the word “mandatory” turns out to mandate so little, the need for stronger mechanisms of enforcement becomes very clear.

Most clinical interventions in current use, however, are based on trials carried out before the era of mandatory registration, and here the task of data retrieval by systematic reviewers and national advisory bodies becomes impossible. Wieseler and colleagues show that the different documents available to researchers and regulators—internally produced study reports, study findings published in peer reviewed journals, and results posted in results registries—supplement each other, but that reporting quality is highest in study reports. However, the effort required to find and collate these sources can be prodigious and seldom guarantees completeness.5 In their just published Cochrane review update on antiviral treatments for influenza, Jefferson and colleagues describe a painstaking search for information from undisclosed trials stretching over several years.6

There is an “Alice in Wonderland” feel to these investigators’ efforts—acting on the public’s behalf, searching over hill and dale and among the paperwork of regulatory bodies and drug companies to put together pieces of data that should have been freely available in the first place. Even when data on individual participants are made available, they only form part of the jigsaw, and Ahmed and colleagues describe the problems of fitting in such data when the whole picture is not known.7

Finally, to find the randomised clinical trials that have been published in the medical literature, nearly every student, clinician, or researcher turns first to Medline among the biomedical databases. But Wieland and colleagues find that many reports of randomised controlled trials entered into Medline between 2006 and 2011 have not been indexed as such; thus, simply entering the search term “randomised controlled trial” into this database will miss many of these trials, despite the best efforts of the Cochrane Collaboration and the US National Library of Medicine.8

What is clear from the linked studies is that past failures to ensure proper regulation and registration of clinical trials, and a current culture of haphazard publication and incomplete data disclosure, make the proper analysis of the harms and benefits of common interventions almost impossible for systematic reviewers. Our patients will have to live with the consequences of these failures for many years to come. Retrospective disclosure of full individual participant data would be an important first step towards better understanding of the benefits and harms of many kinds of treatment. A model for this is provided by Medtronic’s recent agreement to release full individual participant data relating to its controversial bone product—recombinant human bone morphogenetic protein-2—to independent analysis teams; so there is no longer any convincing reason for other companies to refuse similar disclosure of de-identified participant data from all past trials.9

The main challenge is to ensure better systems for the future. Because “the optimal systematic review would have complete information about every trial—the full protocol, final study report, raw dataset, and any journal publications and regulatory submissions,”2 10 a prospective system of research governance should insist on nothing less. This may require the global organisation of a suitable shared database for all raw data from human trials—an obvious next step for the World Health Organization after its excellent work on the International Clinical Trials Registry Platform Search Portal. Concealment of data should be regarded as the serious ethical breach that it is, and clinical researchers who fail to disclose data should be subject to disciplinary action by professional organisations. This may achieve quicker results than legislation in individual countries, although this is also desirable.

These changes have been long called for,11 and delay has already caused harm. The evidence we publish shows that the current situation is a disservice to research participants, patients, health systems, and the whole endeavour of clinical medicine.

Notes

Cite this as: BMJ 2012;344:d8158

Footnotes

• Research, doi:10.1136/bmj.d7202
• Research Methods and Reporting, doi:10.1136/bmj.d8013
• Research, doi:10.1136/bmj.d7292
• Research, doi:10.1136/bmj.d7373
• Research, doi:10.1136/bmj.d8141
• Research, doi:10.1136/bmj.d7501
• Research, doi:10.1136/bmj.d7762
• Competing interests: Both authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
• Provenance and peer review: Commissioned; not externally peer reviewed.

References

1. ↵
   Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ2012;344:d7202.
2. ↵
   Chan A-W. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ2012;344:d8013.
3. ↵
   Ross JS, Tse T, Zarin DA, Hui X, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.BMJ2012;344:d7292.
4. ↵
   Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study.BMJ2012;344:d7373.
5. ↵
   Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on the quality of reporting of clinical drug trials: a comparison of registry reports, clinical study reports and journal publications.BMJ2012;344:d8141.
6. ↵
   Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev2011;12:CD008965.
7. ↵
   Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ2012;344:d7762.
8. ↵
   Wieland SL, Robinson KA, Dickersin K. Understanding why evidence from randomised clinical trials may not be retrieved from Medline: comparison of indexed and non-indexed records. BMJ2012;344:d7501.
9. ↵
   Krumholz HM, Ross JS. A model for dissemination and independent analysis of industry data. JAMA2011;306:1593-4.

   FREE Full Text
10. ↵
    Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective drugs: who can do what it takes? BMJ2011;342:c7258. 

    FREE Full Text
11. ↵
    Chalmers I. Underreporting research is scientific misconduct.JAMA1990;263:1405-8.

    FREE Full Text