Judgements about the quality of evidence

Judgements about the quality of evidence

A short glossary is included at the bottom of this page. For a more complete glossary click here: Glossary of terms.

In making health care treatment and delivery decisions, policymakers, patients and clinicians must trade off the benefits and downsides of alternative strategies. Decision-makers will be influenced not only by the best estimates of the expected advantages and disadvantages, but also by their confidence in these estimates; i.e. the quality of the evidence. The GRADE system, which we have used, provides a structured and transparent system for making judgements about the quality of evidence.¹

Using the GRADE system, we have made separate ratings of evidence quality for each important outcome. Like early systems of grading the quality of evidence, the GRADE system begins with the study design. Randomised trials provide, in general, stronger evidence than observational studies. Therefore, randomised trials without important limitations constitute high quality evidence. Observational studies without special strengths generally provide low quality evidence. However, there are a number of factors that can reduce or increase our confidence in estimates of effect.

The GRADE system considers five factors that can lower the quality of the evidence:

1. Study limitations
2. Inconsistent results across studies
3. Indirectness of the evidence
4. Imprecision
5. Publication bias

and three that can increase the quality of evidence:

1. Large estimates of treatment effect
2. A dose-response gradient
3. Plausible confounding that would increase confidence in an estimate

Factors that can lower the quality of evidence

1. Our confidence in estimates of effects decreases if studies suffer from major limitations that may bias their estimates of the treatment effect. These limitations include, for example, lack of allocation concealment; lack of blinding, particularly if outcomes are subjective and their assessment highly susceptible to bias; a large loss to follow-up; failure to adhere to an intention-to-treat analysis; or failure to report outcomes (typically those for which no effect was observed).
2. Widely differing estimates of effects across studies for which there are no compelling explanations reduces our confidence in knowing what the true effect is. Variability may arise from differences in populations (e.g., drugs may have larger relative effects in sicker populations); from differences in interventions (e.g., larger effects with higher drug doses); or outcomes (e.g., diminishing treatment effect with time). When variability exists but investigators fail to identify a plausible explanation, the quality of evidence decreases.
3. Decision-makers must consider two types of indirectness that can lower the quality of evidence. The first occurs when considering, for instance, use of one of two active drugs, A and B. Although randomized comparisons of A and B may be unavailable, randomized trials may have compared A to placebo and B to placebo. Such trials allow indirect comparisons of the magnitude of effect of A and B. Such evidence is of lower quality than head-to-head comparisons of A and B would provide. The second type of indirectness includes differences between the population, intervention, comparator to the intervention, and outcome of interest, and those included in the relevant studies. Most important here is consideration of whether important outcomes are measured directly or surrogate outcomes are used, such as a biochemical or process measure that may or may not accurately reflect what can be expected in terms of an important outcome, such as mortality or morbidity.
4. When studies include relatively few patients and few events and thus have wide confidence intervals (or a large p-value), we are less confident in an estimate.
5. The quality of evidence will be reduced if there is a high likelihood that some studies have not been reported (typically those that show no effect). The risk of such a “publication bias” is greater when published evidence is limited to a small number of trials, all of which are sponsored by people with a vested interest in the results, such as the pharmaceutical industry.

Factors that can increase the quality of evidence

1. Even well-done observational studies generally yield only low-quality evidence, because of the many potential confounders that either are not known or are not measured. However, occasionally they may provide moderate or even high quality evidence. The larger the magnitude of effect, the less likely it is that this could be explained by confounders and the evidence is, thus, stronger. For example, a meta-analysis of observational studies found that bicycle helmets reduce the odds of head injuries in cyclists involved in a crash by about two-thirds, an effect that could not easily be explained by confounders, given the design of the studies.
2. The presence of a dose-response gradient can increase our confidence in estimates of effects, for example if larger effects are associated with larger doses, as might be expected.
3. When an effect is found, if all plausible confounding would decrease the magnitude of effect, this increases the quality of the evidence, since we can be more confident that an effect is at least as large as the estimate and may be even larger. Conversely, particularly for questions of safety, if little or no effect is found and all plausible biases would lead towards overestimating an effect, we can be more confident that there is unlikely to be an important effect.
   GRADE provides a clearly articulated and comprehensive methodology for rating and summarising the quality of evidence supporting treatment and health care delivery recommendations. Although judgements will always be required for every step, the systematic and transparent GRADE approach allows scrutiny of and debate about those judgements.

[1] Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, and the GRADE Working Group. What is ‘quality of evidence’ and why is it important to clinicians? BMJ 2008; 336:995-8. 

Short glossary (for a more complete glossary click here: Glossary of terms)

Allocation concealment
The process used to ensure that the person deciding to enter a participant into a randomised controlled trial does
not know the comparison group into which that individual will be allocated.

Blinding
The process of preventing those involved in a trial from knowing to which comparison group a particular participant
belongs.

Bias
A systematic error or deviation in results or inferences from the truth.

Confidence interval
A measure of the uncertainty around the main finding of a statistical analysis.

Confounder
A factor that is associated with both an intervention (or exposure) and the outcome of interest. For example, if people in the experimental group of a controlled trial are younger than those in the control group, it will be difficult to decide whether a lower risk of death in one group is due to the intervention or the difference in ages. Age is then said to be a confounder, or a confounding variable.  Randomisation is used to minimise imbalances in confounding variables between experimental and control groups. Confounding is a major concern in non-randomised studies.

Intention-to-treat
A strategy for analysing data from a randomised controlled trial. All participants are included in the arm to which they were allocated, whether or not they received (or completed) the intervention given to that arm.

Loss to follow-up
The loss of participants during the course of a study.

Observational study
A study in which the investigators do not seek to intervene and simply observe the course of events. Changes or differences in one characteristic (e.g. whether or not people received the intervention of interest) are studied in relation to changes or differences in other characteristic(s) (e.g. whether or not they died), without action by the investigator.

Randomised trial
An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials one intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).

Related articles

• No magic answer for Achilles tendinopathy.
• Confirmation bias
• "A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null"
• Fox News Fact Check: Are studies on maternal employment all tinged with political bias?
• The mystery of the missing experiments
• Do Improved Social Signals Cause Improved Rankings? | SEOmoz
• Just How Effective Are Sales Incentive Programmes? Victor Tardieu Examines Full Circle Motivation's Case Studies And The Evidence From The IRF ROI Measures
• Too Impressed by their own Cleverness?
• Can Science Be Trusted?
• 'Evidence-based' only proves it worked in the past

Clinical Trials (journal) (Photo credit: Wikipedia)

General health checks in adults for reducing morbidity and mortality from disease

1. Lasse T Krogsbøll^*,
2. Karsten Juhl Jørgensen,
3. Christian Grønhøj Larsen,
4. Peter C Gøtzsche

Editorial Group: Cochrane Effective Practice and Organisation of Care Group

Published Online: 17 OCT 2012

Assessed as up-to-date: 4 JUL 2012

DOI: 10.1002/14651858.CD009009.pub2

Abstract

Background

General health checks are common elements of health care in some countries. These aim to detect disease and risk factors for disease with the purpose of reducing morbidity and mortality. Most of the commonly used screening tests offered in general health checks have been incompletely studied. Also, screening leads to increased use of diagnostic and therapeutic interventions, which can be harmful as well as beneficial. It is, therefore, important to assess whether general health checks do more good than harm.

Objectives

We aimed to quantify the benefits and harms of general health checks with an emphasis on patient-relevant outcomes such as morbidity and mortality rather than on surrogate outcomes such as blood pressure and serum cholesterol levels.

Search methods

We searched The Cochrane Library, the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Effective Practice and Organisation of Care (EPOC) Trials Register, MEDLINE, EMBASE, Healthstar, CINAHL, ClinicalTrials.gov and WHO International Clinical Trials Registry Platform (ICTRP) to July 2012. Two authors screened titles and abstracts, assessed papers for eligibility and read reference lists. One author used citation tracking (Web of Knowledge) and asked trialists about additional studies.

Selection criteria

We included randomised trials comparing health checks with no health checks in adults unselected for disease or risk factors. We did not include geriatric trials. We defined health checks as screening general populations for more than one disease or risk factor in more than one organ system.

Data collection and analysis

Two authors independently extracted data and assessed the risk of bias in the trials. We contacted authors for additional outcomes or trial details when necessary. For mortality outcomes we analysed the results with random-effects model meta-analysis, and for other outcomes we did a qualitative synthesis as meta-analysis was not feasible.

Main results

We included 16 trials, 14 of which had available outcome data (182,880 participants). Nine trials provided data on total mortality (155,899 participants, 11,940 deaths), median follow-up time nine years, giving a risk ratio of 0.99 (95% confidence interval (CI) 0.95 to 1.03). Eight trials provided data on cardiovascular mortality (152,435 participants, 4567 deaths), risk ratio 1.03 (95% CI 0.91 to 1.17) and eight trials on cancer mortality (139,290 participants, 3663 deaths), risk ratio 1.01 (95% CI 0.92 to 1.12). Subgroup and sensitivity analyses did not alter these findings.
We did not find an effect on clinical events or other measures of morbidity but one trial found an increased occurrence of hypertension and hypercholesterolaemia with screening and one trial found an increased occurence of self-reported chronic disease. One trial found a 20% increase in the total number of new diagnoses per participant over six years compared to the control group. No trials compared the total number of prescriptions, but two out of four trials found an increased number of people using antihypertensive drugs. Two out of four trials found small beneficial effects on self-reported health, but this could be due to reporting bias as the trials were not blinded. We did not find an effect on admission to hospital, disability, worry, additional visits to the physician, or absence from work, but most of these outcomes were poorly studied. We did not find useful results on the number of referrals to specialists, the number of follow-up tests after positive screening results, or the amount of surgery.

Authors' conclusions

General health checks did not reduce morbidity or mortality, neither overall nor for cardiovascular or cancer causes, although the number of new diagnoses was increased. Important harmful outcomes, such as the number of follow-up diagnostic procedures or short term psychological effects, were often not studied or reported and many trials had methodological problems. With the large number of participants and deaths included, the long follow-up periods used, and considering that cardiovascular and cancer mortality were not reduced, general health checks are unlikely to be beneficial.

 

Plain language summary

General health checks for reducing illness and mortality

General health checks involve multiple tests in a person who does not feel ill with the purpose of finding disease early, preventing disease from developing, or providing reassurance. Health checks are a common element of health care in some countries. To many people health checks intuitively make sense, but experience from screening programmes for individual diseases have shown that the benefits may be smaller than expected and the harms greater. One possible harm from health checks is the diagnosis and treatment of conditions that were not destined to cause symptoms or death. Their diagnosis will, therefore, be superfluous and carry the risk of unnecessary treatment.
We identified 16 randomised trials which had compared a group of adults offered general health checks to a group not offered health checks. Results were available from 14 trials, including 182,880 participants. Nine trials studied the risk of death and included 155,899 participants and 11,940 deaths. There was no effect on the risk of death, or on the risk of death due to cardiovascular diseases or cancer. We did not find an effect on the risk of illness but one trial found an increased number of people identified with high blood pressure and high cholesterol, and one trial found an increased number with chronic diseases. One trial reported the total number of new diagnoses per participant and found a 20% increase over six years compared to the control group. No trials compared the total number of new prescriptions but two out of four trials found an increased number of people using drugs for high blood pressure. Two out of four trials found that health checks made people feel somewhat healthier, but this result is not reliable. We did not find that health checks had an effect on the number of admissions to hospital, disability, worry, the number of referrals to specialists, additional visits to the physician, or absence from work, but most of these outcomes were poorly studied. None of the trials reported on the number of follow-up tests after positive screening results, or the amount of surgery used.
One reason for the apparent lack of effect may be that primary care physicians already identify and intervene when they suspect a patient to be at high risk of developing disease when they see them for other reasons. Also, those at high risk of developing disease may not attend general health checks when invited. Most of the trials were old, which makes the results less applicable to today's settings because the treatments used for conditions and risk factors have changed.
With the large number of participants and deaths included, the long follow-up periods used in the trials, and considering that death from cardiovascular diseases and cancer were not reduced, general health checks are unlikely to be beneficial.

 

Résumé

Les bilans de santé généraux chez l'adulte pour la réduction de la morbidité et de la mortalité de maladie

Contexte

Dans certains pays, les bilans de santé généraux font partie intégrante des soins médicaux. Ils visent à détecter les maladies et les facteurs de risque de maladie dans le but de réduire la morbidité et la mortalité. La plupart des tests de dépistage couramment inclus dans les bilans de santé généraux n'ont été qu'incomplètement étudiés. En outre, le dépistage conduit à une utilisation accrue d'interventions diagnostiques et thérapeutiques, qui peuvent tout autant être nocives que bénéfiques. Il est donc important d'évaluer si les bilans de santé généraux font plus de bien que de mal.

Objectifs

Nous avions pour but de quantifier les bénéfices et les inconvénients des bilans de santé généraux en mettant l'accent sur ​​les critères de résultat pertinents pour les patients tels que la morbidité et de la mortalité, plutôt que sur des critères de substitution tels que la tension artérielle et les taux de cholestérol sérique.

Stratégie de recherche documentaire

Nous avons effectué des recherches dans The Cochrane Library, le registre Cochrane des essais contrôlés (CENTRAL), le registre d'essais Cochrane sur l'efficacité des pratiques et l'organisation des soins (EPOC), MEDLINE, EMBASE, Healthstar, CINAHL, ClinicalTrials.gov et le système d'enregistrement international des essais cliniques de l'OMS (ICTRP) jusqu'à juillet 2012. Deux auteurs ont passé au crible les titres et les résumés, évalué l'éligibilité des articles et lu les références bibliographiques. Un auteur a utilisé le suivi de référencement / citation (Web of Knowledge) et interrogé des chercheurs à propos d'éventuelles études supplémentaires.

Critères de sélection

Nous avons inclus des essais randomisés comparant les bilans de santé à l'absence de bilans de santé chez des adultes non sélectionnés pour cause de maladie ou de facteurs de risque. Nous n'avons pas inclus d'essais gériatriques. Nous avons défini le bilan de santé comme le dépistage dans la population générale de plusieurs maladies ou facteurs de risque dans plus d'un système organique.

Recueil et analyse des données

Deux auteurs ont, de manière indépendante, extrait les données et évalué le risque de biais des essais. Nous avons contacté des auteurs de recherches pour obtenir des résultats ou des détails supplémentaires sur des essais. Pour ce qui concerne la mortalité, nous avons analysé les résultats à l'aide d'une méta-analyse à modèle à effets aléatoires, et pour les autres critères de résultat nous avons réalisé une synthèse qualitative car il n'était pas possible d'effectuer de méta-analyse.

Résultats Principaux

Nous avons inclus 16 essais, dont 14 présentaient des données de résultats (182 880 participants). Neuf essais avaient fourni des données sur la mortalité totale (155 899 participants, 11 940 décès), suivi médian de neuf années, donnant un risque relatif de 0,99 (intervalle de confiance (IC) à 95% 0,95 à 1,03). Huit essais avaient fourni des données sur la mortalité cardio-vasculaire (152 435 participants, 4567 décès), risque relatif de 1,03 (IC à 95% 0,91 à 1,17), et huit essais des données sur la mortalité par cancer (139 290 participants, 3663 décès), risque relatif 1,01 (IC à 95% 0,92 à 1,12). Les analyses en sous-groupes et de sensibilité n'ont pas modifié ces résultats.
Nous n'avons pas trouvé d'effet sur les événements cliniques ou d'autres mesures de la morbidité mais un essai avait observé une incidence accrue de l'hypertension et de l'hypercholestérolémie associée au dépistage et un essai avait constaté une augmentation des maladies chroniques auto-déclarées. Un essai avait observé une augmentation de 20 % du nombre total de nouveaux cas diagnostiqués par participant sur six ans, en comparaison avec le groupe témoin. Aucun essai n'avait comparé le nombre total d'ordonnances, mais deux essais sur quatre avaient constaté une augmentation du nombre de personnes utilisant des médicaments anti-hypertenseurs. Deux essais sur quatre avaient observé de petits effets bénéfiques sur la santé auto-évaluée, mais cela pourrait être dû à un biais de déclaration étant donné que les essais n'étaient pas en aveugle. Nous n'avons pas observé d'effet sur ​​les hospitalisations, l'invalidité, l'inquiétude, le nombre de visites supplémentaires chez le médecin ou les congés maladie, mais la plupart de ces critères de résultat n'avaient été que peu étudiés. Nous n'avons pas trouvé de résultats utiles sur le nombre de consultations de spécialistes, le nombre de tests de suivi après des résultats de dépistage positifs, ou la quantité d'actes chirurgicaux générés.

Conclusions des auteurs

Les bilans de santé généraux n'avaient pas réduit la morbidité ou la mortalité, ni dans leur ensemble, ni pour causes cardio-vasculaires ou de cancer, bien que le nombre de nouveaux diagnostics ait augmenté. Les conséquences néfastes importantes, telles que le nombre de procédures diagnostiques de suivi ou les effets psychologiques à court terme, n'avaient souvent pas été étudiées ou rapportées et de nombreux essais avaient des problèmes méthodologiques. Vus le grand nombre de participants et de décès inclus, les longues périodes de suivi mises en œuvre et le fait que les décès causés par les maladies cardiovasculaires et le cancer n'avaient pas diminué, les bilans de santé généraux ne sont probablement d'aucun bénéfice

 

Résumé simplifié

Les bilans de santé généraux chez l'adulte pour la réduction de la morbidité et de la mortalité de maladie

Les bilans de santé généraux pour la réduction des maladies et de la mortalité

Les bilans de santé généraux consistent en de multiples tests chez une personne qui ne se sent pas malade, dans le but de détecter les maladies de manière précoce, de prévenir le développement de maladies ou de rassurer. Dans certains pays, les bilans de santé font partie intégrante des soins médicaux. De nombreuses personnes ressentent intuitivement l'utilité des bilans de santé, mais l'expérience des programmes de dépistage de maladies spécifiques a montré que les bénéfices sont susceptibles d'être plus réduits que prévu et les préjudices plus importants. Un des préjudices possibles des bilans de santé est le diagnostic et le traitement d'affections qui ne seraient pas devenues symptomatiques ni mortelles. Leur diagnostic sera donc superflu et comportera le risque de traitements inutiles.
Nous avons identifié 16 essais randomisés ayant comparé un groupe d'adultes auxquels on avait proposé des bilans de santé généraux à un groupe à qui n'avaient pas été proposés de bilans de santé. Des résultats étaient disponibles pour 14 essais, totalisant 182 880 participants. Neuf essais avaient étudié le risque de décès ; ils portaient sur 155 899 participants et avaient recensé 11 940 décès. Il n'y avait pas d'effet sur le risque de décès, ou sur le risque de décès pour cause de maladies cardio-vasculaires ou de cancer. Nous n'avons pas constaté d'effet sur le risque de maladie mais un essai avait observé un nombre accru de personnes chez qui on avait repéré de l'hypertension artérielle et de l'hypercholestérolémie, et un essai avait trouvé un nombre accru de cas de maladies chroniques. Un essai avait rapporté le nombre total de nouveaux cas diagnostiqués par participant et constaté une augmentation de 20 % sur six ans en comparaison avec le groupe témoin. Aucun essai n'avait comparé le nombre total de nouvelles prescriptions, mais deux essais sur quatre avaient constaté une augmentation du nombre de personnes utilisant des médicaments anti-hypertenseurs. Deux essais sur quatre ont constaté que les bilans de santé faisaient que les gens se sentaient un peu en meilleure santé, mais ce résultat n'est pas fiable. Nous n'avons pas observé que les bilans de santé aient eu un effet sur ​​le nombre d'hospitalisations, l'invalidité, l'inquiétude, le nombre de consultations de spécialistes, de visites supplémentaires chez le médecin ou de congés maladie, mais la plupart de ces critères de résultat n'avaient été que peu étudiés. Aucun des essais n'avait rendu compte du nombre d'examens de suivi après des résultats de dépistage positifs, ou de la quantité d'actes chirurgicaux générés.
Une des raisons de l'absence apparente d'effet pourrait être que les médecins généralistes interviennent déjà lorsqu'ils soupçonnent chez un patient un risque élevé de développer une maladie à l'occasion de visites pour d'autres raisons. En outre, les personnes à risque élevé de développer une maladie sont susceptibles de ne pas répondre aux invitations à des bilans de santé généraux. La plupart des essais étaient anciens, ce qui retire beaucoup d'utilité à leurs résultats étant donné que les traitements recommandés pour les affections et les facteurs de risque ont changé.
Vus le grand nombre de participants et de décès inclus, les longues périodes de suivi mises en œuvre dans les essais et le fait que les décès causés par les maladies cardiovasculaires et le cancer n'avaient pas diminué, les bilans de santé généraux ne sont probablement d'aucun bénéfice

Notes de traduction

Traduit par: French Cochrane Centre 18th December, 2012 Traduction financée par: Minist�re des Affaires sociales et de la Sant�

Related articles

• Internet-Delivered Cognitive Behavioural Therapy for Adults with Mild to Moderate Depression and High Cardiovascular Disease Risks: A Randomised Attention-Controlled Trial
• Cranberries Don't Help Urinary Tract Infections?
• Efficacy of Anti-Interleukin-5 Therapy with Mepolizumab in Patients with Asthma: A Meta-Analysis of Randomized Placebo-Controlled Trials
• Delirium Prevention: Hold On to the Dream
• Effect of Antioxidant Vitamin Supplementation on Cardiovascular Outcomes: A Meta-Analysis of Randomized Controlled Trials
• The Process and Benefits of Participating in a Drugs Trial
• Polycystic Ovary Syndrome (PCOS) - Your Questions Answered
• Study: Randomized, Controlled Trial of Transcendental Meditation and Health Education in Blacks
• Fluid replacement with hydroxyethyl starch linked with higher mortality
• 600th Heart Attack Patient Recruited Into Phase III CicloMulsion CIRCUS Trial