Judgements about the quality of evidence

Judgements about the quality of evidence

A short glossary is included at the bottom of this page. For a more complete glossary click here: Glossary of terms.

In making health care treatment and delivery decisions, policymakers, patients and clinicians must trade off the benefits and downsides of alternative strategies. Decision-makers will be influenced not only by the best estimates of the expected advantages and disadvantages, but also by their confidence in these estimates; i.e. the quality of the evidence. The GRADE system, which we have used, provides a structured and transparent system for making judgements about the quality of evidence.¹

Using the GRADE system, we have made separate ratings of evidence quality for each important outcome. Like early systems of grading the quality of evidence, the GRADE system begins with the study design. Randomised trials provide, in general, stronger evidence than observational studies. Therefore, randomised trials without important limitations constitute high quality evidence. Observational studies without special strengths generally provide low quality evidence. However, there are a number of factors that can reduce or increase our confidence in estimates of effect.

The GRADE system considers five factors that can lower the quality of the evidence:

1. Study limitations
2. Inconsistent results across studies
3. Indirectness of the evidence
4. Imprecision
5. Publication bias

and three that can increase the quality of evidence:

1. Large estimates of treatment effect
2. A dose-response gradient
3. Plausible confounding that would increase confidence in an estimate

Factors that can lower the quality of evidence

1. Our confidence in estimates of effects decreases if studies suffer from major limitations that may bias their estimates of the treatment effect. These limitations include, for example, lack of allocation concealment; lack of blinding, particularly if outcomes are subjective and their assessment highly susceptible to bias; a large loss to follow-up; failure to adhere to an intention-to-treat analysis; or failure to report outcomes (typically those for which no effect was observed).
2. Widely differing estimates of effects across studies for which there are no compelling explanations reduces our confidence in knowing what the true effect is. Variability may arise from differences in populations (e.g., drugs may have larger relative effects in sicker populations); from differences in interventions (e.g., larger effects with higher drug doses); or outcomes (e.g., diminishing treatment effect with time). When variability exists but investigators fail to identify a plausible explanation, the quality of evidence decreases.
3. Decision-makers must consider two types of indirectness that can lower the quality of evidence. The first occurs when considering, for instance, use of one of two active drugs, A and B. Although randomized comparisons of A and B may be unavailable, randomized trials may have compared A to placebo and B to placebo. Such trials allow indirect comparisons of the magnitude of effect of A and B. Such evidence is of lower quality than head-to-head comparisons of A and B would provide. The second type of indirectness includes differences between the population, intervention, comparator to the intervention, and outcome of interest, and those included in the relevant studies. Most important here is consideration of whether important outcomes are measured directly or surrogate outcomes are used, such as a biochemical or process measure that may or may not accurately reflect what can be expected in terms of an important outcome, such as mortality or morbidity.
4. When studies include relatively few patients and few events and thus have wide confidence intervals (or a large p-value), we are less confident in an estimate.
5. The quality of evidence will be reduced if there is a high likelihood that some studies have not been reported (typically those that show no effect). The risk of such a “publication bias” is greater when published evidence is limited to a small number of trials, all of which are sponsored by people with a vested interest in the results, such as the pharmaceutical industry.

Factors that can increase the quality of evidence

1. Even well-done observational studies generally yield only low-quality evidence, because of the many potential confounders that either are not known or are not measured. However, occasionally they may provide moderate or even high quality evidence. The larger the magnitude of effect, the less likely it is that this could be explained by confounders and the evidence is, thus, stronger. For example, a meta-analysis of observational studies found that bicycle helmets reduce the odds of head injuries in cyclists involved in a crash by about two-thirds, an effect that could not easily be explained by confounders, given the design of the studies.
2. The presence of a dose-response gradient can increase our confidence in estimates of effects, for example if larger effects are associated with larger doses, as might be expected.
3. When an effect is found, if all plausible confounding would decrease the magnitude of effect, this increases the quality of the evidence, since we can be more confident that an effect is at least as large as the estimate and may be even larger. Conversely, particularly for questions of safety, if little or no effect is found and all plausible biases would lead towards overestimating an effect, we can be more confident that there is unlikely to be an important effect.
   GRADE provides a clearly articulated and comprehensive methodology for rating and summarising the quality of evidence supporting treatment and health care delivery recommendations. Although judgements will always be required for every step, the systematic and transparent GRADE approach allows scrutiny of and debate about those judgements.

[1] Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, and the GRADE Working Group. What is ‘quality of evidence’ and why is it important to clinicians? BMJ 2008; 336:995-8. 

Short glossary (for a more complete glossary click here: Glossary of terms)

Allocation concealment
The process used to ensure that the person deciding to enter a participant into a randomised controlled trial does
not know the comparison group into which that individual will be allocated.

Blinding
The process of preventing those involved in a trial from knowing to which comparison group a particular participant
belongs.

Bias
A systematic error or deviation in results or inferences from the truth.

Confidence interval
A measure of the uncertainty around the main finding of a statistical analysis.

Confounder
A factor that is associated with both an intervention (or exposure) and the outcome of interest. For example, if people in the experimental group of a controlled trial are younger than those in the control group, it will be difficult to decide whether a lower risk of death in one group is due to the intervention or the difference in ages. Age is then said to be a confounder, or a confounding variable.  Randomisation is used to minimise imbalances in confounding variables between experimental and control groups. Confounding is a major concern in non-randomised studies.

Intention-to-treat
A strategy for analysing data from a randomised controlled trial. All participants are included in the arm to which they were allocated, whether or not they received (or completed) the intervention given to that arm.

Loss to follow-up
The loss of participants during the course of a study.

Observational study
A study in which the investigators do not seek to intervene and simply observe the course of events. Changes or differences in one characteristic (e.g. whether or not people received the intervention of interest) are studied in relation to changes or differences in other characteristic(s) (e.g. whether or not they died), without action by the investigator.

Randomised trial
An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials one intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).

Related articles

• No magic answer for Achilles tendinopathy.
• Confirmation bias
• "A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null"
• Fox News Fact Check: Are studies on maternal employment all tinged with political bias?
• The mystery of the missing experiments
• Do Improved Social Signals Cause Improved Rankings? | SEOmoz
• Just How Effective Are Sales Incentive Programmes? Victor Tardieu Examines Full Circle Motivation's Case Studies And The Evidence From The IRF ROI Measures
• Too Impressed by their own Cleverness?
• Can Science Be Trusted?
• 'Evidence-based' only proves it worked in the past

Epidemiology, the “Data Deluge,” and the Problem of “Good” Information

Source: Somatosphere By Theresa MacPhail

This article is part of the series: Information in the Biosciences

Walking down the halls of a public health agency in the fall of 2009, I quickly became recognizable as the person doing research on information-sharing and sensemaking during infectious disease outbreaks. Two weeks into my tenure, I started being hailed by my academic association and playfully taunted with echoes of my research question: “Hey, Berkeley! Have you figured out the problem of information yet?”

The joke belied the fact that people were often extremely eager to talk about the various issues associated with information in public health: gathering data, getting access to various types of data or information, deciphering information in the form of graphs or tables or numbers, generating and recirculating information, and discerning what was often referred to as any “actionable information” that might be used to help halt the spread of a growing pandemic. Often after I explained the research goals of the interdisciplinary team project I was on, people would let out an audible sigh expressing an “information fatigue” brought on by dealing with the daily glut. The public health professionals I knew well or interviewed  – working in public health agencies in the United States and Hong Kong – habitually referred to the steady stream of emails, phone calls, meetings, and teleconferences as part of a “sea of information” or a veritable “data deluge.” Already taxed with their regular duties of disease surveillance, prevention efforts, and outbreak response, public health workers everywhere felt that their burdens had increased exponentially throughout the first ten months of the 2009 H1N1 pandemic.

People regularly complained about “drowning” in information, about being bowled over by a never-ending series of “waves” of data, about having “barely a drop” of usable information in the oceans that crossed their desks each day. I rapidly discovered that the collective goal wasn’t necessarily to become adept swimmers; rather, it seemed to be simply learning to tread water in the midst of a virtual sea of information. The experts and analysts I worked alongside or interviewed throughout the year-long pandemic continuously voiced a common longing for a more permanent solution to the problem of too much information, for a method or practice or tool that might help them cope with the overflow produced by rapidly improving technological systems of data generation and information-sharing. In 2009, the primary problem was no longer necessarily getting access to information, but of effectively coping with an overabundance of it.

Post-SARS in 2003, it had become apparent to those within the global public health community that information on infectious disease outbreaks of global importance needed to be: 1.) verifiable from a trusted or validated source; 2.) more readily circulated; and 3.) shared at a faster rate. The public health community’s subsequent emphasis on fostering greater transparency and information-sharing in public health, spearheaded by changes to the WHO’s system for reporting infectious diseases, including the revision of the International Health Regulations (IHR), solved some of the concerns over access to information, yet at the same time added an increased pressure to more quickly report validated – or good – information. The modern “myth” that increased transparency and access to more information would produce “better” information had been born. And yet, during the world’s first influenza pandemic in decades, it became increasingly apparent to everyone working in public health that more information was not necessarily better information. Instead, the reality of information-sharing during the 2009 H1N1 pandemic had highlighted other, more social – or human – problems tied to the quality of the information being readily shared.

The book I’m working on now examines how information in global public health networks is produced, managed, understood, and circulated during an outbreak. Using the 2009 H1N1 pandemic as a specific case study for examining the social practice and politics of information-sharing, my data suggests that informal networks – consisting of personal relationships – were crucial to the process of sharing sensitive, unvalidated, or what people called “good” information. In particular, the recent drive to foster greater efficiency in information sharing has in turn created various technological, scientific, and institutional temptations to decontextualize information in order to share it more quickly. The end result of all this is a problem of quality, not quantity. In other words, the largely political push toward greater transparency and faster information-sharing in public health has aggravated a need for what the people I worked with often called “context.”

As a concept used by public health professionals, context refers to details of personal or clinical experience and intuition about a disease outbreak. To them, context is the key to transforming uncertainty into certainty. To me, context as a concept refers to the human relationships and daily practices and experiences at the heart of both the production and understanding of epidemiological information. If “information” is more about the production and circulation of data or facts, then “context” is more about the production of knowledge and the circulation of experience and beliefs. Without context, “facts” (or the type of validated information that epidemiologists and scientists traffic in) are still viewed with a certain suspicion as to their soundness or applicability.Contextual information is the alchemic force that helps to turn “information” into “knowledge.” Without its attendant context, information produced and circulated during the pandemic was deemed mostly, if not entirely, useless.

Context lies at the very nexus of the human and the technological. It is the dividing point or connecting bridge between “data” and “knowledge” as well as the symbol of a chronic lack in the midst of informational overload. Throughout my fieldwork during the 2009 pandemic, the thing that people most wanted to acquire, what they spent the largest amount of their time trying to gain access to, was not more information about case counts, or symptoms, or even about virulence, but information about how people were aggregating, analyzing, and producing information about the outbreak. In essence, the public health professionals I knew were desperate to better understand their peers’ thinking processes. They believed that this type of contextual information would help them to better decide which pieces of generic information – or aggregated data – about the outbreak were most important. In sum, then, they wanted context to help them separate out the important signals from the collective noise. Context was considered key to making good decision, to taking the right response actions.

To deal with the increasing volume of data and information, health organizations utilize a set of criteria for determining “good” information and have developed a protocol for information-sharing. Yet the epidemiologists who work within large public health institutions or agencies still have to individually “make sense” of each unique situation by using that set of criteriaas a guideline. In order for certain response actions or decisions to take place, epidemiologists must rely upon each other’s analyses and personal judgments. Information-sharing and the use of context captured from my fieldwork and described above suggests to me that information in global public health moves through the following informational stages:

1. gathering information, or aggregating data from unofficial, surveillance, or informal sources
2. searching for and understanding context, or analyzing all previously aggregated information in light of personal opinions, unvalidated information, or contextual details of disease outbreaks
3. producing, (re)circulating, and using ‘good’ information to affect official response actions or recommendations for local action.

While information on an outbreak might “look” exactly the same, the contextual information produced by people who interpret that information will necessarily be different. In other words, different conclusions will be based on the same information. This difference is qualitative and due to the common daily practice of producing contextual information that is itself based on the unique lived experiences of individuals working in public health. It is this type of past lived experience as context that global public health information systems have trouble sharing through any formal channels. A brief, but pertinent, example here: During my time observing analysts inside the public health agency, an outbreak of H1N1 occurred in a far-removed location that seemed as though it might be more “serious” than the milder outbreaks happening elsewhere. Lab data on viral samples collected from patients at this location were circulated freely, as was information on overall case counts and some clinical information. However, analysts complained that the “context” was still missing. They wanted to see some type of personalized interpretation of the lab results. They asked questions about who had conducted the lab tests and what type of assays had been used, They also wanted to hear from someone they already knew and trusted in the remote location to confirm the lab data and to talk about what was actually happening on the ground. They had questions about the political situation at the location that might be causing reports from news sources to be skewed and inaccurate. The multitude of teleconferences, meetings, emails and personal telephone calls which I observed throughout my fieldwork were all attempts to gather such context – all in a concerted, if misplaced, effort to qualify and quantify what it was difficult for many individuals to describe, little alone to capture in an email or standardized form.

Scholars working on topics and issues associated with the development of formal information systems have coined a name for humans living in the so-called Information Age – inforgs. Inforgs are loosely defined as “interconnected informational organisms” that consist of both “biological agents and engineered artefacts” that live in a world “ultimately made of information, the infosphere” (Floridi 2010: 9). Floridi sees this transformation from human to inforg as something that is fundamentally “re-ontologizing” what it means to be human and to live in the 21^st century (Floridi 2007). I find the concept of inforgs compelling, even if I also find myself pushing against such a too-easy neologism. The daily practices of checking emails, looking for the latest news online, and of livestreaming meetings are merely a few common examples of the practice of epidemiology in the infosphere. While many studies have paid attention to how various experts, such as the analysts discussed here, gather and consume information, little attention has been paid to the human/technology interface that produces such information in the first place. One solution might be to take the use of information and information technologies more seriously from an anthropological viewpoint.

Right now, I’m working through the issues of defining “good” information, the 21^st century “data deluge,” and the role of context in an attempt to craft an ethnography of the daily practice of turning information into actionable knowledge. These issues highlight the various difficulties of gathering, analyzing, and reporting information not only related the 2009 pandemic, but are indicative of the messy and complex process of making sense out of a daily barrage of information in any scientific or data-driven field. The already nascent ‘anthropology of information’ needs to pay particular attention to points where the human and the technological become enmeshed with each other.

As Bowker and Star have argued, there is “a permanent tension between universal standardization” of information-sharing systems and “the local circumstances of those using them” (139). Efforts at further standardization of information systems in global public health are only doomed to worsen the problem if they fail to take the problem of context more seriously. And context can only be understood at the level of the social and the cultural – or the realm of anthropology. I see the anthropology of information as a field that has rich potential not only for further research but to bridge the gap between theory and application. Information is a part of our daily lives; as inforgs, we need to get much better at understanding how we use it, think about it, and relate to it.

Theresa MacPhail received her PhD in Medical Anthropology from UC-Berkeley/UC-San Francisco. Her first book, Siren Song: A Pathography of Influenza and Global Public Health, is based on her dissertation research on the science and epidemiology of influenza in Hong Kong, the United States, and Europe. She is currently a Faculty Fellow/Assistant Professor in Science Studies in the John W. Draper Interdisciplinary Master’s Program in Humanities and Social Thought at New York University.

• Share this:
• Share
• 
  

Similar Posts

• Emerging thoughts on swine flu
• Swine flu, cont.: mini-post
• Anthropologists on H1N1 flu
• Microbes and Anthropology
• An fMRI primer

[view academic citations]

Related articles

• A decade after devastating outbreak, is the world better prepared for next SARS?
• Why GVSU is launching a master's program in public health
• House bill introduced to require VA reporting of Legionnaires' and other infectious diseases
• New cell to tackle disease outbreak
• Judge to Bloomberg: unhand that soda
• iPhone app tracks swine flu outbreaks
• Vaccinate your children, MP urges in measles-hit borough
• Are Canadian kids undervaccinated? Or is it that we just don't know?
• AIDS group wants L.A. to break with county health department
• Caribbean on alert after 20 die in China from bird flu outbreak

English: German Network for Evidence Based Medicine Deutsch: Deutsches Netzwerk Evidenzbasierte Medizin (Photo credit: Wikipedia)

MBE | Bases of evidence based medicine
:

Authors: Martín Muñoz P¹, González de Dios J^2
1Director de Unidad Clínica en Atención Primaria, CS La Plata. Hospital Universitario Virgen del Rocí­o. Sevilla. Sevilla (España). ²Departamento de Pediatría. Hospital General Universitario de Alicante. (España). 

Correspondence: Pedro Martín Muñoz. Email: pedromartinm@telefonica.net

Palabras clave: práctica clínica basada en la evidencia; medicina basada en pruebas; guías de práctica clínica

Keywords: evidence-based practice; evidence-based medicine; clinical practice guidelines

Publication date: 01/09/2010   

De la evidencia a la recomendación: una tarea pendiente

Una aspiración irrenunciable de la medicina actual, reclamada por todos sus protagonistas (enfermos, profesionales y administraciones), es que los actos médicos se sustenten en conocimientos científicos obtenidos de procesos de investigación clínica rigurosa. Decidir si una intervención clínica resulta adecuada para un paciente determinado equivale a establecer si existe un grado razonable de certeza de que el balance entre los beneficios, por un lado, y los riesgos, los inconvenientes y los costes, por el otro, de dicha intervención resulta lo suficientemente favorable como para que merezca la pena aplicarla. Los conceptos de calidad (nivel) de la evidencia yfuerza (grado) de las recomendaciones constituyen un pilar fundamental de la práctica basada en la evidencia, en su intento por estandarizar y proporcionar a los clínicos reglas para analizar la literatura científica, determinar su validez y considerar su utilidad en la asistencia sanitaria.

Cada vez toma más cuerpo el tomar decisiones médicas que estén fundamentadas en el mejor nivel de evidencia (indica hasta qué punto nuestra confianza en la estimación de un efecto es adecuada para apoyar una recomendación) y la mayor fuerza de recomendación (indica hasta qué punto podemos confiar si poner en práctica la recomendación conllevará más beneficios que riesgos).

La calidad (nivel) de evidencia se ha relacionado, generalmente, con el diseño del estudio (estudios descriptivos o analíticos, observacionales o experimentales) y la calidad de los mismos. La meta de la investigación es la agudeza en la medición, lo que implica precisión (limitar el error aleatorio) y validez (limitar el error sistemático). En este sentido, por las características propias de cada diseño, el “nivel” de evidencia será mayor en los estudios analíticos que en los descriptivos, y superior en los estudios experimentales (ejemplo, ensayo clínico) que en los observacionales (ejemplo, estudios de cohortes y estudios de casos y controles). Sin embargo, no toda pregunta clínica se puede abordar con el mismo diseño científico: el ensayo clínico es el patrón oro para intervenciones terapéuticas, pero no será el diseño apropiado para preguntas sobre diagnóstico o pronóstico.

Se establecen unos criterios de calidad propios para cada tipo de diseño. Así, podemos considerar cinco criterios de calidad en el ensayo clínico (definición clara de la población de estudio, intervención y resultado de interés; correcta aleatorización; adecuado enmascaramiento; seguimiento completo - menos del 20% de pérdidas -; análisis correcto - análisis por intención de tratar y control de covariables no equilibradas con la aleatorización -), que serán diferentes a los criterios de calidad barajados en el caso de estudios de valoración de pruebas diagnósticas (comparación con un patrón de referencia válido; muestra representativa; descripción completa de los métodos de realización de la prueba diagnóstica; control de sesgos - comparación ciega e independiente -; control de sesgos de incorporación, verificación diagnóstica y revisión; análisis correcto - datos que permitan calcular indicadores de validez -) o de cohortes (cohortes representativas de la población con y sin exposición, libres del efecto o enfermedad de interés; medición independiente, ciega y válida de exposición y efecto; seguimiento suficiente - superior al 80% -, completo y no diferencial; control de la relación temporal de los acontecimientos – exposición/efecto - y de la relación entre nivel de exposición y grado de efecto - dosis/respuesta -; análisis correcto - control de factores de confusión y modificadores de efecto -), por ejemplo.

La fuerza (grado) de las recomendaciones indica hasta qué punto podemos confiar en que poner en práctica la recomendación conllevará más beneficio que riesgo. En la elaboración de las recomendaciones se debe tener en cuenta, en primer lugar, el nivel de evidencia, pero también otras consideraciones: balance entre beneficios y riesgos, consistencia de los estudios, aplicabilidad práctica en mi paciente o población (incluyendo el riesgo basal en mi población), valores y preferencias de la población diana a la cual va dirigida, costes, etc. Establecer una recomendación, a favor o en contra de una intervención, no significa que todos los pacientes deban ser tratados de la misma manera, pues en la toma de decisión la evidencia procedente de la investigación es sólo uno de los cuatro círculos en una toma de decisiones basada en pruebas (figura 1).

Figura 1. Modelo actualizado en la toma de decisiones basada en pruebas. Mostrar/ocultar

Ambos conceptos, aunque relacionados y complementarios, se ocupan de aspectos distintos. Aunque la fuerza de una recomendación se apoya, decisivamente, en la calidad de la evidencia que la sustenta, ello puede no resultar suficiente de ser por ejemplo muy pequeña la magnitud del efecto sobre las variables primarias, tener poca precisión la estimación realizada o ser irrelevante desde el punto de vista clínico el resultado medido (diferencia entre significación estadística e importancia clínica). Por último, el elemento clave para decidir el grado de recomendación se obtiene al considerar el binomio beneficio/perjuicio neto para la salud, consecuencia del análisis de varios factores (magnitud del efecto y daño, disponibilidad social y coste).

El primer intento serio de introducir rigor y transparencia en la jerarquización de la evidencia fue realizado hace ya más de 30 años por la Canadian Task Force on Preventive Health Care (CTFPHC)1, adaptado posteriormente por la United State Preventive Services Task Force (USPSTF)2. Desde entonces numerosas organizaciones e instituciones, entre las que destacan el Centre for Evidence-Based Medicine (CEBM) de Oxford3, el Scottish Intercollegiate Guidelines Network (SIGN)4, el National Institute for Health and Clinical Excellence (NICE)5 o la U.S. Agency for Health Research and Quality (AHRQ)6, han ido desarrollando sus propios sistemas jerárquicos y, actualmente, se contabilizan más de cien herramientas, 19 sistemas para evaluar la calidad y 7 para graduar las recomendaciones7. En síntesis, las escalas pueden utilizar letras (ej. A, B, C, etc.), números (ej. I, II, III, etc.) o una combinación de ambos (ej. Ia, Ib, IIa, etc.). Sin embargo, la situación a la que se ha llegado dista de ser satisfactoria8,9. La comparación entre las distintas propuestas existentes (tabla 1) pone de manifiesto diferencias sustanciales en los criterios de gradación, con una baja sensibilidad y reproducibilidad de los mismos, múltiples posibilidades para evaluar y estructurar la evidencia y diferentes interpretaciones de los grados de recomendación. Además, la proliferación de escalas genera confusión y dudas en los usuarios, constatándose la inexistencia, hasta ese momento, de un modelo adecuado que pudiera ser universalmente aceptado10-12.