KNOWLEDGE APPRAISAL FOR COMPLEX PROBLEMS

The speed at which guidance needs to be produced during the COVID-19 outbreak poses a huge challenge to guidance producing bodies around the globe. Outbreak guidance is known for its ability to be fast but right, but now that COVID-19 is affecting health systems, including other parts of clinical care, primary care, long-term care, care for ageing adults, mental health care, and many more, guidance for a wide range of sectors is being developed at a fraction of the time that is usually required.

Knowledge appraisal models that give priority to evidence from randomized controlled trials leave evidence producing bodies largely empty handed. In times of outbreaks, that evidence is mostly absent, and the questions are too complex to be addressed through study designs that have specialized in establishing simple causal relations.

Moreover, clinical study designs that have a high internal validity (a high confidence about a causal relation between intervention and effect) mostly achieve such validity by maximally excluding contextual factors. This often is at the expense of external or ecological validity (whether the outcome of the study can be expected to occur in a real-world setting). In the case of outbreak management, or containment strategies, ecological validity is far more important because:

  1. Outbreaks are complex, unstructured problems, taking place in many different settings and affecting a wide range of people;

  2. An outbreak can only be managed if none of those societal groups are substantially excluded. Without reaching even the most vulnerable in societies, infectious diseases will continue spreading and reemerging.

The COVID-19 outbreak therefore poses enormous challenges to guidance development practices and infrastructures. One of the things we have noted in our collaborative research, is that, in the lack of evidence from clinical studies, much guidance is now ‘expert based’. But who are the experts on complex, unstructured problems with huge societal consequences? How are their different knowledge, experiences and values weighed? And how are these integrated with bits of clinical evidence that are becoming available? And finally, how can the integration of such wide ranging knowledge, experience and values be facilitated? These questions are at the heart of the Athena Institute’s research and teaching.

Research collaboration
Evidence at the margins
Evidence on how lockdowns are affecting marginalized communities are scarce and often based on modelling. By collaborating with NGOs and government agencies in countries like India and Laos, we are helping build the empirical evidence base about the effects of lockdowns in more detailed and robust ways. What are the needs of the homeless day wagers in Delhi, and how can they be reached? What alternatives do governments in low and middle-income countries (LMICs) have for setting up outbreak management strategies that do not exclude the working poor and homeless? These are questions
Harsh Mander, of Athena’s long-standing collaborative partner the Centre for Equity Studies in Delhi, pertinently raises. See some of their publications and movies here.

Appraising and including different knowledge in guidance
How are guideline developers currently facing the challenges posed by the high pace and complex questions related to the COVID-19 pandemic? Can they rely on their established guideline development methods and practices? Do they have to change them? Throw them overboard? The baby out with the bathwater? Are they finding inventive solutions? And how will these affect the future of guideline development? In collaboration with the Working Group of the Guidelines-International-Network on Appraising and Including Different Knowledge (
AID Knowledge), Athena Institute researchers are running two surveys with follow up interviews, and knowledge appraisal workshops, to capture the moment in guidance development around the globe. See more in this blogpost.

Athena Institute publications related to this initiative