Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines

Published research on COVID-19 is increasing rapidly [1], [2], [3], [4], [5]. Clinicians, patients, citizens and others decision-makers are pressured to make urgent decisions and adapt to the rapid and changing demands in clinical and public health [6]. For both decision-makers and knowledge synthesis (KS) and knowledge translation (KT) experts who package evidence for decision-makers, registries and portals have emerged to catalogue primary studies and syntheses of evidence on COVID-19. Among those are the Norwegian Institute of Public Health living evidence map (NIPH) [7], the COVID-19 L•OVE platform [8], COVID-END, McMaster's COVID 19 Evidence Alerts [9], the Cochrane Library and the Open-Access Data and Computational Resources to address COVID-19 by the National Institute for Health. Moreover, methodologists have supported the global community of KS producers by providing guidance on methodologies relevant to the urgency of COVID-19 [10], [11], [12].

Similarly, clinical, public health and health policy guidelines have been developed by various organizations in response to the COVID-19 pandemic [6]. Notable are those produced by the World Health Organization (WHO) as they consider a broad context when producing evidence-based recommendations, which can be adapted locally, where needed; this adaptation process is an efficient means of supporting decision-making rather than producing de novo guidelines. Particularly in emergencies, WHO undertakes approaches that optimize the quality of guideline development that reduce the biases brought by the need for a rapid process [14,15]. When these guidelines address and provide evidence for complementary considerations important for decision-making, such as cost and cost effectiveness, impact on health equity, acceptability, feasibility, their contextualization to other settings becomes possible; this is particularly important during emergencies [11]. The GRADE Adolopment [16,17] approach allows the adoption (without modifications) of a recommendation, its adaptation (with modifications) and de novo development when no recommendation exists in the source guideline.

The GRADE Evidence to Decision (EtD) framework in guideline development facilitates the consideration of criteria that are important for informing a recommendation, each addressed with the best available evidence, where possible. Contextualization of an existing guideline should integrate research or other evidence specific to the particular circumstance, such as a geographic region, for one or more of the above mentioned EtD criteria, e.g. cost or feasibility. This process culminates in a decision to adopt a recommendation as is or to adapt it because of that contextual information [13,18]. It is also possible that the available guideline information is incomplete and de novo development is required.

The trustworthiness of guidelines can vary depending on the methods used to assemble and evaluate the evidence [14,19,20] the completeness and transparency of reporting on the process undertaken [21,22] and addressing conflicts of interest [23], [24], [25], [26], [27], for example, among other standards proposed by the Guidelines International Network and the Institute of Medicine [28]. The Appraisal of Guidelines, REsearch and Evaluation II (AGREE II) tool is a long-established instrument used to appraise the quality of clinical guidelines, for example in the evaluation of emerging guidance in COVID-19 [35].

As with previous public health emergencies, the current pandemic has triggered a rapid rise in guideline development to answer decision makers’ questions [4]. Of central importance in a quickly shifting COVID-19 decision-making landscape is making that information available while considering the quality of the guidance and contextualization of those recommendations to specific circumstances. Accordingly, our team has capitalized on the innovations and advances in methodological and technological aspects of guideline development to respond to the current pandemic by building a living catalogue of evidence and recommendations (https://covid19.recmap.org) through funding by the Canadian Institutes for Health Research in collaboration with a list of partners (https://covid19.recmap.org/about).

Our foray into electronic recommendations maps began with earlier work with the WHO's Global Tuberculosis department to build the WHO eTB living map of recommendations (https://who.tuberculosis.recmap.org/). The eTB tool is freely accessible, collects all WHO's recommendations on TB prevention and care, and categorizes them for efficient presentation for different stakeholders [29].

Building on the TB model, we have created a catalogue of COVID-19 recommendations as our direct response to the increased need for a structured access to high quality guidance in the field. The COVID19 map of recommendations and gateway to contextualization (https://covid19.recmap.org) is a living project whereby the emerging guideline literature is added on an ongoing basis, allowing granular access to individual recommendations. This is a product of the collaboration between Cochrane Canada and other Cochrane entities, the WHO Collaborating Center for Infectious Diseases, Research Methods and Recommendations at McMaster University, GRADE centers, the Norwegian Institute of Public Health, the Guidelines International Network, the National Institute of Health and Care Excellence (NICE), WHO/PAHO, and many other institutions or organizations (https://covid19.recmap.org/about).

Our scope of work is focused on identifying COVID-19 guidelines, assessing their quality, and making their recommendations available for stakeholders (Fig. 1 ). We provide an ‘adolopment’ module to facilitate contextualization and implementation by decision-makers across the globe. To achieve this, we have developed bibliographic searches that pre-filter citation yields to capture guidelines on COVID-19 through our partners Health Information Research Unit at McMaster University; they also use Application Process Interface (API) from ECRI Clinical Guidelines [30], International Database of GRADE Guidelines [31], National Institute for Health and Care Excellence, the World Health Organization, the Center for Disease Control, and Guidelines International Network's library. This search is updated twice per week. In parallel we search websites of guideline organizations: European CDC, Public Health Agency of Canada, Scottish Intercollegiate Guidelines, COVID Network Meta-Analysis Initiative and Canadian Task Force on Preventive Health Care. The captured guidelines are then screened manually to ensure that their methodology follows that of a guideline and that their population is COVID-19 suspected, confirmed or at high risk. We do not limit our searches for language or to any COVID-19 topic. Two trained members of the research team use AGREE II instrument to critically appraise the included guidelines. On our map we only report the results of three domains that address credibility: scope and purpose (domain 1), rigor of development (domain 3) and editorial independence (domain 6). Meanwhile, the scores for the other AGREE II domains are kept in our records and provided upon request. When updates of guidelines are published, our team appraises the latest version and modifies the content on the map, accordingly.

To support this complex and intensive workflow for a living recommendation map a customized GRADEpro module (www.gradepro.org) was developed by our technology partner EvidencePrime, Inc. The customization included a guideline level data extraction module, including but not limited to source and search date; and a recommendation level module to capture granular information to be displayed on the map or used for searching the repository. This modular structuring of the information facilitates updating content of the map and will support future guideline development work downstream the recmap (for example, adolopment).

The unit of organization of the map is the individual recommendation. The user can find what they are looking for through free text searching, browsing a list of recommendations, or consulting a grid that maps recommendations according to topic area (and interventions included) in the columns, and population in the rows. When users select a recommendation of their interest, the detailed information on this individual recommendation is given, including the AGREE-II appraisal, the certainty of the evidence, strength of the recommendation and additional information, depending on what the guideline developers included (for example, evidence profiles).

A novel feature of this map is the self-directed contextualization of the recommendations using the adolopment module. The GRADE-Adolopment approach [18] uses the EtDs to adopt, adapt or synthesize de novo recommendations to answer context specific questions. Through our map, stakeholders can access the EtDs developed by the guideline developers, and adapt them for their context through the “adolopment module”: a direct linkage to the original guideline project on GRADEpro to access original content, select what needs to be contextualized and go through the steps of development of adapted recommendation. The adapted recommendation can then be pushed to the map as well, and those seeking adolopment can identify relevant high-quality evidence through the linkages to the NIPH and L.OVE platforms of living evidence.

Dealing with the large volume of guidelines in the times of the pandemic is burdensome, and a prioritization process is necessary to maximize efficiency and responsiveness to changing needs. Our prioritization approach aims at balancing globally trending topics (e.g., masks, vaccination, schools) or the direct requests from stakeholders (e.g., WHO, school administrations) and our team's capacity to process a high but finite number of guidelines at the same time. Our prioritization approach is informed mainly by systematic reviews of guideline prioritization methods and input from experts, and includes three phases: pre-prioritization, prioritization and post-prioritization. The full description of this process is being finalized and will be disseminated in the coming months.

The overall project is collective effort made possible by a large international and multidisciplinary team of global experts in guideline development, software development, and stakeholders who have joined efforts to develop, maintain and disseminate this easy-to-navigate and freely accessible electronic map. We have organized ourselves in nine working groups: information technology, information science, quality appraisal, data extraction, equity, adolopment, implementation and dissemination, cost/cost-effectiveness and WHO Essential Medicines List. An all-round open steering group meets weekly to review the progress of the work, identify and address upcoming new needs, prioritize topics, and collectively work on reports like this one.

The engagement of stakeholders is crucial for the uptake, maintenance and sustainability of our map, similarly to their engagement in health research [32,33]. We initially engaged consumer experts in parallel to inform the map's development, and developed a feedback survey that is available on our map for any user to complete. Additionally, we have recently collaborated with Guidelines International Network (GIN) PUBLIC [34] public versions of guideline recommendations that appear on the our map. Our implementation and dissemination working group is currently working on a plan to optimize the public versions development and dissemination. We have established a mechanism of informing guideline developers that their work will be hosted on our map by direct contact where we introduce them to our work and inform them that their guideline is captured and undergoing appraisal and extraction. They also have the option to provide input or clarification, where needed. Additionally, we have presented webinars on different platforms (GIN, COVID_END, WHO Evidence informed Policy Network, Evidence in Health and CanCOVID), and our team members are available to provide more.

The COVID19 and the WHO Tuberculosis recommendation maps can save time, resources and ensure uptake of up-to-date recommendations by decision-makers, guideline developers and the public of evidence-informed recommendations through this one stop shop portal. We are currently undertaking a randomized trial to test users’ preferences between the traditional publication of WHO TB guidelines on their website and on the eTB map. We will use the results related to the functionality of the map to improve both maps.

We do believe that the concept of recommendation mapping is a novel one, and the recent uptake by agencies such as the Canadian Agency for Drugs and Technology in Health (CADTH) suggests others see its value (https://tuberculosis.cadth.ca/). Whether the end-user aim is to identify or to contextualize relevant recommendations, a one stop shop portal of evidence-informed recommendations built with intuitive functionalities, easy to navigate and with a support team ready to guide users across the maps undoubtedly represents a long-needed tool for decision-makers, guideline developers and the public at large.