1.2 Knowledge synthesis: formally identifying and assessing relevant evidence

Implementing a policy or intervention that has only been shown to be effective in one research study can be problematic. Few studies by themselves are persuasive enough to change policy or practice; in fact, individual studies may even be misleading due to chance or bias⁽³⁷⁾. Therefore, after carrying out the situational analysis and identifying the health need and desired outcomes, implementers need to perform a formal synthesis of evidence on potential policies and interventions – termed a knowledge synthesis.

Failure to use such a knowledge synthesis can lead to delays between the generation of research evidence about an intervention and the time when clinical experts make recommendations in line with research findings. For example, Antman observed a 15-year gap between the time when meta-analysis could have demonstrated the effectiveness of a particular method to treat myocardial infarction and widespread recommendations for its use⁽³⁸⁾.

Stages of knowledge synthesis

There is a growing range of methods for knowledge synthesis; most involve an initial review of existing literature. Depending on time, resources available, and other constraints, this literature review can be rapid or involve a lengthier meta-analytical process. These issues are explored in more details later in this guide; in this section, we highlight the common stages in knowledge synthesis (see below) based on published frameworks.^{(39, 40, 41)}

Stage 1: Stating the objectives of the policy or intervention to be implemented
Stage 2: Defining the eligibility criteria for evidence to be assessed
Stage 3: Defining a search strategy to identify relevant evidence
Stage 4: Searching for relevant evidence (applying the search strategy)
Stage 5: Assessing the quality of evidence found
Stage 6: Assembling and analysing the most complete data set feasible
Stage 7: Making an informed decision based on a structured report of the research

Stage 1: Stating the objectives of the policy or intervention to be implemented

The first stage in the knowledge synthesis process is to formulate the objectives of the synthesis; this is arguably the most important stage in the process and is partially informed by the situational analysis described earlier in Section 1.1.

The more explicit the objectives (for example, in terms of how specifically the population or the intervention is defined), the more you will limit available evidence. On the other hand, making the objectives broader is likely to require more resources as there will be more evidence to sift through and assess.

One method for devising objectives is summarized by the acronym PICO:

Population
Intervention
Comparison
Outcome

For example, your objectives may be to identify evidence on approaches to the prevention of secondary heart attacks (outcome) in elderly men and women who live in rural areas (population). These are very broad objectives, so you may refine these based on information gathered during the situational analysis. For instance, the situational analysis may have revealed that the majority of people in your target population have access to mobile phones; you may therefore want to narrow your objectives by stating that you are seeking evidence on mHealth programmes (interventions) and their effectiveness compared with face-to-face secondary prevention services (comparison).

The PICO model is very widely used and it is recommended by Cochrane (previously The Cochrane Collaboration) as a strategy for formulating questions and search strategies and for characterizing clinical studies or meta-analyses. An example of PICO and a template to apply this to your context can be found below.

Download PICO framework (DOCX 32.6 KB)

Stage 2: Defining the eligibility criteria for evidence to be assessed

In the second stage you should set the criteria that will determine whether you retain (and assess) a particular piece of evidence that you identify or whether you should discard it. This stage is partially guided by the objectives outlined earlier.

First, you need to specify the characteristics of the evidence (e.g. research studies) that are to be included in your knowledge synthesis – in effect, the ‘eligibility criteria’. The following are typical:

the nature of what was studied (e.g. specific policies or interventions);
the context (in other words setting and population – e.g. adults; ethnic groups);
the date of research (e.g. ever; since 1920; since 1990);
the research methods (e.g. all methods; only empirical; only certain designs);
the language of report (e.g. English only; French only; both).

Taking forward the scenario mentioned in stage 1 above, you might want to limit your search to ‘evidence on interventions that use information and communication technology for secondary prevention of heart attacks in men and women living in rural areas’. Ideally, you would also limit your search to evidence from your own country, although this may not always be possible as the evidence available may be too limited or nonexistent. In that case you may wish to look for evidence from the region (e.g. South Asia if you are based in Bangladesh; sub-Saharan Africa if you are based in Uganda; and so on).

You also have to decide if you only wish to search for recent evidence (often the case, to ensure relevance) and if you want to include peer-reviewed studies that use randomized controlled designs only, or whether you also want to include grey literature.

Grey literature is information that is unpublished or not published commercially; its value has been formally acknowledged by prestigious evidence-based research organizations, including Cochrane. Examples include (but are not limited to) conference papers, reports, policy statements, government documents, statistics, interviews and focus groups reports. Grey literature is particularly useful when the literature relating to polices and interventions of interest is limited in quantity and/or has limited applicability to the local context.

Stage 3: Defining a search strategy to identify relevant evidence

After you have set the objectives for the knowledge synthesis, and after you have decided which evidence you will assess, you need to prepare the search strategy. This specifies the detailed method for conducting the search; it outlines exactly which terms (in a structured list) you will search for in databases, how these terms will be linked and what databases you will use. The search strategy should be grounded in the research question and should be recorded in detail.

Your choice of key terms will be guided by the objectives. Bear in mind that the same concept may be referred to in a number of ways (e.g. self-esteem might be referred to as self-worth elsewhere). You therefore need to examine each of your concepts and develop a list of the different ways in which they could appear in the literature. You will also need to think about how your search terms may be linked.

Searches are usually conducted online, using existing research literature and/or policy databases. Key databases that are useful in the identification of relevant evidence include:

Different databases work in different ways, so you may need to adapt your search strategy to each database that you use. This process is often referred to as ‘tailoring’ your search. You may also decide to develop separate search strategies for different aspects of your research.

Note that implementers do not all always have access to all databases or to the evidence identified through the databases. This underpins the importance of establishing an intersectoral collaborative team from the outset (as mentioned in Section 1.1); ideally, this team will include academic researchers who will typically have access to most online databases and sources of evidence.

When searching for relevant evidence there is a trade-off between sensitivity and specificity; specificity decreases as sensitivity increases. Searches that are highly sensitive will identify all or most of the relevant literature, however they will also likely identify literature that is not relevant. Searches that are highly specific will exclude all or most of the literature that is not relevant, however they may also exclude some of the literature that is relevant. The more sensitive the search, the more time needs to be spent sifting out irrelevant studies. Given that implementers are often time-constrained or resource-limited, some sensitivity may have to be sacrificed in the knowledge that some potentially relevant evidence may be missed. Defining a search strategy will benefit from the expertise of an information specialist. They can provide guidance on defining search terms that will help balance sensitivity and specificity.

Stage 4: Searching for relevant evidence (applying the search strategy)

This stage involves searching for all relevant evidence using the selection criteria identified and the predetermined search strategy for a specific database(s). The search will aim to identify as much of the literature that meets the inclusion criteria as possible.

If time and resources allow, it is a good idea to have more than one person performing the same search independently, and then comparing the evidence identified to make sure that findings are consistent and there is no bias in the way that searches are made and evidence is selected.

When searching for research evidence, it is important to ensure you consider which study design will best answer your research question. For example, a systematic review of randomized control trials is ideal if you wish to determine the best type of intervention to prevent or manage a condition. However, if you are wishing to know how common the problem is, then local and current random sample surveys (or censuses) would be more appropriate. The Oxford Centre of Evidence Based Medicine (OCEBM) provides a hierarchy of evidence depending on the research question.

Stage 5: Assessing the quality of evidence found

The quality of evidence is likely to vary considerably. Therefore, you must decide on explicit criteria for appraising studies in order to separate those of higher quality from those of lower quality.

Three main dimensions considered when appraising the quality and relevance of studies⁽⁴¹⁾ are:

the methodological quality of the study;
the relevance of that research design to the objectives;
the relevance of the study focus to addressing the objectives.

Checklists such as the Jadad scale (also known as the Oxford quality scoring system) are commonly used for assessing the methodological quality of trails⁽⁴¹⁾. The Joanna Briggs Institute provides a range of different checklists to critical appraise other study designs.

WHO uses the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach to assess the quality of a body of evidence. WHO uses this approach as it represents internationally agreed standards for making transparent recommendations. Detailed information on GRADE is available through the WHO Guidelines Review Committee (GRC) secretariat and on the following websites:

Other sources of information can also help you in assessing evidence:

The Trip Database is a medical search engine that allows you to search for evidence based on PICO and to find out about evidence by study design, relevance and timeline.
The Critical Appraisal Skills Programme (CASP) provides a range of training, workshops and tools to help you critically evaluate the quality, results and relevance of research.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is another widely used method to assess an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

Assessing evidence and developing evidence summaries using GRADE (or otherwise) is a specialized task that is best done by a methodological expert, so it may be prudent to enlist such help. This is especially important if you are assessing evidence for large scale NCD prevention and control policies. Nevertheless, if appropriate it is possible to assess the evidence more rapidly and in-house using the three dimensions listed above with existing online resources.

Stage 6: Assembling and analysing the most complete data set feasible

After assessing the evidence, you will have to collate and analyse all your assessments to determine if there are sufficient grounds to implement the policy or intervention that you are interested in. This is likely to be the stage that implementers, particularly policy officials, are most interested in. You should therefore ensure that output from the knowledge synthesis is presented in a clear format that meets their needs (for example, by drawing out policy implications).

There are various approaches to collating, analysing and presenting the evidence you found; below, you can download an example of a matrix for the synthesis of relevant evidence.

Download example matrix (DOCX 84.6 KB)

Stage 7: Making an informed decision based on a structured report of the research

Only when all available evidence has been collated and assessed, and evidence for the effectiveness has been ranked, is it possible to select a policy or intervention for adaptation to and implementation in your local context.

Systematic reviews and other approaches to knowledge synthesis

The guidance provided above is sufficient to identify and assess evidence relevant to the effectiveness of policies and interventions for an outcome of interest within a relatively short amount of time and with limited resources. A more rigorous approach to identifying, assessing and synthesizing evidence from numerous sources is to carry out a systematic review. Systematic reviews bring the same level of rigour to reviewing research evidence as should have been used in producing that research evidence in the first place. Using the systematic review approach, however, is time and resource consuming and is not usually possible in the circumstances where most programme implementers are seeking to implement a new policy or intervention (or to implement an existing policy or intervention in a new setting).

A faster approach is that of rapid evidence assessment, which uses targeted literature searches to produce a report in a relatively short period of time. This is less rigorous than a full systematic review, but more so than an ad hoc search. It is well aligned with the approach described here. The UK Government has developed and made available online a toolkit which provides additional information on rapid evidence assessment.

Case study 2 provides an example of knowledge synthesis in practice.

Case study 2: Knowledge synthesis – Tobacco-related NCDs and plain packaging policies, India