3.2 What are implementation outcomes?
In implementation research studies, implementation outcomes describe the intentional actions to deliver a policy or an intervention(18, 78); they are distinct from, but related to, health outcomes.
Implementation outcome variables include: acceptability, reach, adoption, fidelity, implementation cost and sustainability. Acceptability has been covered in Section 2.1; cost and sustainability are discussed later in Sections 3.6 and 3.8, respectively. In the following subsections we will focus on reach, adoption and fidelity.
What is the reach of the policy or intervention?
Reach is defined as the absolute number, proportion and representativeness of a study sample(79). Population impact is a function of how well a policy or intervention is implemented, its effectiveness at the individual level and its reach.
For example, the population impact of a smoking cessation programme depends both on how many smokers are reached and try to stop smoking, and what the average success rate is. Research indicates that in this example the reach of the programme has much greater impact, since the success rate seems to vary less.
Reach is a combination of both the number of people reached by a policy or intervention and how representative they are of the target population. Most studies report the size of the study sample and the proportion of individuals who are willing to participate, but few report on representativeness (which indicates the similarities and/or differences between those who participate in the study and those who are eligible but do not).
The representativeness is important, as generalization of an intervention into real-world settings is likely to have better impact if sample representativeness – and therefore reach – is good. Comparisons for representativeness should be based on basic demographic characteristics and, when possible, on primary outcomes. Case study 5 (below) explores the issue of reach.
What is the adoption of the policy or intervention?
Adoption has been defined as the absolute number, proportion and representativeness of settings (contexts) and intervention agents (implementers) that are willing to initiate a programme (policy or intervention).(79)
Different contexts (e.g. worksites, medical offices, schools, communities, etc.) and implementers (e.g. health practitioners, policy-makers, government staff, researchers, etc.) can differ in their adoption of a policy or intervention, as this is affected by the availability of resources, the level of expertise and the commitment to programmes. Researchers seldom report on issues of adoption, but understanding how adoption varies among different contexts and implementers is critical to the impact of a policy or intervention.
The approach to measuring adoption should change depending on the policy or intervention, as well as on whether adoption at the context level or implementer is of interest. Providing detailed guidance on the many tools and approaches for the assessment of adoption is beyond the scope of this guide. The National Institutes of Health in the United States of America, however, maintains the Grid-Enabled Measures Database – a database of measures that are used to assess adoption (as well as other implementation outcomes).
What is the fidelity of the policy or intervention?
Implementation fidelity refers to the extent to which a policy or intervention is delivered as intended by its developers and in line with the programme model.(80) Evaluation of implementation fidelity is important because it may affect the relationship between an intervention and its outcomes. It may also:
- prevent potentially false conclusions from being drawn about an intervention’s effectiveness in achieving the intended health outcomes;
- help in the achievement of improved outcomes;
- give primary researchers confidence in attributing health outcomes to the intervention;
- give evidence-based practitioners confidence that they are implementing the chosen intervention properly;
- give secondary researchers more confidence when synthesizing studies.(81)
Two aspects of interventions affect fidelity(81) – structural and dynamic.
- Structural aspects – e.g. adherence to basic programme elements such as hiring high-quality staff or providing participants with the recommended service dosage and duration.
- Dynamic aspects – these relate to quality and content of the relationship between the implementer and the consumer.
There are also two primary components to implementation fidelity – initial and ongoing fidelity.
- Initial fidelity – reflects the implementer’s ability to meet the initial requirements for implementing a new policy or intervention in a new site, e.g. compliance with all training requirements.
- Ongoing fidelity – reflects the implementer’s ability to maintain the implementation and reporting standards of the intervention once a particular site is operational, e.g. the ability to consistently implement an intervention.
Assessment of the dynamic aspects of fidelity may require the development of unique fidelity criteria that relate to the components of policies and interventions being evaluated. These should be objective and measurable.(82, 83) Find templates for the development of tools for the assessment of structural and dynamic fidelity below.