Evaluation of Under Screened Recruitment Program 2014
Professor Margaret Kelaher
The Victorian Department of Health (DH) has funded Cancer Council Victoria (CCV) for three years (2014-2016) to develop, implement and evaluate the Under Screened Recruitment Program (USRP). The USRP aims to increase breast, bowel and cervical cancer screening knowledge, awareness and participation in under screened communities. The under screened communities that this project focuses on include: Aboriginal and Torres Strait Islander (Aboriginal), Culturally and Linguistically Diverse (CALD, and low socio-economic (low SES) communities in Victoria.
As one of the key deliverables for the project, a Partnership Agreement was been established in 2014. The Partnership Agreement outlines the expectations for the partners. In order to enact the agreement, the USRP Advisory Group was established to support CCV to develop and implement the USRP. The Advisory Group includes: CCV (managing USRP and representing components of the National Bowel Cancer Screening Program), PapScreen Victoria, (the Victorian Cervical Cytology Registry, BreastScreen Victoria (BSV) , Ethnic Communities Council Of Victoria, DH Aboriginal health unit and DH (providing oversight, representing the USRP and guidance on the NBCSP).
CCV has contracted Kyabram District Health Service (KDHS) to implement the low SES project. The USRP is working with two Aboriginal communities – one in Bendigo and the other in the metropolitan regions of Melbourne (DH North & West district). USRP is also working with the Chinese (predominately Mandarin speaking) community in the City of Whitehorse.
The evaluation included two key data collection methods. The first method involved reviewing program documentation associated with the implementation of USRP in 2014 (until Dec 2014). Over 130 documents were reviewed including:
- Partnership agreement and contracts;
- Monthly reports (Jan-Dec 2014);
- Interviews and outcomes of community consultations;
- Minutes of USRP Advisory Group meetings; and,
- Planning and review documents
- Evaluation materials and program logics prepared by CCV.
In 2014, the USRP achieved most of its objectives in terms of developing the partnership agreement, reporting, community consultation, establishing local partnerships and developing 2015-16 plans. There was evidence of adaptive learning and responsiveness of key stakeholders particularly documenting community engagement, the nexus between local and expert inputs and governance.
Overall, there was a strong commitment by members of the USRP Advisory Group members to the USRP aim of improving participation in screening among under screened populations though a community development approach and with a focus on joint screening. There was also a high level of recognition of the value of USRP Advisory Group and the strength of the partnerships developed as part of the initiative.
Progress has been slower than anticipated due to delays in selecting communities/locations, methodology change, extended time to build relationships with communities and resourcing/capacity to implement the program. The plans that have been developed still require significant work to assess whether they will lead to the development of screening strategies, how feasible and implementable they are, whether they meet community needs and, ultimately, whether they will increase screening. This is a considerable issue given the short time frames of the USRP and the expectation that it will influence participation and provide learnings that are transferable to other communities.
The results of evaluation suggest 5 key areas for improvement to facilitate the development of the USRP to the critical implementation phase.
Recommendation 1: Roles and Responsibilities in relation to contract management, participation in the USRP Advisory Group and achieving deliverables should be documented to ensure greater accountability and transparency.
The USRP combines a multi-layered governance structure with complexity of working in a community development framework. The Partnership Agreement that guided USRP activities in 2014 did not clearly define the roles of the partners individually or as part of the USRP Advisory Group in terms of decision-making. A lack of clarity about roles and responsibilities led to frustration and concern about lack of appropriate and timely engagement between partners. It also acted to reduce the trust and goodwill that the partners brought to the process. This has been recognised by all partners and measures are in place to ensure roles are more clearly defined in the future and in turn to ensure greater transparency and accountability.
Recommendation 2: CCV should incorporate specific strategies for bringing together local and expert advice in planning deliverables. Reasonable time frames for response should be included.
The USRP provides an important platform to bring together expert knowledge and local expertise. The evaluation found that there were concerns about the quality of plans and lack of innovation from experts. At the same time there were concerns about lack of support for local staff and CCV. These issues arose confusion from the partners about their roles in the process and the process itself. This suggests that specific measures should be put in place to facilitate the confluence between expert and local knowledge and foster an environment where meaningful dialogue can occur.
Recommendation 3: CCV with DH, USRP Advisory Group and local partners should develop a plan to improve communication associated with the development and delivery of the USRP.
The USRP requires a relatively high level of reporting. This is reflected in large number of documents reviewed as part of this evaluation (>130). Despite this several significant misunderstandings have arisen in the course of 2014. Based on these findings, it seems quality of communication rather than the quantity of communication is the issue. It may be useful to review various mechanisms for communication in order to streamline reporting and improve the level of communication.
Recommendation 4: CCV should work with local partners to develop project plans that are sufficiently detailed to enable an assessment of the feasibility, appropriateness, acceptability and likely effectiveness of proposed strategies.
Moving the USRP successfully into implementation will require building on current planning documents to develop project plans that are sufficiently detailed to enable an assessment of the feasibility, appropriateness, acceptability and likely effectiveness of proposed strategies to increase screening participation. For example, if strategies specifically target only those people who are eligible for all three types of screening this has a major impact on the size of the population who could potentially impact. Similarly, some communication strategies will reach a whole community and others just pockets. The partners were very supportive of the joint screening approach although they did recognise feasibility issues. The reviewed project documentation did not include any background analysis of the benefits or risks associated with a joint rather than individual approach to improving screening participation, particularly in the local context. Ensuring that USRP resources are optimised requires careful consideration of the costs, benefits and risks associated with the different approaches. The processes for achieving this are outlined in the USRP 2015-16 plans.