Module 2: Focus the Evaluation. Engage Stakeholders. Develop an Evaluation Question.

Before an evaluation can be conducted, it is necessary to determine the purpose of the evaluation. By focusing the evaluation in this sense, it may allow evaluators to cut down on unnecessary steps in the process, saving both time and money. The evaluation team works towards clarifying the purpose of the evaluation and developing a strong, measurable evaluation question; this is the first step in evaluation and is necessary whether a process, outcome, or impact evaluation is being conducted. To help guide the reader though the first steps of evaluation, a case study has been chosen to emphasize key points. The case study, in its entirety is available here: Zambia (2007): Maternal and Child Health TRaC Study Evaluating the Use of Water Purification Solution Among Women.

Case Study: Evaluation of a water purification solution in Zambia(1)

Since 2005, Society for Family Health (SFH), a local satellite of Population Services International (PSI), has promoted the use of Clorin, a commercially-produced, water purification solution, to women in order to decrease incidence of diarrhea and cholera throughout Zambia.  The intervention includes health education to show the benefits of the solution, as well as social marketing of the product to increase demand.

Engage Stakeholders. Stakeholders can be extremely valuable throughout the evaluation process but it is important to engage them early and often, especially in the planning stages of the evaluation. There are several ways to identify who program stakeholders are, including a semi-systematic stakeholder analysis. In general, stakeholders are the individuals or groups who are involved or affected by outcomes from evaluation.(2) By engaging stakeholders, it will become clear that there are a number of differing opinions and agendas on why and how the evaluation should be conducted, as well as how the resulting information should be utilized. It is the role of the evaluation team to synthesize these opinions, taking into consideration the time, budget, and contextual constraints in order to develop a coherent and useful evaluation plan.(3) If stakeholders are not engaged, the evaluation team risks focusing on elements that are not valued by stakeholders and potentially receiving criticism or rejection from these key individuals.(4)

Case Study Continued: Identifying and Engaging Relevant Stakeholders in Zambia

Population Services International (PSI) conducted the Clorin evaluation and therefore, it was the responsibility of PSI to determine the stakeholders for this project. After several years of implementation the evaluators already had a good understanding of the beneficiaries of this particular project, as surmised from baseline data from two years earlier (in 2005) from women in each province of Zambia.

Although the intervention was aimed at altering women’s water sanitation behaviors, a reduction in diarrhea has a positive effect on children, men, communities, as well as the economy as a whole. In this sense, each community where the intervention took place could be considered stakeholders. The evaluation was funded, in part, by the United States Agency for International Development (USAID), which places USAID in the position of the client, with an invested interest in the outcomes of the evaluation. With limited understanding of the long-term benefits of using Clorin solution, USAID may have been interested in promoting Clorin elsewhere if the project was successful (i.e. scale up). USAID, therefore, has an interest in what data is collected and what the analyzed results show. The producers and marketers of Clorin solution were directly invested as stakeholders because disagreeable findings could have led to diminished profits. Similarly, as program implementers, SFH maintained a stake in the results of the evaluation with contributions to the design of the evaluation and indicators, as well as facilitation of PSI’s access to the target population. Engagement of stakeholders requires collaboration and shared information among all groups to ensure access to target populations and bolster support for their mission in each of the communities. Part of the framework for collaboration involved clarified expectations of each group, along with open communication and the opportunity for candid feedback throughout the process.

Define the Purpose. Once stakeholders have been engaged and the evaluation team has collected the opinions of all parties involved, it is necessary to define the purpose of the evaluation. Particularly, why is this evaluation being conducted, and what do program implementers, stakeholders, and the health field stand to learn from it? The answers to these questions may be derived from a number of programmatic details, such as the goals and objectives of the project.(5) A goal may be broad, such as the endeavor to reduce under-five mortality rates by 15% in the next five years. Based on this goal, evaluators may assess goal attainment based on whether the program was successful in achieving this goal. The evaluation team may also decide, based on input from stakeholders and a limited budget, to look at the adequacy of supply chain distribution of oral rehydration sachets (ORS) to rural clinics, according to program indicators. Both strategies are considered evaluations, though each looks at a different aspect and level of the program. To reiterate, the purpose of an evaluation is often defined in concert between the evaluation team and input from key stakeholders (and a client, if the evaluation team is an external consultant). Defining the purpose allows evaluators to create a substantial, measurable evaluation question upon which the entire evaluation process is built.(6) There are various technical terms for further definition of an evaluation purpose that are discussed in more detail in the Purpose of Evaluation module.

Case Study Continued: Defining the Purpose of the Clorin Evaluation in Zambia(7)

Prior efficacy and cost-effectiveness studies determined that Clorin was effective at preventing diarrhea and cholera and was also a significantly cheaper water purification option than boiling water. With Clorin being widely distributed through commercial outlets, the focus of evaluation was in Clorin use by individuals or communities in need. Therefore, evaluators did not need to perform a randomized control trial but were interested in, if a community or individual has access to the solution, what determines whether they use the solution. With continued high rates of diarrhea throughout Zambia, it is possible that access may not be sufficient to ensure reduced rate of diarrheal disease. Therefore, assessing the determinants of use helps to inform future iterations of the Clorin program in Zambia.

Develop an Evaluation Question. The evaluation question is precisely what the evaluation process seeks to answer. As stated, this is defined by the program, as well as the stakeholders, client, and evaluation team based on the current stage of the project and associated essential information. Process evaluation questions look at how the program was actually implemented. A process evaluation question would ask, did all designated clinics receive shipments of ORS according to the project timeline?(8) This is often the purpose of an evaluation for a program that aims to improve implementation strategies during “roll out” or for future scale up efforts.  Outcome or impact evaluation questions look at how the program activities affect the target population. An example outcome or impact evaluation would ask, did the project decrease incidence of acute diarrhea in children under five? This is often the purpose of evaluation for programs seeking additional or continued funding.(9)

Case Study Continued: Evaluation Questions for Evaluating Clorin in Zambia(10)

Flowing directly from the purpose of evaluation, the question for the evaluation of Clorin was identified: What are the determinants that influence the use of Clorin solution? Classified as a process evaluation, this question sought to determine the factors influencing use of Clorin. A meaningful evaluation question drives all subsequent steps in the evaluation plan: choice of indicators, application of measurement tools, and process of data collection and analysis procedures. Therefore, it is important that this question be exceptionally clear and understood across all parties and stakeholders.

Footnotes

(1) Population Services International (PSI). (2009). Zambia (2007): Maternal and child health TRaC study evaluating the use of water purification solution among women, second round. Washington, DC: PSI Research and Metrics. https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OMKUAT.

(2) Aubel, J. (1999). Participatory program evaluation manual: Involving program stakeholders in the evaluation process. Calverton, MD: ORC Macro Child Survival Technical Support Project (CSTS). https://www.alnap.org/system/files/content/resource/files/main/participatory-program-evaluation-manual.pdf.

(3) Bamberger, M., Rugh, J., and Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(4) Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR, 48(RR11). https://www.cdc.gov/mmwr/PDF/rr/rr4811.pdf. Accessed on 14 June 2019.

(5) Frechtling, J. (2002). The 2002 user friendly handbook for project evaluation. The National Science Foundation, Directorate for Education and Human Resources. https://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf.

(6) Bamberger, M., Rugh, J., and Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(7) Population Services International (PSI). (2009). https://www.psi.org/.

(8) Centers for Disease Control and Prevention. (2018). Developing process evaluation questions. Evaluation Briefs, no. 4. https://www.cdc.gov/healthyyouth/evaluation/pdf/brief4.pdf.

(9) Jones, N. Walsh, C., Jones, H., and Tincati, C. (2008). Improving impact evaluation coordination and uptake: A scoping study commissioned by the DFID evaluation department on behalf of NONIE. Overseas Development Institute. https://www.researchgate.net/publication/260386862_Improving_Impact_Evaluation_Production_and_Use.

(10) Population Services International (PSI). (2009). https://www.psi.org/.