Module 6: Impact Evaluation

“With the growing emphasis on the assessment of aid effectiveness and the need to measure the results of development interventions, it is no longer acceptable for governments, official development agencies, and nongovernmental organizations (NGOs) to simply report how much money has been invested or what outputs have been produced. Parliaments, finance ministries, funding agencies, and the general public are demanding to know how well development interventions achieved their intended objectives, how results compared with alternative uses of these scarce resources, and how effectively they contributed to broad development objectives such as the Millennium Development Goals and the eradication of poverty.”

-World Bank Independent Evaluation GroupAn impact evaluation seeks to determine the link between a program and subsequent improvements in the welfare of the target population. Specifically, impact evaluations determine whether or not a project’s activities can be directly associated with improvements seen in the target population, and state with increased certainty that such improvements are not due to any external sources like local or national policies, major environmental events, or political hostilities.(1)Impact evaluations (IE) are not always mandatory, depending on the kind of project and purpose of the evaluation, but contribute to the evidence-base which benefits future programs and policy; this is because impact evaluations typically allow for researchers to state, based on statistical significance, that improvements in health are attributed to project activities.(2)(3)  The movement towards evidence-based policy and global health programming is a manifestation of the need for cost-containment and proof of effectiveness prior to large-scale implementation. Although contribution to the evidence-base is laudable, considerations of time and budget need to be made prior to embarking on impact evaluations; impact evaluation can be extensive and budget-consuming which is one of the main reasons why they are not ubiquitous in global health. If stakeholders, clients, funders, and evaluators want to determine the causality between program activities and impacts, then evaluators should consider doing an impact evaluation.Many elements of impact evaluations have already been discussed in this course: study design, process evaluation, ethics, focusing the evaluation, and understanding constraints. With this foundation, we can examine a case study that demonstrates the overall concept of an impact evaluation. The components of many IEs are generally standard; developing an evaluation question, determining the best ways to measure the evaluation question (see module on Study Design), locating or creating a control group from historical data or using a group of individuals who do not receive the intervention, collecting data through quantitative or qualitative methods and then analyzing the data.

Case Study: Impact of Bed Nets in Togo(4)(5)

Integrated distribution of long-lasting insecticidal nets (LLINs), which is a distribution method that pairs free LLINs with immunization programs, had some success via two pilot programs in sub-Saharan Africa. In 2004, Togo implemented a similar system using a measles immunization campaign to distribute the LLINs with the expected impact to lower rates of child morbidity and mortality. A short-term impact evaluation was planned to show the effect of integrated distribution of LLINs.Both ethics and context played a large role in determining what kind of study design was used. With the effectiveness of bed nets well established in developing countries, along with the pre-existing wide distribution of nets, it is both unethical and logistically difficult to create a control group for this assessment. In terms of the purpose of the evaluation, a probability assessment is unnecessary because of the founded evidence toward both integrated distribution and the effectiveness of bed nets. Therefore, this evaluation is considered a plausibility assessment. Evaluators chose to use a ‘historical control group,’ meaning that one randomized cluster of villages would be used as both the control group prior to joint intervention of immunizations and bed nets and as the intervention group following implementation. This differs from more traditional control groups where there are two separate groups with one receiving the intervention and not the other. See Study Designs for more examples of these types of control groups.

Evaluators used the same 90 locations for randomized household selection at both time points, though they did not follow specific respondents over time (i.e. this was not a longitudinal study). Instead, a cross-sectional study design was used, where both measurements were “snapshots” of the health conditions in those areas at those times.

A random sample of households was taken from 90 different rural areas. One household was defined as a primary caregiver and his/her children. A survey was administered to the caregiver to collect medical and demographic data, and a blood sample was drawn from any children under 59 months for biological testing purposes (rapid diagnostic tests, blood smears, and hemoglobin counts). This measurement was taken twice from 2,532 children at three months prior to distribution and nine months after distribution. LLIN use was confirmed via verbal response during the survey, as well as a visual confirmation during household selections to ensure the net was hanging properly.

Researchers found both an increase in LLIN use and reduction in malaria-associated morbidities (i.e. moderate and severe anemia) among the surveyed villages after the Togo Child Health intervention.

With a less-than-robust study design, evaluators found a number of limitations when analyzing the data. Primarily, by using a historical control group (one group serving as both the control group prior to intervention and the intervention group following implementation), researchers were unable to separate the effects of integrated intervention in child health from the benefits of only LLIN use. For instance, some aspects of intervention, such as treatments and supplements, would have had similarly positive effects on rates of anemia. However, with a more significant drop in anemia among children who were reported as using an LLIN, researchers were able to associate the impact on health with LLIN distribution in the 2004 program through the use of a plausibility assessment. These results indicate that the use of LLIN distribution in tandem with other national child health programs may be beneficial. The reduction of malaria-associated morbidity correlated with the LLIN distribution methods used in Togo is further evidence in support of the national malaria reduction strategy.

Footnotes

(1) Khandker, S.R., Koolwal, G.B., Samad, H.A. (2010). Handbook on impact evaluation. Washington, DC: The International Bank for Reconstruction and Development/The World Bank. http://documents.worldbank.org/curated/en/650951468335456749/pdf/520990PUB0EPI1101Official0Use0Only1.pdf.

(2) Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B., and Vermeersch, C.M.J. (2011). Impact evaluation in practice. Washington, DC: The International Bank for Reconstruction and Development/The World Bank. https://pdfs.semanticscholar.org/42c8/b526b2d095eeaae15335f98c1e6d7316fe2e.pdf.

(3) United States Department of Agriculture (USDA). (2005). Nutrition education: Principles of sound impact evaluation. Washington, DC: Food and Nutrition Service, Office of Analysis, Nutrition and Evaluation. https://fns-prod.azureedge.net/sites/default/files/EvaluationPrinciples.pdf.

(4) Terlouw, D.J., Morgah, K., Wolkin, A., Dare, A., Dorkenoo, A., Eliades, M.J., Eng, J.V., Sodahlon, Y.K., Kuile, F.O.T., and Hawley, W.A. (2010). Impact of mass distribution of free long-lasting insecticidal nets on childhood malaria moribidity: The Togo National Integrated Child Health Campaign. Malaria Journal. 9:199.

(5) Eliades, M.J., Wolkon, A., Morgah, K., Crawford, S.B., Dorkenoo, A., Sodahlon, Y., Hawley, W.A., Hightower, A.W., Kuile, F.O.T., and Terlouw, D.J. (2006). Burden of malaria at community level in children less than 5 years of age in Togo. The American Journal of Tropical Medicine and Hygiene. 75(4):622-629.