Course 1: Evidence-Based Program Design

The Importance of Evidence-Based Social Interventions

Non-governmental and governmental organizations worldwide implement programs to combat social problems, including poverty and lack of adequate health care.  However, the programs are often designed and executed based on assumptions rather than based on data and facts.(1)  In her TED talk entitled “Social Experiments to Fight Poverty,” MIT economist Esther Duflo compares the implementation of social programs that are not evidence based to the use of leeches by doctors in the medieval period.(2)  Doctors used leeches due to prevailing assumptions about the body and illness during that period. Sometimes the leeches worked, but they were oftentimes ineffective. In some cases, the leeches caused blood loss that exacerbated the patient’s condition. Centuries later, evidence-based medicine and rigorous analysis became central to medical practice. Until recently, social policies and interventions have been developed and implemented based on assumptions rather than evidence. Evidence-based public health programs are essential.

The nonprofit organization Innovations For Poverty Action further explains that “Two voids exist in developmental policy: insufficient incorporation of results from social science research, and insufficient evaluation (in particular, replication of studies) to learn concretely what works, what does not, and why.”(3)  Some organizations and research centers have recently begun conducting evaluations of various interventions to determine their efficacy in practice rather than in theory.   Their overall goal is to positively influence the design and implementation of policies and programs by international organizations, non-governmental organizations, and governments. 

Design of Evaluations and Sample Interventions

When developing new interventions or programs, it is important for program implementers to utilize existing research to determine “how people make financial decisions, adopt new technology, use social networks to help survive crises, respond to incentives, decide how much education to acquire, etc.” (4)  This application of academic theory is needed to develop sample interventions.  Evans et al. note that “Better understanding of the barriers (at the person, family and provider level) is essential before strategies can be devised, and these strategies need to be evaluated carefully. Only then can we know if we are doing more good than harm and spending limited resources wisely.” (5) Abdul Latif Jameel Poverty Action Lab in Boston integrates social and behavioral sciences and public health research, which were previously primarily confined to academia, into policy design.   This type of important research allows program designers and implementers to determine the expected efficiency, effectiveness, and potential impact of a new program idea prior to implementation.

After a program has been implemented, it is crucial to implement ongoing assessments and evaluations.  Randomized evaluations, which are used extensively by the Abdul Latif Jameel Poverty Action Lab (J-PAL), are a type of impact evaluation.  A randomized evaluation may also be known as randomized controlled trials, social experiments, random assignment studies, randomized field trials, and randomized controlled experiments.  J-PAL emphasizes that randomized study designs provide the most accurate, unbiased, and reliable statistics out of all other impact evaluation options.  In order to determine a program’s impact, it is essential to have a randomly-selected control group of participants who are statistically identical to the experimental group.  Both of the groups are considered microcosms of the larger population and therefore equal in representation to each other.   When applying this design, any changes between the control and experimental group can be attributed almost solely to the effects of the program.(6)  For more information on randomization and reliable data, see Validity of Research and Quality Health Data. 

The Purpose of Evaluation

Abdul Latif Jameel Poverty Action Lab discusses the goals of evaluation: "If thoughtfully designed and implemented, it [randomized evaluation] can answer the questions, how effective was it?  Were there unintended side-effects?  Who benefitted most?  Who was harmed?  Why did it work or not work?  What lessons can be applied to other contexts, or if the program was scaled up?  How cost-effective was the program?  How does it compare to other programs designed to accomplish similar goals?"  (7)

After randomized evaluations have been conducted on test interventions and have answered these chief questions, the results can be utilized by program designers and program implementers.  Based on the research and considerations of context, policymakers can decide which interventions are best to scale up. Evidence-based social interventions maximize positive social outcomes and ensure that policies and programs are effective rather than harmful to the target population.   Foreign aid interventions illustrate the necessity of both of these goals.  For generations, the debate on foreign aid has been wrought with controversy, mostly because evidence has not been developed or analyzed to assess impact.  For example, Duflo notes that we do not know whether Africa would be better off now if we had given more aid or less aid since 1970. (8)   Without supporting evidence, implementation of policies and interventions can be a waste of resources or worse, exacerbate social problems. (9)

Examples of Implementation Without Evaluation           

Case Study: Health Fairs Lowering Quality of Care

While those who implement health fairs have good intentions, the tests administered at health fairs are not regulated and may often provide inaccurate results.  For example, hypertension screenings, which are often performed at health fairs, should be performed multiple times over a certain period because “Single reading screens have been shown to result in 33% false negatives (missed hypertensives) and up to 66% false positives.”(10)   As discussed at  Challenges and Failures of Health Fairs and Community Screenings, “health fairs are not a substitute for seeking comprehensive care with a physician and are potentially harmful because they may cause those who receive false negative results to think that they do not need to seek comprehensive medical care when they actually do.”  This example illustrates that well-intentioned health interventions can in practice be detrimental to patients in the long term and that evaluation can help expose these harmful results. 

Case Study: Injections Causing the Spread of Blood-Borne Pathogens

“In Africa, 20 million medical injections contaminated with blood from a patient with HIV are administered every year.”(11)  In India in 2009, a 38 year old woman who had been treated for ringworms died of hepatitis B due to her doctor’s reuse of needles.  An additional 29 people in Modasa, Gujarat died due to contaminated needles over the course of one week.  This doctor and others in Gujarat are now being accused of homicide.  These doctors may have been reusing needles to save money.  Other healthcare professionals may reuse needles due to lack of knowledge or resources.  Through India’s public healthcare system, a mass vaccination program is being implemented to vaccinate 300,000 people.  Out of 495 people already treated, however, 92 have died due to unclean syringes.(12)  In fact, Marc Koska, founder of SafePoint Trust charity notes that 62% of injections in India are unsafe.   In a TED video, he discusses the widespread reuse of needles and syringes in Indian hospitals. He notes that while malaria kills 1 million people per year, unsafe injections kill 1.3 million people per year.  He shows the pictures of two children who were infected with HIV as a result of contaminated needles and then were abandoned by their parents because of their HIV+ status.  While syringes enable the delivery of vaccinations and life-saving medications, unsafe injections cause significant harm. (13)  Unfortunately, many international organizations and Ministries of Health promote the mass distribution of vaccines and other essential medicines without evaluating their administration, which ultimately contributes to significant mortality.(14)  Research and ongoing evaluation of every implemented program is essential to ensure positive outcomes. 

Examples of Evidence-Based Interventions

Case Study: Increasing the Use of Malaria Nets

We know that insecticide-treated malaria nets can drastically reduce rates of malaria.  However, how can one ensure that the malaria nets are used as intended?  Randomized controlled trials must be conducted.  Duflo states that with the scientific tests, “We can take the guesswork out of policy-making by knowing what works, what doesn’t work, and why.”  Through randomized controlled trials in Kenya, it was found that the more expensive the malaria nets, the less likely people are to buy them.  If people receive malaria nets for free or purchase them, they are equally likely to use them.  The research also found that if malaria nets are distributed for free, recipients will buy malaria nets in the future, indicating that markets are not negatively affected.  These findings suggest that in order to increase the use of malaria nets most effectively and without harm to local markets, malaria nets should be distributed for free.   As a result of this research, Population Services International (PSI), which provides malaria control support to over 30 countries’ Ministries of Health, now promotes the free distribution of malaria nets.  In Kenya, PSI has scaled-up the distribution of free malaria nets to pregnant women through 3,000 antenatal clinics.(15)

Case Study: Reducing Incidence of Diarrheal Disease

J-PAL conducted a study in Kenya and Pakistan to determine the most cost-effective means of cleaning water and reducing incidence of diarrhea.  The study dispensed free chlorine at water sources, delivered free chlorine to households, delivered  free chlorine to households with a clean storage container, encased sources of spring water to prevent ground water contamination, and distributed free soap with promotion of hand washing.  Out of all of these interventions, dispensing free chlorine at water sources with paid community promoters was determined to be most cost effective, averting 689 incidents of diarrhea per $1,000 spent on the intervention.  In order to reduce incidence of widespread diarrheal disease, this intervention can be explored in many other localities with inadequate clean water supplies.(16)

Organizations that Conduct and Utilize the Results of Randomized Evaluations

The Abdul Latif Jameel Poverty Action Lab was created in 2003 as an association of academics applying randomized evaluations for poverty alleviating programs.  The J-PAL website notes, “J-PAL’s policy group works with governments, NGOs, international development organizations, foundations and donors to help them determine the most cost-effective ways to achieve certain outcomes in the fields of health, education, governance, environment, agriculture, and microfinance.”  In addition to J-PAL, research centers that focus on evaluation of social policies and programs include Innovations for Poverty Action, Center for International Development’s Micro-Development Initiative, Center of Evaluation for Global Action (CEGA), and Ideas 42.  Since its foundation, J-PAL has worked to conduct randomized evaluations with over 200 organizations including the Bill & Melinda Gates Foundation, the World Bank, Acumen Fund, CARE, Oxfam, and USAID.  The governments of Kenya, Sierra Leone, and some states in India have partnered with J-PAL and utilized their research as well.(17)

Footnotes

(1) “MSc / MPhil in Evidence Based Social Interventions (EBSI).” University of Oxford. 

(2) Duflo, Esther. “Social Experiments to Fight Poverty.” (2010) TED: Ideas Worth Spreading. https://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty/transcript?language=en.

(3) “Innovations for Poverty Action.”  https://www.poverty-action.org/.

(4) Ibid.

(5) Evans, J., Smeeth, L., and Fletcher, A. “Vision Screening.” British Journal of Ophthalmology. 93 (2009): 704-705.

(6) “J-PAL: Abdul Latif Jameel Poverty Action Lab.”  https://www.povertyactionlab.org/.

(7) Ibid.

(8) Duflo, Esther. “Social Experiments to Fight Poverty.” (2010) TED: Ideas Worth Spreading. https://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty/transcript?language=en.

(9) “MSc / MPhil in Evidence Based Social Interventions (EBSI).”  Department of Social Policy and Intervention at University of Oxford. 

(10) Getliffe, K., Crouch, R., Gage, H., Lake, F., and Wilson, S. “Hypertension awareness, detection and treatment in a university community: results of a worksite screening.” Public Health. 114 (2000): 361-366.

(11) “The Climb Tanzania 2010 is launched.” http://safepointtrust.blogspot.com/2010/06/.

(12) “SafePoint Trust.” http://www.icapcharityday.com/charities/2015-the-safepoint-trust.

(13) Koska, Marc. “1.3m Reasons to Re-Invent the Syringe.”  (2009) TED: Ideas Worth Spreading https://www.ted.com/talks/marc_koska_1_3m_reasons_to_re_invent_the_syringe?language=en.

(14) “SafePoint Trust.” http://www.icapcharityday.com/charities/2015-the-safepoint-trust.

(15) Duflo, Esther. “Social Experiments to Fight Poverty.” (2010) TED: Ideas Worth Spreading. https://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty/transcript?language=en.

(16) “Child Diarrhea.” J-PAL: Abdul Latif Jameel Poverty Action Lab.  

(17) “J-PAL: Abdul Latif Jameel Poverty Action Lab.”  https://www.povertyactionlab.org/.