Course 1: Outcomes are Essential in Global Health

In performance assessment, outputs are defined as the goods or services produced by programs or agencies, whereas outcomes are defined as the impact that those outputs have on social, economic, or other indicators.(1) Typically, outputs are used to document the amount or volume of use of the project's products or services. While outputs are important to track, evaluation needs to focus on measuring outcomes that reveal the extent and kinds of impact that the project has on its participants. Impact could be reported in the amount of change in behavior, attitude, skills, knowledge or condition of the target population. For example, an output would be the number of microfinance loans distributed, or the number of loans repaid. This type of information is not very useful when seeking to determine the true effectiveness of the intervention. A microfinance recipient might repay a loan by taking an additional loan from a different microfinance institution. Therefore, repayment of a loan does not indicate that the individual has implemented a business that has enabled them to increase their income and repay their loan. In contrast, the outcome would be the number of microfinance participants who have significantly increased their income or risen out of poverty due to the loans. Similarly, the output for a job training program is the number of people enrolled in the program.  However, perhaps the job training program is so ineffective that none of the participants have been able to secure a job. The measurable impact is the outcome, which reports on the number of people who were able to get a new, higher-paying job due to the training program, as measured by income improvement or poverty reduction.

The Importance of Measuring Outcomes

Case Study: Mosquito Nets 

The dichotomy between outcomes and outputs can be seen in the case of mosquito net distribution.  The Kenyan newspaper The Daily Nation reported that many people are using mosquito nets for alternate activities such as for fishing and making wedding dresses, especially in the Nyanza Province.  In response, mosquito net manufacturers are collaborating with the local government to prosecute people who use the products for purposes other than covering beds. Dr. Juma, head of malaria control under the Ministry of Public Health and Sanitation remarked:

“This is wrong and totally unacceptable. Bed-nets are supposed to play a noble role of preventing deaths that would be caused by malaria. I think there is need for further sensitization and education for the entire public to understand the importance of sleeping under insecticide treated mosquito nets.”(2)

In this case, the output was the number of mosquito nets distributed to villagers.  If evaluation metrics merely looked at this number, the program may have been seen as a success.  However, the real impact metric is the outcome. Are the malaria nets being utilized, and are the rates of malaria therefore decreasing? It is important to recognize that it takes far more than just handing out nets to change behavior. Without metrics, global health programs may fall into the pitfall of spending resources on the quantity of an intervention, rather than working to implement a quality program that achieves a real impact. Programs that distribute mosquito nets, condoms, and other supplies are common.  While these interventions are popular because they can “help” a lot of people, there is often a discrepancy between their outputs and outcomes.

Case Study: PlayPumps

PlayPumps is an organization that created and distributed an innovation that enabled children to play on a merry-go-round while pumping clean water into a storage well. PlayPump’s main goal was to bring clean drinking water to communities in sub-Saharan Africa, and the organization boasted impressive outputs. By 2007, it had installed over 900 PlayPumps and had the goal of installing 4,000 PlayPumps by 2010.(3) Unfortunately, PlayPumps failed to look at the outcomes of their installations and did not realize their negative impact.  The organization failed to assess whether the PlayPumps actually improved the supply of clean water, nor the amount of time people spent gathering water in villages where they were installed. “The promise of PlayPumps remains largely unfulfilled. In some cases, the PlayPumps actually made situations worse, displacing previous water sources.” (4) Many PlayPumps were unused, while others broke and could not be repaired due to a lack of spare parts for repairs. Villagers reported that the PlayPumps actually made their living situations worse. However, the organization only focused on its outputs, seemed unaware of its negative outcomes, and the organization continued to install increasing numbers of PlayPumps.(5) Eventually, the organization failed and dissolved. This case illustrates the importance of conducting follow-up studies with the local population to assess the impact of an intervention.  If PlayPumps had assessed their impact by following up with local communities, the organization would have realized that the PlayPumps were oftentimes not actually improving access to clean water; rather, they were causing more harm than good.

Case Study: Cell Phone Loans in Rwanda

An organization decided to loan cell phones to villagers in Rwanda, thereby allowing them to open a mobile phone business for selling air time to others in the village. The organization reports that it now has more than 5,000 Village Phone operators providing phone service in Rwanda. However, the fact that the organization has loaned 5,000 phones to village operators says very little about the effects of making the cell phones available. How have the phones impacted the lives of the operators and the communities at large? Did the phones increase the phone operators’ daily income? These are questions that outputs cannot and do not answer. Fortunately, an independent study was conducted in 2009 that assessed the impact and outcomes of the village phone product in Rwanda. The authors of the study found that “impacts at the household level were muted by the relatively small size of village phone businesses and airtime usage rates, implying that profit must be transferred from other sources in order to pay off the phone in six months… positive impacts on consumption or overall business profits were not found.”(6) Moreover the study found that the Village Phone enterprise is uniquely time consuming, yet it generates revenues, costs and profits that are lower than other enterprises. “The average mechanical profits from operating the phone are $30.50 per month in our sample, meaning that the profits from the phone only cover 70% of the debt-servicing costs.” (7) Since the average profits associated with the phone did not cover the amount needed to pay back the loan, the village phone program may actually have had a detrimental effect on those that they sought to help. This study again highlights the importance of measuring the outcome of a program and shows how outputs alone do not accurately reflect a program’s value.

Measuring Outcomes

Misrepresenting Results by Only Reporting Outputs

Though the importance of measuring outcomes is clear, many organizations unfortunately only report and/or measure their outputs. For example, an organization which distributes bed nets has an abundance of data about the number of nets they distribute. They report that 53% of the population in sub-Saharan Africa is now protected by bed nets compared to 2% in 2000, but there is no data available on bed net usage, or their impact on malaria rates.(8) Similarly, another organization whose mission is to reduce poverty in Bangladesh by providing daily healthy nutrition in the form of yogurt only reports output data. For example, in 2007 they sold 3,414 cups of yogurt per day, while in 2008 the number rose to 6,532 cups per day, and it rose even further in 2009 to 35,319 cups per day. While the organization has extensive data on the amount of yogurt distributed, there is no data assessing whether the distribution of yogurt has reduced poverty or decreased childhood malnutrition in Bangladeshi children. (9)

Some Organizations Report Theoretical Outcomes

Some organizations use their outputs to extrapolate theoretical outcomes. For example, an organization which distributes light bulbs in developing countries sold two million lights by the end of 2010, which is an output. They used theoretical calculations to reach conclusions on their impact. The organization states that by distributing 2 million light bulbs, there would be $60 million dollars in savings for families who are no longer dependent on kerosene, and $65 million dollars in increased productivity. However, these calculations have not been assessed on the ground. Without collecting data from actual people who have received the light bulbs, the organization’s impact and outcomes remain unknown.(10) Additional questions need to be explored as well. For example, are the lightbulbs still operational after 5, 10, or 90 days? Are the lightbulbs being utilized by those who purchased the product?

An organization which strives to bring low-cost reading glasses to the developing world has assessed its impact in a similar manner.  The organization claims that a pair of their reading glasses can increase a customer’s productivity by 35%. Therefore by multiplying the average number of working days in two years by average daily income, and then by 35%, they claim that each $4 pair of glasses can produce $381 in increased earning over two years. The organization goes on to extrapolate that they have generated more than $230 million in economic impact due to the 610,000 pairs of eyeglasses sold.(11) However, this is a purely theoretical calculation and does not say anything about the organization’s actual impact. One does not know if the eyeglass purchasers are wearing their reading glasses 90 days, or even 5 days, after making their purchase. One also does not know whether the eyeglass purchasers have indeed increased their productivity by 35%. Additionally, an increase in one's productivity oftentimes does not translate to an increase in salary. For example, the eyeglass purchaser might be a rickshaw driver, who needs quality distance vision in order to succeed at his job. Purchasing reading glasses may now allow him to read or see near objects, but this does not increase or enhance his ability to drive the rickshaw. Though he purchased reading glasses and may consequently be more productive at reading, he is unlikely to see a change in his income. Likewise, a farmer's income largely depends on crop yield, and this is not impacted by one's use of reading glasses. While some reading glass purchasers may increase their income, one certainly cannot assume that most or all of the reading glasses sold will produce $381 in increased earning over two years for each person. Likewise, until and unless there is an actual assessment of outcomes, it is impossible to determine the organization's actual economic impact from only theoretical models.

Similarly, an organization which provided bicycles to people in Sri Lanka after the 2004 tsunami reported that it donated 24,300 bicycles. Though it has released a report on its impact, it includes largely theoretical calculations. The report claims that bicycles have the potential to save a recipient 1.7 hours per day in transportation time. The report goes on to state that if the time saved were spent on productive activities, then $0.52 per day in additional income could be produced. This is a very nice theoretical calculation, but there is no actual data provided. The theoretical calculations do not take into account whether people are actually saving that much time, or how they are spending the time they’ve saved. Do they choose to spend it on incoming-generating activities? Are the bicycles being used each day, or perhaps once per month? If the bicycle owner usually walks only 15 or 30 minutes per day, then using the bicycle will not reduce travel time by 1.7 hours. Additionally, one does not know whether the bicycle is operational, or in disrepair and not being used at all.

If PlayPumps had similarly used theoretical calculations using their outputs, they might have reported that with the 900 PlayPump installations by 2007, X number of households had access to clean water. They may have then multiplied this number by the expected reduction in water-borne disease (Y%) that is expected to occur with access to clean water. This would enable them to theoretically report how much disease was prevented by PlayPumps. They might also have calculated the increased life expectency due to the 900 PlayPump installations. In reality, however, PlayPumps oftentimes fell into disrepair and prevented the communities from accessing clean water. In other communities, the PlayPumps replaced existing working hand pumps which were already providing clean water.

As we can see in these examples, theoretical calculations do not provide any information about impact or outcomes and are not appropriate for monitoring and evaluation reports. It is essential for organizations to assess the actual impact of programs on the ground.(12)

Essential Reporting: Actual Outcomes

In contrast to the examples that have previously been discussed, KickStart is an organization that develops and markets new technologies in Africa and conducts thorough impact assessments.  The organization explains that “we could base our claims of success on the number of pumps we've sold to-date. But this tells us nothing about whether we are meeting our mission--helping people get out of poverty. To know this, we have to measure how much more money the buyers of our technologies earn as a result of owning them.”(13) In order to do so, KickStart has developed a systematic, replicable method to measure their impacts. All of their products come with a one-year guarantee, and every buyer fills out a guarantee form when they buy the product. The guarantee reduces the perceived risk of buying the product, and the forms give KickStart a database of all pump owners. From this database, KickStart selects a statistically valid sample of recent purchasers. These customers are then visited within a month of purchasing the products, before any impacts have been realized, then again at eighteen months, and again three-years after purchase to assess how the products have affected their customers. (14)

Endeavor Global is another organization which measures its outcomes directly. Endeavor’s mission is to transform emerging countries by establishing high-impact entrepreneurship and promoting sustainable economic development. The company has comprehensive statistics about its impact on promoting financial capital, human capital, social capital, intellectual capital and cultural capital. Instead of just stating that the organization has worked with over 550 entrepreneurs (output), it measures how those entrepreneurs are affected by working with Endeavor. For example, the company has measured the average growth rate of entrepreneurs’ businesses after engaging with Endeavor at 64%.(15)

VillageReach is an international NGO that works to improve access to healthcare for remote, underserved communities and launched a 5-year pilot project to ensure prompt and universal access to vaccines in Cabo Delgado, a province in northern Mozambique. In order to assess their outcomes, they conducted a comprehensive evaluation analyzing the impact of VillageReach’s pilot project in Cabo Delgado. The evaluation found that VillageReach’s project increased immunization rates from 68.4% to 95.4%, reduced vaccine stockouts from 1% in 2006 compared to almost 80% in 2004, and improved training and supervision for health center staff.(16) Furthermore, “attributing these impacts to the Project is supported by the comparisons to Demographic and Health Surveys data from 1999 and 2003, other administrative data, and the fact that vaccination coverage rates in the neighboring province of Niassa, where the project did not undertake any activities, were significantly lower than those found in Cabo Delgado for a similar time period. In addition, no other organizations were working to improve vaccination coverage in all districts of Cabo Delgado during the project period.” (17) By conducting an outcomes assessment, VillageReach has shown the program’s impact on increasing vaccination rates in Mozambique.

One Acre Fund is an organization that helps East African farmers grow their own way out of hunger by providing small-scale farmers with an investment package of high-quality seed and fertilizer, education courses, and access to output markets.  The organization thoroughly and comprehensively measures their impact and outcomes. They not only report how many families they have reached (output), but they also report the average impact they have on a family. For example, in 2010, they reached 28,000 families, and those families who received help from the One Acre Fund doubled their income per year compared to matched controls. Every six months, the organization puts together a performance report stating how many families they have reached, how participants’ income has been affected, and the %age of farmers that are able to repay their program fees.(18)

Unite For Sight invests human and financial resources in order to develop outreach infrastructure at eye clinics in developing countries. Unite For Sight annually assesses the increase in each eye clinic's cataract surgical rate, targeted for patients living in poverty. While Unite For Sight tracks the number of patients living in poverty who receive eye care through Unite For Sight's outreach programs (output), the organization also evaluates the number of patients with significantly improved vision after surgery. Unite For Sight requires extensive documentation from each eye clinic partner, including every patient's preoperative and postoperative visual acuity data, which is analyzed to ensure the quality, effectiveness, and social impact of the programs.

Surgical Outcomes

Oftentimes medical missions or organizations will simply report the number of surgeries that they have sponsored or provided, while they neglect to take into account the surgical outcomes. Just because an organization has provided a large number of surgeries does not mean that it is conducting valuable or quality work. Surgical outcomes must be recorded and analyzed. For example, the patient below reported that he had cataract surgery years ago with the hope that his sight would be restored. Unfortunately, due to low quality of care and no follow-up care, his eye was destroyed. The organization that provided this surgery can claim they provided a surgery (an output) to this patient. However, the patient's vision was not restored, and he instead had a severe eye infection that caused permanent and irreversible blindness.

As the demand and volume of surgery has increased dramatically in all parts of the world, ensuring the safe delivery of high quality surgical care is a major public health concern. The Surgical Apgar score was developed as a simple, low-cost metric that provides rapid feedback to surgery teams in many settings. The Surgical Apgar Score is a 10-point tally based on the three parameters: the estimated intraoperative blood loss, the lowest heart rate, and the lowest mean arterial pressure. The score is designed to provide rapid feedback for clinical teams and is predictive of morbidity and mortality, even after controlling for preoperative patient factors. A study which evaluated the Surgical Apgar Score in eight countries found that a higher score was correlated with lower rates of complications. For example, those who scored 0-4 had complication rates of 32.9% and death rates of 7.9%, while those who scored a 10 (the best score) had complication rates of 3% and death rates of 0.5%. (19) This study corroborates the importance of measuring outcomes and shows how the Surgical Apgar score is an accurate predictive measure for doing so.

Context and Nuances Matter

It cannot be assumed that a program will work just because it worked in another location, or by another organization. Similarly, it cannot be assumed that just because a program produced a certain outcome in one context, that it would produce the same outcome in another. A study conducted on microfinance about women and masculinity found that “much of the current debate and polemic focus on attempts to answer a complex question in a simple manner: Have women been empowered or not? In essence, experts look for a zero-sum answer. But it is entirely possible that microcredit has empowered some groups of women in certain ways while disempowering others.” (20) The fact that microcredit may have different impacts on women shows how one cannot make assumptions about the impact that a program or intervention will have just because it worked previously in a different context. Therefore, when implementing any program, it is essential to monitor and evaluate it continuously, even if it has been carried out in the past.

Conclusion

When assessing the impact of an organization or an intervention, it is critical to keep in mind that distribution does not equal value. Just because an organization has given out a certain number of bed nets or reading glasses and has performed a certain number of surgeries does not mean anything by itself. In order to assess the impact of an organization, it is necessary to see what effect the bed nets or eyeglasses have on a community. Have they decreased malaria rates or increased productivity? Similarly were the surgeries of high quality and associated with minimal post-operative complications? It is critical that these types of questions are assessed and that outcomes are measured. It is not a productive use of resources if bed nets and eyeglasses are given away and not used properly by the people they attempt to help, or if surgeries cause more harm than good. Thus, for an organization to make positive, lasting sustainable change it is critical that they measure outcomes and tailor their programs accordingly.

Footnotes

(1) McAllister, K. (1999). Understanding Participation: Monitoring and Evaluating Process. Outputs and Outcomes, IDRC, Ottawa.

(2) Esipisu, I. "Villagers using bed-nets for wedding gowns." (22 April 2009). Daily Nation. https://www.nation.co.ke/lifestyle/artandculture/1222-562818-4b7r61z/index.html.

(3) “Troubled Water.” (29 June 2010). Frontline. https://www.pbs.org/frontlineworld/stories/southernafrica904/video_index.html.

(4) “The Promise and Perils of PlayPump.” (7 July 2010) The World. https://www.pri.org/stories/2010-07-07/promise-and-perils-playpump.

(5) “Troubled Water.” (29 June 2010). Frontline. https://www.pbs.org/frontlineworld/stories/southernafrica904/video_index.html.

(6) Futch, M. D., & McIntosh, C. T. (2009). Tracking the introduction of the village phone product in Rwanda. Information Technologies & International Development, 5(3), pp-54.

(7) Ibid.

(8) Nothing But Nets. “Our Solution: Why Nets?” https://nothingbutnets.net/.

(9) Rangan, V. K., & Lee, K. L. (2012). Grameen Danone Foods Ltd.: A Social Business. A Social Business (January 31, 2012). Harvard Business School Marketing Unit Case, (511-025).

(10) Goldman, S. “D. light: Two Million Lives Impacted.” (23 November 2010). Acumen Fund. http://acumen.jeanhules.com/blog/press-releases/d-light-two-million-lives-impacted/.

(11) “Our Impact.” Kickstart: The Tools to End Poverty. http://kickstart.org/.

(12) World Bicycle Relief. “Project Tsunami Report Confirms the Power of Bicycles.” (September 2007) https://b.3cdn.net/bicycle/c3e2aa813961f122fa_jem6i7tyn.pdf. See also, Kirkpatrick, S. J. B. (2018). Pedaling disaster: citizen bicyclists in disaster response—Innovative solution or unnecessary effort?. Natural hazards, 90(1), 365-389.

(13) “Our Impact.” Kickstart: The Tools to End Poverty. http://kickstart.org/.

(14) Ibid.

(15) “Endeavor Global.” https://endeavor.org/.

(16) Kane, M. (2008). Evaluation of the Project to Support PAV (Expanded Programme on Immunization) In Northern Mozambique, 2001–2008: An independent review for VillageReach with programme and policy recommendations (http://villagereach. net/vrsite/wp-content/uploads/2009/08/Evaluation-ExecSum-only. pdf). DTP3 coverage in Cabo Delgado province. See also, https://www.villagereach.org/

(17) Ibid.

(18) "One Acre Fund." https://oneacrefund.org/.

(19) Haynes, A., Regenbogen, S., Weiser, T., et. al. “Surgical outcome measurement for a global patient population: Validation of the Surgical Apgar Score in 8 countries.” Surgery. (2011).

(20) Ahmed, F. E. (2008). "Microcredit, men, and masculinity." NWSA Journal, 20(2), 122-155.