Module 2: Evaluation Approaches: Case Studies

How do different organizations evaluate their programs? What indicators do they use?

A growing number of researchers, investors, practitioners, entrepreneurs and donors have contributed to an impressive body of knowledge on how to create and measure impact. Some of these studies, approaches and tools have been conducted in collaboration across the public and private sectors. Others have been devised and developed within one organization. Their focus ranges from identifying the most effective business models and understanding the best conditions under which to launch those businesses, to surveys of beneficiaries and attempts to quantify the social return. No single model can capture everything.  These approaches, however, are all directly or indirectly connected.

The following excerpts represent several approaches in the field of metrics and impact measurement. 

Mulago Foundation(1)

The Mulago Foundation’s approach provides five straightforward steps that are essential to achieving measurable results.

1. “Figure out what you’re trying to accomplish”: Since wordy mission statements can be unhelpful, the Mulago Foundation suggests articulating the ultimate goal in eight words or less. Examples of these concise statements include “getting African one-acre farmers out of poverty” or “preventing HIV infection in Brazil.”(2)

2. "Pick the right indicator”: It is crucial to determine what factor will best demonstrate whether or not the goal has been achieved. Multiple indicators are sometimes necessary to conclude that effective changes have been made, and this combination must be selected carefully.

3. “Get real numbers”: In order to declare success, a tangible change must have been made. This includes an accurate measurement of baseline rates and a re-measuring of the rates following the intervention. It is also important to ensure that the correct sample size was used. It is also crucial to measure the impact at practical intervals in order to confirm that impacts are long lasting and sustained.

4. “Make the case for attribution”: You must prove that the intervention efforts directly caused the observed change. During this step, it is helpful to consider other outside influences and alternative explanations for the measured impact.

5. “Calculate bang-for-the-buck”: The Mulago Foundation recommends dividing total donor money spent by the total impact as a simple, effective way of calculating bang-for-the-buck. In measuring impact, it is also key to maintain “an honest, curious, and constructive skepticism.”(3)

Acumen Fund(4)

The Acumen Fund follows metrics at the individual investment level based on four main principles:: “financial sustainability, social impact, scale, and cost effectiveness.”(5) The Acumen Fund strives to calculate the social impact of its funds by comparing it to the other current charitable options for that specific social issue.  This method assists contributors in determining where their investments will be most effective on a long term scale, and provides an illustration of how much social output will be produced in comparison to the “best available charitable option.”  This approach, called the BACO ratio (for “best available charitable option”), serves as the Acumen Fund’s method for evaluating the social impact and cost‐effectiveness of their investments.(6)

The analysis of the BACO ratio targets the least common denominator of output provided – for example, people with markedly improved housing, or individuals receiving ARV treatment.  In summary, the BACO ratio expresses the net cost per unit of social impact.  While the BACO is largely successful and beneficial in many ways, several challenges still persist: the BACO does not reveal the long-term impact of Acumen’s work, requires a comparable charitable option, and often struggles with the difficulty of equating “apples to oranges” when it comes to determining which option is more cost-effective at improving poverty on a broader, more general scale.  Acumen is continually improving this methodology, and BACO serves as an extremely useful tool in understanding the larger context of Acumen’s funds.(7)

USAID –Poverty Assessment Tools(8)

Microenterprise development programs have had marked success in bettering the lives of the poor. In 2000, U.S. Congress authorized the Microenterprise for Self-Reliance and International Anti-Corruption Act, providing at least 50% of all USAID Microenterprise funds to assist the very poor.  In order to ensure that USAID meets these criteria, low-cost tools for evaluating the poverty levels of microenterprise clients have been developed.  Each USAID-developed Poverty Assessment Tool includes a short, country-specific survey and a data entry template.  Microenterprise implementing partners must utilize these tools to measure and report the results, therefore gaining an accurate estimate of the number of clients who are very poor.

In each of the 29 countries for which tools have been developed and certified, the USAID implementing partner must report their results using the Microenterprise Results Reporting system (MRR).  Each survey tool includes a data entry template to be used to process the data collected from client interviews, and it is then processed using the statistical program Epi Info.(9)

KickStart

KickStart uses a systematic approach to "lift millions of people in Africa out of poverty, quickly, cost-effectively and sustainably." KickStart designs and produces low cost, high quality irrigation tools that meet the needs of Africa’s poorest farmers. KickStart sells irrigation systems through a private sector supply chain, which ensures the sustainability of their approach. The profits made from sales are reinvested within the organization in order to develop new technologies.(10)

When it comes to evaluating their own success, KickStart asks three crucial questions:

• Do the people whom we’ve helped out of poverty, stay out of poverty?

• Can more people avail themselves of the solution, without additional investment from KickStart?

• Is KickStart becoming more self-sufficient as an organization?(11)

Because KickStart is constructing a money-making, private-sector supply chain, they have replacement parts available to keep all pumps functioning smoothly.  Entrepreneurs continue to develop their businesses, and because pumps are disseminated locally, new pumps can be purchased at any time.  Once a tipping point is reached in any given area, KickStart will gain a profit from each sale. KickStart predicts that the tipping point will take between 12 and 14 years.(12)

KickStart’s choice to sell their tools instead of merely giving the pumps away cost-free is a unique and interesting approach to fighting poverty.  While at first glance it may seem that giving pumps for free is the less expensive and more effective method, KickStart’s strategy is actually the most realistic when it comes to helping individuals generate their own small businesses.  Because these entrepreneurs are making investments, they are psychologically more committed to using these pumps effectively. In fact, 80% of pumps bought were used to generate jobs and income, whereas less than 30% of free pumps were used for this purpose.  In summary, selling pumps is beneficial for several key reasons: investors are more committed, a lasting supply chain is created, and spare parts are made cheaply available.

It is imperative that KickStart does not evaluate success based on the number of pumps sold, but rather on the number of people helped out of poverty.  Therefore, KickStart created an organized method to track its impacts.   Each buyer fills out a one-year guarantee form upon purchasing the product, which is added to a database of all pump owners. Kickstart extracts a statistically valid sample of recent consumers from this database.  These individuals are visited three times: 1 month after the initialpurchase, at 18 months, and at 3 years after the purchase.  Not only do the conversations establish baseline levels of family income, health, and education, but a relationship is also fostered.  The first visit is used as a point of comparison for the subsequent two visits in order to observe impact in a real way.(13)

Unite For Sight: Surgical Delivery Impact(14)

Unite For Sight invests human and financial resources to develop each eye clinic's outreach infrastructure, and the programs are led, managed, and delivered by the local eye clinic's ophthalmic staff.  Unite For Sight tracks the number of patients who receive eye care through the organization’s outreach programs.  The organization evaluates the number of patients living in poverty who are receiving surgery by the local eye clinics, and data is also collected to track the number of patients with significantly improved vision after surgery.  Unite For Sight requires extensive documentation from each eye clinic partner, including every patient's preoperative and postoperative visual acuity data, which is analyzed to ensure the quality, effectiveness, and social impact of the programs. Unite For Sight also assesses the increase in each eye clinic's cataract surgical rate, targeted for patients living in poverty.  Comparison measurements are made for cataract surgical rate before and after the introduction of the Unite For Sight partnership.

Conclusion

In conclusion, different organizations utilize different methods to measure their impacts and the success of their results.  As the Rainer Arnhold Fellows Program specifies, impact is "a change in the state of the world brought about by an intervention. It’s the final result of behaviors (outcomes) that are generated by activities (outputs) that are driven by resources (inputs)."(15) Together, these case studies demonstrate the various approaches to analyzing and evaluating impacts.  The benefits and drawbacks of each method emphasize the need for organizations to work together, continuing to inspire one another and create innovative evaluation approaches.

Footnotes

(1) Starr, K. and Hattendorf, L.  “The Mulago Foundation: How We Think About Impact.” Mulago Foundation. https://mulagofoundation.org/stuff/how-we-think-about-impact.

(2) Ibid.

(3) Ibid.

(4) Plone Foundation et. al. “Acumen Fund Method.” Real World Systems: Logical Outcomes (2010). https://acumen.org/.

(5) Ibid.

(6) Ibid.

(7)  Acumen Fund Metrics Team.  “The Best Available Charitable Option.”  Acumen Fund Concepts (2007). 

(8) IRIS Center, University of Maryland.  “Poverty Assessment Tools: Quantifying the Very Poor.” US AID From the American People (2005 – 2008).

(9) Ibid.

(10) KickStart. http://kickstart.org/how-we-work/#our-model.

(11) Ibid.

(12) KickStart.  http://kickstart.org/.

(13) Ibid.

(14) Unite For Sight. "Impact." https://www.uniteforsight.org/what-we-do/results.

(15) Rainer Arnold Fellows. "Our Approach to Design." (2010) https://mulagofoundation.org/fellows/programs/rainer-fellows-program.