Module 1: Metrics, Monitoring, and Evaluation

Outcomes and Outputs

Traditionally, metrics in global health and development work have tended to look at outputs (number of people trained), but are now increasingly looking at outcomes (changes in behavior as a result of the training). The term ‘impact’ refers to the results that donors ultimately seek as a consequence of the outcomes (jobs created, people lifted out of poverty).  While needed the most, outcomes and impact are difficult metrics to measure and report in meaningful ways.

There is an important difference between outputs and outcomes, and it is crucial that global health interventions are evaluated based on their outcomes.  For example, an output would be the number of individuals enrolled in a job training program.  While sometimes this fact can be a proxy for an outcome or impact, it is not necessarily synonymous with the impact itself.  In this case, the number of people enrolled is not a valuable measurement if none of those enrolled in the training program end up obtaining a job. 

Therefore, in this scenario, the more important metric is the outcome regarding how many people enrolled in the job training program now have a job. It is critical to articulate the difference between outcomes and outputs when designing a measurement system and reporting results, lest activities done with the intent of creating results be confused with the results themselves.

Causality

In addition to outcomes and outputs, causality is an important concept in monitoring and evaluating global health work. For example, in the job training case, how do we know that participants have a job specifically due to the training program and not some outside factor?  It is often not sufficient to report the output and outcome; organizations must show that the output is the cause of the resulting impact.

There is an expectation that inputs such as funding and human resources will ultimately lead to developmental impacts. Therefore, in order to measure impact, one must create a counterfactual to determine what would have happened without the intervention. Often when organizations report outcomes, they imply that ‘without our intervention, this would not have happened’.  Evaluation models should always be clear about causality: what is expected to happen as a direct result of the intervention. In practice, causality is difficult to assess due to a variety of reasons outlined below:(1)

• One has to demonstrate that the measured impacts resulted from the intervention, rather than from other interventions (perhaps by another agency), or from something completely different.

• One also has to demonstrate that those who did not benefit directly from the intervention did not suffer at the expense of those who did benefit; for example, did management training enable some entrepreneurs to do better, penalizing those who did not receive the training?

• A skeptic might wonder if the observed impacts would not have happened anyway, just as a continuation of prior trends.

Monitoring vs. Evaluation

Monitoring and evaluation of global health and development activities provides NGOs, governments, and donors with a means to learn from past experience, improve interventions, allocate resources, and demonstrate results.  Within the global health and development community there has been a strong push to focus on results, yet there is often confusion about how to measure the impact of programs and interventions.  The tools of monitoring and evaluation offer a solution and include the following: (2)

• Performance indicators

• Formal surveys

• Rapid appraisal methods

• Participatory methods

• Public expenditure tracking surveys

• Cost-benefit and cost-effectiveness analysis

• Impact evaluation

Often used synonymously, monitoring and evaluation in fact refer to two different activities.  Monitoring involves on-going measurement of performance, examining parameters such as cost efficacy, and assessing whether things are going well.  Ideally, these measurements are conducted and reviewed locally by the intervention team and can lead to improvements throughout the implementation stage.  Monitoring can be formally defined as "a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds”.(3)Therefore, monitoring involves the regular tracking of resources, activities, and outputs at the project, regional, national, or global level. This includes the monitoring of a country’s progress against the sustainable development goals, or other standard measures of success.

Evaluation, on the other hand, is often conducted by external advisers, and is focused on proving impacts rather than improving interventions.  Evaluation seeks to answer questions about whether the right things are being done, and is a measure of the performance of the program design, of the implementing team, and of the implementing agency. Evaluation can be defined as “the process of determining the worth or significance of a development activity, policy or program ….. to determine the relevance of objectives, the efficacy of design and implementation, the efficiency or resource use, and the sustainability of results.  An evaluation should (enable) the incorporation of lessons learned into the decision-making process of both partner and donor”.(4) Although different, monitoring and evaluation are complementary. Monitoring information is a necessary part of identifying best practices, yet does not provide a sufficient nor holistic assessment. Monitoring information is used for ongoing management purposes, but solely relying on monitoring data can be misleading because it often covers only specific dimensions of a project’s or program’s activities.(5)

In order to obtain the full picture, evaluation is needed to provide a more balanced interpretation of an intervention’s impact. However, evaluation is a more detailed and time-consuming activity and is thus conducted less frequently. Organizations therefore often rely on monitoring information to identify potential problems that require more detailed investigation via evaluation.

Footnotes

(1) Gallagher, S. (2019). Banking Performance and Socio Economic Development. Scientific e-Resources.

(2) World Bank. (2004). "Monitoring and Evaluation: Some Tools, Methods & Approaches." http://documents1.worldbank.org/curated/en/829171468180901329/text/246140UPDATED01s1methods1approaches.txt.

(3) Organization for Economic Co-Operation and Development: Development Cooperation Directorate (DCD-DAC). (2002). "Glossary of Key Terms in Evaluation and Results Based Management." https://www.oecd.org/dac/evaluation/2754804.pdf.

(4) Ibid.

(5) Gallagher, S. (2019). Banking Performance and Socio Economic Development. Scientific e-Resources.