Course 5: Global Health Metrics

Unprecedented amounts of resources are invested in public health and healthcare worldwide.(1) Data from 2000 to 2004 found a continuation of this trend, with donor funding for global health approaching $14 billion in 2004.(2) Because of the vast amounts of resources going into global health, funders are beginning to demand performance data in return for their financial assistance. “This global commitment is now coming under intense scrutiny.”(3) Donors want to know results regarding health and health-system performance in the countries in which they are investing, and they want to set goals and standards for those countries to meet. Thus, it is in the mutual interest of the Global South and the Global North to invest in strengthening systems for the collection of health information.(4)

“Sound information on financial and human resources invested in health, health interventions delivered to people in need, and the impact of these efforts on people's health is critical for planning health systems, implementing programs, epidemic response, allocating budgets for research and development, monitoring progress, and evaluating what works and what does not.”(5)

The availability of health information to inform decisions can ensure funding and further the efficacy of global health interventions. This article defines and explores the role of indicators and metrics as an integral part of results-based accountability in global health. Output and outcome indicators enable decision-makers and donors to assess progress toward intended goals and best practices.

Outputs versus Outcomes

In performance assessment, outputs are defined as the goods or services produced by programs or agencies and outcomes are defined as the impacts on social, economic, or other indicators arising from the delivery of outputs.(6) Typically, outputs are used to document the amount, quality, or volume of use of the project's products or services.

While outputs are important to track, evaluation needs to focus on measuring outcomes that reveal the extent and kinds of impact the project has on its participants. Impact could be reported in the amount of change in behavior, attitude, skills, knowledge, or condition of the target population. For example, an output would be the number of microfinance loans distributed or the number of loans repaid, whereas the outcome would be the number of microfinance participants who have significantly increased their income or risen out of poverty due to the loans. Similarly, job training programs have an output of the number of people enrolled in the program. The outcome would be the number of people who were able to get a job due to the training program.

Outcomes and Outputs: Mosquito Net Case Study

The dichotomy between outcomes and outputs can be seen in the case of mosquito net distribution. A Kenyan newspaper, The Daily Nation, reported that many people are using mosquito nets for alternate activities such as making wedding dresses and fishing, especially in the Nyanza Province. In response, mosquito net manufactures are collaborating with the local government to prosecute people who use the products for purposes other than covering beds. Dr. Juma, head of malaria control under the Ministry of Public Health and Sanitation remarks:

“This is wrong and totally unacceptable. Bed-nets are supposed to play a noble role of preventing deaths that would be caused by malaria. I think there is need for further sensitization and education for the entire public to understand the importance of sleeping under insecticide treated mosquito nets.”(7)

In this case, the output was the number of mosquito nets distributed to villagers. If evaluation metrics merely looked at this number, the program may have been seen as a success. However, the real “success” metric is the outcome: whether people used the net in the proper way and whether this led to decreased rates of malaria. In understanding the difference between outputs and outcomes, it is important to remember that it takes far more than just handing out nets to change behavior. Without metrics, global health programs may fall into the pitfall of spending resources on the quantity of an intervention over its quality and potential impact. Programs that give out mosquito nets, condoms, and other supplies are common. While these interventions are popular because they can “help” a lot of people, there is often a discrepancy between their outputs and outcomes.

Health Information Systems

Health information is the foundation of public health.(8) A basic prerequisite of any health program is its ability to state unequivocally how much mortality or morbidity it will avert and what proportion of the global burden of disease it will address. A health information system is an "integrated effort to collect, process, report and use health information and knowledge to influence policy-making, program action and research."(9) Such systems are biased towards quantitative data and should include information regarding:(10)

• Health determinants (socioeconomic, environmental behavioral and genetic factors) and the contextual and legal environments within which the health system operates;

• Inputs to the health system and related processes including policy and organization, health infrastructure, facilities and equipment, costs, human and financial resources, and health information systems;

• Health outcomes (mortality, morbidity, disability, well-being, disease outbreaks, and health status); and

• Health inequities in determinants, coverage and use of services, and outcomes.

Leaders in Health Metrics

One example of a health information system is the Health Metrics Network (HMN), which began in 2005 as the first global partnership focused on generating health information. HMN, in collaboration with the World Health Organization, engages in a variety of work highlighting how accurate, reliable information leads to effective decision making and better health outcomes.

Another leader in health metrics is the Institute for Health Metrics and Evaluation (IHME), an independent research center at the University of Washington which “rigorously measures the world’s most pressing health issues and provides scientific evaluations of health system and health program performance in order to guide health policy and accelerate global health progress.”(11) The IHME assesses health outcomes, health services, resource inputs, decision analytics, and evaluations. Goals include generating scientifically sound estimates of health outcomes – including mortality, causes of death, and the overall burden of disease. The IHME tracks, measures, and analyzes three components of funding for health – development assistance, government health expenditure, and private health expenditure. This information is used to conduct evaluations of policies and programs to best understand the progress being made in health.

In addition, international institutions such as the World Bank, WHO, and International Monetary Fund provide health and socioeconomic measures. For example, the WHO Statistical Information System is an interactive database which brings together core health statistics for the 193 member states and comprises more than 100 indicators. International financial institutions often provide valuable statistics on life expectancy, population growth, economic activity, and development indicators in low- and middle-income countries. Such statistics are important to track given the well-established link between poverty and poor health. When assessing global health, international organizations must negotiate various indicators of health and decide what is accepted as valid and what is not. For example, how does one establish a “poverty line”? Should this line be based upon an objective dollar per day value or should poverty limits be established relative to surrounding populations?

Conclusion

Ultimately, information on outputs and outcomes is essential for public health action, and it is the foundation for policy making, planning, programming, and accountability. Unfortunately, high-quality health information is not widely available in the Global South, and outcomes are frequently not measured by organizations. Health metrics are necessary and important to ensure accountability and achieve best practice standards.

Footnotes

(1) Murray, C. J., Lopez, A. D., & Wibulpolprasert, S. (2004). Monitoring global health: time for new solutions. Bmj, 329(7474), 1096-1100.

(2) Kates, J., Morrison, J. S., & Lief, E. (2006). Global health funding: a glass half full?. The Lancet, 9531(368), 187-188.

(3) Horton, R. (2005). The Ellison Institute: monitoring health, challenging WHO. The Lancet, 366(9481), 179-181.

(4) Stansfield, S. (2005). “Structuring Information Systems to Improve Health.” Bulletin of the World Health Organization 83 (8): 562.

(5) Ibid.

(6) McAllister, K. (1999). Understanding participation: Monitoring and evaluating process, outputs and outcomes. Rural poverty and environment working paper series; no. 2.

(7) Esipisu, I. "Villagers Using Bed-Nets for Wedding Gowns." Art & Culture. Daily Nation, 22 Apr. 2009. https://www.nation.co.ke/kenya/life-and-style/horizon/villagers-using-bed-nets-for-wedding-gowns--589592.

(8) Evans T, & Stansfield S. (2003). Health information in the new millennium: a gathering storm? Bulletin of the World Health Organization; 81:856.

(9) Lippeveld T. (2001). Routine health information systems: the glue of a unified health system. Keynote address at the Workshop on Issues and Innovation in Routine Health Information in Developing Countries, Potomac, March 14-16. Washington, DC: JSI; 2001.    

(10) Ibid.

(11) "IHME." Institute for Health Metrics and Evaluation, http://www.healthdata.org/.