Module 11: Health Indicators

Indicators are used in public health programming and evaluation to numerically portray the health status of a population. Traditionally, this was done via mortality rates, but over time, it was found that non-biological determinants (such as access to care, quality of care, and living conditions) were also pertinent to a more holistic picture of a community’s health.(1) Indicators range in complexity from basic counts (e.g. number of new cases of HIV in a specified area, number of existing tuberculosis cases) or complex proportions (e.g. HIV prevalence per 100,000, neonatal mortality rates).(2) Some indicators may also act as proxies, if the actual measurements are difficult to define or collect. For example, many organizations measure the percentage of 1-year olds who have received the third dose of Diphtheria Tetanus Pertussis as a proxy to assess the provision of and access to general health services in a particular area.(3) Evaluation indicators must be tailored to the characteristics, content, and subject of the program. Thus, there is no set list of indicators that should be included in a monitoring and evaluation plan. However, the criteria compiled below provide guidance for developing or reviewing health indicators.

Case Study: Expanded Program on Immunization

In response to globally-low immunization rates among infants, the World Health Organization (WHO) developed a global program for immunization called the Expanded Program on Immunization (EPI). This program works to ensure that infants are fully immunized with all recommended vaccines (polio, DTP, TB, measles, hepatitis B, haemophilus influenza type B, among others).(4) Although there are a number of vaccines in the EPI package, WHO evaluates the process of distribution based on what percentage of children, nationally, have received the third (and final) dose of DTP. The goal of this WHO program is to achieve of rate of 90% coverage among infants for the last dose of DTP in each country.(5) Other coverage rates are reported but the DTP indicator is used as an ‘at-a-glance’ projection of the EPI coverage success, especially useful for funders who are determining the success of an EPI program.(6) The third DTP dose acts as a proxy because, if the child has received three doses of DTP, this implies three, separate and individual contacts with the health system in their community which not only implies that the child has received all other immunizations (DTP is one of the only infant immunizations with three doses), but also indicates that the health system is strong enough to allow for extended access to these services.(7)

(1) Specific. The goal of the indicator must be specific and clearly stated; a goal or direction should be explicit to the reader.(8) This includes citing who is being measured, as well as what, where, and when they will be measured (i.e. the level of disaggregation).(9) An indicator lacking in specificity will make it very difficult for the evaluators to decide how and from whom to collect the data.

(2) Measurable. The indicator must be quantifiable; otherwise the indicator has little meaning. If the indicator is not measurable, there is no way of determining the success of the program in that specific domain.(10) An indicator that is not measurable may contain words or phrases that are undefined, such as "access" or "quality".(11) These words are subjective and may be defined differently across regions or professions. This highlights the necessity of specific and extensively defined words in all indicators to ensure their measurability.

(3) Attainable. With limited resources for monitoring and evaluation activities in global health, it is necessary to choose indicators that are attainable in terms of the time, budget, personnel, and data collection constraints placed on the evaluation team.(12)(13) The program development and evaluation teams should be acutely aware of these constraints when choosing program indicators.

(4) Relevant. Indicators should be precise in measuring exactly what the evaluation is commissioned to determine (i.e. programmatic success). Although there will be various indicators that relate to the concept of the indicator, it is important for the purposes of constructing or choosing indicators that evaluators are only collecting data on essential information. For example, if the program seeks to reduce malnutrition rates among children under five, the indicator would be to measure the percentage of children under five in a specified area who are malnourished. Although nutritional quality of meals eaten and access to food sources may also be relevant to this topic and program, they are not specific to the program’s goal of reducing malnutrition rates.(14)

(5) Time-bound. Development programs, especially those focused on health, are dependent upon time-bound measurements, as disease rates can change over time. Thus, it is important to create time sensitive indicators that reflect this characteristic. There should not only be an expected date of achievement for each indicator (e.g. taking into consideration the timeline of the project and expected dates of report dissemination) but the indicator timing should also reflect an understanding of what is being measured.(15)(16)

Qualitative Indicators

Although indicators are more often quantitative, numerical representations of a population’s health, there are several health-related qualitative indicators that may be used. The main issue with qualitative indicators is that their meaning is often subjective, and can be reported in different ways; in contrast, quantitative indicators leave minimal room for debate once they have been compiled.(17) Oftentimes qualitative indicators provide supplemental information in addition to a more traditional set of quantitative indicators, because qualitative indicators allow for the collection of richer, more informative data. The risk, however, is that this data cannot be generalized to the wider population because the information is usually context-bound.(18) Examples of qualitative indicators include an individual’s perceived risk of contracting an infectious disease (ranked on a 1-10 or low-high scale), or a stakeholder’s perceptions of solidarity with project objectives.(19) More often than not, these indicators will be perceptions or beliefs of the subjects, as opposed to concrete values like quantitative indicators. By using qualitative indicators to supplement quantitative indicators, evaluators can gain a better sense of the target population’s experience with and views on the project.(20)

Footnotes

(1) Pascal, J., and Lombrail, P. (2001). Mesure de l’etat de santé de la population. La Revue du Prat. 51:1931-1938

(2) Pan American Health Organization. (2001). Health indicators: Building blocks for health situation analysis. Epidemiological Bulletin. 22(4).

(3) Becker, L., Wolf, J. and Levine, R. (2006). Measuring commitment to health: Global health indicators working group consultation report. Washington, DC: Center for Global Development.

(4) World Health Organization (WHO). (2011). The Expanded Programme on Immunization: Benefits of immunization. Geneva, Switzerland: WHO.

(5) World Health Organization. (2011). Global routine vaccination coverage, 2010. Morbidity and Mortality Weekly Report. 60(44):1520-1522.

(6) Burton, A., Monasch, R., Lautenbach, B., Gacic-Dobo, M., Neill, M., Karimov, R., Wolfson, L., Jones, G., and Birmingham, M. (2009). WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bulletin of the World Health Organization. 87:535-541.

(7) Pavignani, E. and Colombo, S. (2009). Analysing disrupted health sectors: A modular manual. Geneva, Switzerland: World Health Organization.

(8) United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. New York, NY: United Nations Development Programme.

(9) United Nations Development Programme. (n.d.). RBM in UNDP: Selecting indicators. New York, NY: United Nations Development Programme.

(10) United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. New York, NY: United Nations Development Programme.

(11) United Nations World Food Programme. (n.d.) Identifying M&E indicators. Rome, Italy: UNWFP Office of Evaluation and Monitoring.

(12) United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. New York, NY: United Nations Development Programme.

(13) United Nations World Food Programme. (n.d.) Identifying M&E indicators. Rome, Italy: UNWFP Office of Evaluation and Monitoring.

(14) Ibid.

(15) United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. New York, NY: United Nations Development Programme.

(16) Centers for Disease Control and Prevention. (n.d.). Evaluation guide: Writing SMART objectives. Washington, DC: Department of Health and Human Services.

(17) United States Agency for International Development. (1996). Performance monitoring and evaluation. Washington, DC: USAID Center for Development Information and Evaluation.

(18) Kapoor, I. (1996). Indicators for programming in human rights and democratic development: A preliminary study. Gatineau, Quebec: Canadian International Development Agency.

(19) World Health Organization. (2001). Malaria early warning systems- Concepts, indicators and partners- A framework for field research in Africa. Geneva, Switzerland: World Health Organization.

(20) Canadian International Development Agency (CIDA). (1997). Guide to gender-sensitive indicators. Quebec: Minister of Public Works and Government Services.