Module 8: Five-Year Evaluation of the Global Fund

The Global Fund to Fight AIDS, Tuberculosis and Malaria provides health program financing to over 130 countries and as of July 2016 has disbursed US$ 30.7 billion in the fight against these three diseases. Evaluation results released in 2016 showed laudable outputs including 9.2 million people on antiretrovirals, 15.1 million people treated for TB, and 659 million bed nets distributed for malaria prevention.(1) However without an impact evaluation, there is no way to determine whether or not these outputs show improvements in the ultimate goal of the Fund to save lives and reduce disease prevalence. The Global Fund had pre-emptively determined the need for a major evaluation during the planning phase. After the first round of Global Fund grants had been distributed and were beginning to expire, the Fund hired an independent group of evaluators to evaluate the impact of Global Fund programs. This group of evaluators will be referred to as the Technical Evaluation Reference Group or TERG.

The first step was to focus the evaluation and define several evaluation questions to answer; the TERG consulted frequently with Global Fund program developers for technical assistance with program details, as well as with stakeholders to get a sense of the priorities of key personnel. From this process, TERG developed three large-scale evaluation questions for the impact evaluation of the Global Fund. The first two focused on capacity, systems, and partnership strengthening ( 1) Does the Global Fund perform efficiently? 2) How efficient is the Global Fund partnership system?) and the third focused on reduction of the disease burden.(2) More specifically, the third question asked: What was the overall reduction of the burden of AIDS, tuberculosis and malaria and what was the Global Fund’s contribution to that reduction? The Global Fund evaluation is a quintessential example of how focusing an evaluation and determining the scope prior to implementation can be beneficial. One of the main components that stakeholders and evaluators agreed upon is that the scope of the evaluation should focus on contribution, rather than attribution; this stipulation is reflected in the research question. With the understanding that most impacts on health burdens are often a result of joint efforts between various programs and organizations, the evaluation would be better served to focus on contributions the Fund had made, as opposed to attributing improvements in welfare to the Fund’s specific grants. To determine the effect of only Global Fund grants would be useful for specific program efforts but would not relay any information about the successes of the collective impact.

A multi-country design allowed the TERG to determine contributions of the Global Fund across 18 different countries over time; with varying contextual factors in each country, a single country evaluation would not have been sufficient. The Global Fund evaluation is a prime example of how constraints on data collection may alter the course of an evaluation. According to evaluators, the timeline for the evaluation was too short; full scale-up in most Global Fund target countries occurred between 2004 and 2005, while evaluators were asked to begin the assessment in 2007. Ideally, evaluators would have preferred to align many of the evaluation and data collection procedures with existing country processes and begin to institutionalize and build capacity for future assessments, thereby increasing country ownership of the program and the evaluation. However, the limited timeline allotted for the evaluation did not allow this to occur.(3) The short timeline also affected the type of disease outcomes and impacts that the TERG was able to measure. In any health program, there is a lag between program implementation or scale-up and a reduction in disease burden; there is also a secondary time lag between when the reduction occurs and when it can be measured and documented. This gap between scale-up and measurement is further elucidated in other modules in this course. With limited options for determining the impact on each country’s health burden, evaluators used the logic model to show how essential aspects of the program, including funding, access, and coverage, eventually improve health outcomes. The assumption was made, based on theory, that increased funding would lead to increased access and coverage in the target areas, subsequently reducing the HIV, TB and malaria health burden.(4) These assumptions are part of a program theory: a set of theory and research-based expectations that program developers use to determine how program activities will lead to program outcomes and impacts.(5) With limited time for disease impact measurements, the use of program theory and evaluation processes that reduce the health burden will help demonstrate whether the program is on track to reach the expected impacts. According to the evaluation’s inception report, “The basic approach to this evaluation is to study trends in the important indicators related to HIV/AIDS, malaria and tuberculosis and to relate those trends to any changes in the service environment and behaviors and to changes in the financial allocation to the different diseases. This approach reflects the basic philosophy that changes in funding should have an effect on services and that changes in services should have an effect on outcomes.”(6)

To determine disease-specific financing to target countries, evaluators studied funding flows over time into these particular nations. Additionally, health services data was collected on three general topics: quality of services, the cost of delivery, and access among specific populations. For service quality, the pre-existing questionnaires were used to determine appropriate indicators, and collect information on various dimensions of service quality (i.e. number of patient beds, patient flow, facility amenities, and training opportunities for staff). The delivery costs included fees for specific services related to household health care expenditures. Last, access among specific populations was determined by a household survey focused on various demographics, as well as coverage of disease-specific interventions (i.e. bed nets or antiretroviral use).(7)

By synthesizing funding patterns, health burden, and demographic data, the TERG was able to present The Global Fund with a number of recommendations. Again, the emphasis of the evaluation was on collective impact, not attributing reductions in disease burden directly to specific Global Fund grants. For HIV/AIDS, evaluators concluded that increased funding, including Global Fund grants, to target countries improved access to healthcare, and a possible decline of HIV incidence in some countries. The TERG report claims that the inability to state with certainty that incidence had decreased was due to limited data and the curt timeline of the evaluation. Results show limited improvement in TB incidence and TB treatment success rates, with incomplete data on diagnostic quality. Evaluators suggested that the first step toward improving TB outcomes in target countries was to ensure multi-year funding streams, since the Global Fund is the main source of TB-specific grant money in many of these regions. The malaria burden in several of the target countries has decreased, particularly with respect to parasite presence and under-5 malaria-related mortality. However, the TERG reported limited diagnostic and treatment capacity in most of the countries surveyed, suggesting that the Global Fund prioritize malaria-specific grants in the future. Overall, evaluators found less than adequate health information systems, limited availability of diagnostics and treatment for all three diseases, and poor quality of health systems. These issues of health system quality need to be resolved in order for improvements to be made in disease-specific arenas.(8)

The Five Year evaluation of the Global Fund is not a perfect example of an impact evaluation but it illustrates many of the constraints that evaluators face. For example, with an extended timeline, the TERG might have been able to determine the full impact of the funding contributions on disease-specific health burdens. The TERG would have also been able to integrate capacity building into evaluation activities in order create a more sustainable data collection process in the future. Although typical impact evaluations strive to attribute program activities to program impacts, the TERG relied heavily on Global Fund stakeholders to direct the evaluation and avoid any unnecessary data collection and analysis. The focus of the stakeholders and the Global Fund on contribution, as opposed to attribution, is what made this particular evaluation exceptionally unique.  

Footnotes

(1) The Global Fund. (2016). Corporate 2016 Results Report Summary.

(2) The Global Fund to Fight AIDS, Tuberculosis and Malaria. (2006). Framework document on the scale and scope of the five-year evaluation. Fourteenth Board Meeting, Guatemala City, 31 Oct – 3 November.

(3) Technical Evaluation Reference Group (TERG). (2009). The five-year evaluation of the Global Fund to Fight AIDS, Tuberculosis, and Malaria: Synthesis of study areas 1, 2 and 3. Calverton, MD: Macro International Inc.

(4) Final report: Global Fund five-year evaluation: Study area 3 – The impact of collective efforts on the reduction of the disease burden of AIDS, tuberculosis, and malaria. Calverton, MD: Macro International Inc.

(5) Bamberger, M., Rugh, J., and Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(6) Macro International Inc. (2007). Five-year evaluation of the Global Fund- Study area 3: Health impact inception report. Calverton, MD: Macro International Inc.

(7) Ibid.

(8) Technical Evaluation Reference Group (TERG). (2009). Study area 3: Health impact of scaling up against HIV, tuberculosis and malaria: Evaluation of the current situation and trends in 18 countries. Washington, DC: The Global Fund to Fight AIDS, Tuberculosis and Malaria.