Module 15: Ethics of Evaluation

Ethics, the discipline of rights, morals, and principles that guide behavior, is an especially important topic in the evaluation of health programs, as many of the responsibilities of program evaluators involve interactions with other human beings.(1) The following is a brief discussion of some of the ethical considerations that evaluators face throughout the evaluation process, as well case studies to provide insight into real-world ethical issues in global and public health. More general ethics topics (such as the importance of obtaining informed consent) are also relevant to evaluation activities, and discussions of these topics can be found in Module 3, Module 4, and Module 8 in the Global Health Research course. It is important to keep in mind that evaluation and research often overlap; as mentioned previously, evaluation can be considered a subset of global health research

Using a Control Group

Including a control group in a study design allows evaluators to determine program effects, as a control group acts as a counterfactual – what would have happened if the program had not been implemented. The differences between the control and intervention group indicators are used to identify the program effects. For more information on the concept and necessity of control groups, refer to the module on study design. However, an especially relevant question in global health programs centers on whether it is ethical to knowingly deny a group of individuals a particular intervention if previous research has shown that the intervention is at least partially effective.(2) Fortunately, evaluators and program implementers can employ certain strategies that allow for the use of a control group while upholding these ethical considerations. One option is a “phase-in” approach, in which the program is expected to be rolled out in a number of different areas over time. This allows for areas that are not initially chosen for implementation to be used as the control group, under the expectation that they will receive the intervention in the future. This strategy was used in the implementation and evaluation of Progressa, an education and health-based conditional cash transfer program in Mexico. Slowly phasing the program into regions of the country ultimately enabled program implementers to manage the large-scale program effectively, and allowed evaluators to use those regions that had not yet been chosen for the intervention as the control groups.(3) A phase-in approach eases ethical concerns because eventually everyone will receive the intervention and, more importantly, those who receive the intervention at a later date may receive an updated, improved version of the intervention due to changes made based on data collected in previous phases.

Case Study: Rural Immunization Impact Evaluation in India

Researchers Banerjee, Duflo, Glennerster, and Kothari were criticized by their peers when they published an article detailing the results of an impact evaluation from their work in rural India. The evaluation compared intervention and control groups, where one intervention group was given a “low-power” incentive (a portion of raw lentils and a plate for immunization completion) to attend the immunization clinics which the program held in each of the villages one time each month, another was given access to the same clinics but no incentive, and the control group was not given the incentive and control villages did not hold the same monthly immunization clinics.(4) The primary investigators found that access to the clinic (difference between control and intervention groups) made a small effect on attendance at the monthly immunization clinic. However, the difference between the two intervention groups (one with incentive, one without) was significant, with those who were incentivized attending the clinics more often than those who had access but no incentive. Researchers who were not involved in the original evaluation criticized the project’s ethics, explaining that because there is general knowledge of the benefits of both immunization programs and incentivization of public health care, there was no ethical justification for having a control and secondary intervention group.(5) This ethical issue falls under the principle of “essentiality” meaning that the research is considered essential and cannot be conducted using alternative methodology. The rebuttal argument, provided by a third party source, cited (as justification) the paucity of knowledge about how incentivizing health care is applied in rural India, as compared to more researched, urban areas.(6)

Working with Vulnerable Populations

Vulnerable populations, which include women, children, prisoners, people living with HIV/AIDS, economically or educationally disadvantaged individuals, and mentally disabled individuals, are often the subject of health interventions. In the past, these groups have been the targets of various unethical procedures and experiments, which is one reason for heightened vigilance when dealing with these populations.(7) These vulnerable groups have been historically underserved in terms of health care, and may be difficult to evaluate for follow-up purposes, for reasons including stereotyping, health worker bias, socioeconomic factors, or intimidation in health care settings.(8) Public policies (changes in social services, health care costs, infrastructure changes etc) also have significant effects on these populations, which is important for evaluators to keep in mind when giving recommendations to clients based on evaluation results.(9) Depending on the intervention and the goals of the evaluation, there may be certain restrictions or even quotas of minority or vulnerable populations that must be met; evaluators should be aware of these requirements when designing the evaluation plan.

Case Study: Evaluating the Effectiveness of Penicillin on Institutionalized Patients(10)

The Tuskegee Syphilis Study is an infamous example of a vulnerable population that was unethically treated—or rather untreated, as hundreds of African-American men in Alabama infected with syphilis were observed by doctors and researchers and never told of their disease, or given treatment.(11) In a similar case, though less well known during the 1940s, American doctors had been conducting an evaluation of Penicillin, using prison inmates and institutionalized patients, as participants. These two groups of individuals, as discussed above, are considered highly vulnerable groups, and additional procedures are now in place to ensure that these groups are not taken advantage of in medical or behavioral experiments. Doctors evaluated the effectiveness of the antibiotic by purposely infecting men with syphilis (unlike Tuskegree, in which the researchers studied the natural progression of the disease in men who had already been infected). Although there was no control group used (all men who contracted the disease received the treatment), the entire experiment was unethical. First, the experiment does not follow a number of the rules laid out in the Belmont Report - a report which includes the principles and guidelines for conducting research on human subjects - regarding ways of reducing harm to the participant. Further, it was not necessary to carry out the experiment in the particular manner used – there are a number of alternatives for testing the efficacy of a drug; for example, seeking out a sample of individuals who are already infected with a bacterial disease to test an antibiotic instead of purposely infecting the sample. Furthermore, without a control group, evaluators could not determine the true efficacy of the drug. Lastly, the groups chosen for “participation” did not freely provide informed consent and were not empowered to voice opposition. This example of evaluation highlights the importance of ethical codes for protecting vulnerable populations.

When conducting any kind of evaluation research, whether domestically or internationally, it is important to know the policies and regulations that ensure that the proposal is in line with existing ethical codes and standards. Developing a highly ethical but flexible evaluation plan is of the utmost importance.

Footnotes

(1) Issel, L.M. (2009). Health program planning and evaluation: A practical, systematic approach for community health (2nd ed.)  Sudbury, MA: Jones and Bartlett Publishers.

(2) Penslar, R.L. (1993). Institutional review board guidebook. Indiana, IN: The Poynter Center for the Study of Ethics and American Institutions.

(3) Asian Development Bank. (2006). Impact evaluation: Methodological and operational issues. Manila, Philippines: Asian Development Bank. https://www.adb.org/sites/default/files/institutional-document/33014/files/impact-analysis-handbook.pdf.

(4) Banerjee, A.V., Duflo, E., Glennerster, R., and Kothari, D. (2010). Improving immunization coverage in rural India: Clustered randomized controlled evaluation of immunization campaigns with and without incentives. BMJ. 340(c2220).

(5) Deepak, M.G. (2010). Missed vaccination- Violation of ethics? BMJ. Response to Banerjee et. al, (2010).

(6) Friedman, J. (2011). The ethics of a control group in randomized impact evaluations- the start of an ongoing discussion. Development Impact.

(7) Schuklenk, U. (2000). Protecting the vulnerable: Testing times for clinical research ethics. Social Science and Medicine. 51(6):969-977.

(8) Steinke, E.E. (2004). Research ethics, informed consent, and participant recruitment. Clinical Nurse Specialist. 18(2):88-97.

(9) Bindman, A.B., Grumback, K., Keane, D., and Lurie, N. (1993). Collecting data to evaluate the effect of health policies on vulnerable groups. Fam Med. 25(2):114-9.

(10) McNeil, D.G. (2010). U.S. apologizes for syphilis tests in Guatemala. New York, NY: The New York Times, A1.

(11) Centers for Disease Control and Prevention (2016). U.S. Public Health Service Syphilis Study at Tuskegee. https://www.cdc.gov/tuskegee/index.html.