Module 4: Constraints on Evaluations

In the process of conducting an evaluation, evaluators should be conscious of the constraints that exist in almost any health environment. Constraints can be separated into four categories: time, budget, data availability, and political/contextual factors.

Frequently, evaluators are not integrated into the program design process from the start, but are instead consulted once the project is already in its advanced stages. This cuts down on the time for the evaluation, which increases the pressure on evaluators, who now must balance the timeliness and quality of the data collection process.  Thus, bringing in evaluators during the late stages of the project implementation (while maintaining the original program schedule) reduces the quality of the evaluation process and the data collected.(1) Budget constraints also have similar effects on the evaluation. Budget constraints often become an issue when evaluation is not introduced until most of the funding has been allocated for other purposes; this is especially true with impact evaluations that program managers might not consider until the project is completed.(2) It may also be the case that even when evaluation is considered from the initial planning phases, there is insufficient funding to match the type of evaluation needed to collect quality data.(3)

Data availability is also related to the evaluation timeline. Often, when evaluators are not immediately integrated into program planning, the data required to determine a baseline for key indicators is not collected.  Depending on its study design, a non-integrated evaluation may not be able to select a proper control or comparison group for data collection. This obstacle requires the evaluation team to choose a less robust study design, or collect baseline and comparison data through less reliable and compatible secondary sources.(4)(5) Data availability may also be limited on culturally sensitive topics such as HIV risk behaviors, sexual activity, and drug use. Whether the topic is culturally taboo or the individual is concerned about confidentiality, the evaluator and data collector must be aware of the context of each community and determine the best way to collect the data.(6)

Last, politics and local contextual factors may create barriers to quality evaluations. This can refer to the politics of both the country and the community in which the evaluation is taking place, as well as the aims and perspectives of the client and stakeholder groups. The politics of these groups affect the evaluation at all stages but become most relevant in the reporting stage, when evaluators disseminate the findings and conclusions back to clients and stakeholders. Clients may oppose negative program feedback from evaluators, or ask that only positive findings be published.(7) This is called selective publishing, where researchers or clinicians do not publish unfavorable outcomes, especially when the research or evaluation is sponsored.(8) Although evaluators may feel pressured to report positive findings to the client or program manager, the evaluation team must also consider the need to be objective and ethical, as well as the interests of the community they are ultimately serving.(9)

How to Overcome Constraints

It is possible to overcome time constraints by modifying the evaluation plan. Although this situation is not ideal, the duration of an evaluation can be cut by choosing a study design that does not include pre-tests or comparison groups - essentially removing variables for evaluation. Of course, both of these options will make the evaluation less robust, but may be necessary to complete the evaluation on time, and produce useful results. This strategy greatly reduces both time and effort that go into the data collection process.(10) Other options that do not necessarily require altering the study design include clarifying the client needs so the team is only collecting the most essential data, using existing data or documents so that only analysis needs to be conducted, and reducing the sample size to limit the number of surveys or interviews that need to be conducted.(11) All of these strategies will maximize available time, though it should be noted that each uniquely diminishes the standards of the methodology.

Budget constraints can be overcome using similar strategies to those described in relation to time constraints. Ensuring that all proposed data collection is essential, reducing sample size or level of personnel time needed, and incorporating various data collection techniques (e.g. self-administered questionnaires) to reduce costs of data inputs are all strategies used to stretch a minimal evaluation budget.(12)

Various organizations have developed ways of overcoming the data accessibility issues surrounding a number of global health topics. RTI International has conducted numerous surveys on sensitive global health topics. As the organization continues to improve existing methodology to collect the most accurate and highest quality data, RTI International has developed a list of tools to overcome the confidentiality and sensitivity barriers that exist for many pertinent health issues. Self-administered questionnaires, computer-assisted interviewing tools, and neutral-site interviews are all methods that RTI International uses to allow for increased privacy and confidentiality among program participants.(13)

In terms of overcoming the constraints posed by political influence, the first step is accepting that these influences will always exist, even when the stakeholders and clients have discussed agendas and priorities. Therefore, it is not possible to conduct an evaluation completely free of political influence, and it is necessary to build a foundation to manage political agendas in any situation. Ensuring that all details of the evaluation process (such as deadlines, specific deliverables, and general expectations) are clear between the client and the evaluation team is critical. It is also helpful to get a sense for the potential reactions to negative evaluation findings, and to use non-inflammatory wording in final reports. Overall, the goal is to preemptively manage the various political influences.(14)

Footnotes

(1) United States General Accounting Office, Program Evaluation and Methodology Division. (1991). Considering the evaluation’s constraints. Designing Evaluations, 19-24.

(2) Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation. 25(5).

(3) Bamberger, M., Rugh, J., & Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(4) Ibid

(5) The World Bank. (2006). Conducting quality impact evaluations under budget, time and data constraints. Washington, DC: Independent Evaluation Group, Knowledge Programs and Evaluation Capacity Development (IEGKE). http://documents.worldbank.org/curated/en/255551468178129539/pdf/382670ENGLISH01evaluations01PUBLIC1.pdf.

(6) RTI International. Survey Methodologies. https://www.rti.org/service-capability/survey-methodologies.

(7) Bamberger, M., Rugh, J., & Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(8) Mackey, T. & Liang, B.A. (2011). Off-label promotion reform: A legislative proposal addressing vulnerable patient drug access and limiting inappropriate pharmaceutical marketing. University of Michigan Journal of Law Reform, 45(1).

(9) Markiewicz, A. (2008). The political context of evaluation: What does this mean for independence and objectivity? Evaluation Journal of Australasia. 8(2):35-41.

(10) Bamberger, M., Rugh, J., & Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

(11) Ibid.

(12) Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation. 25(5).

(13) RTI International. (n.d.). Survey Methodologies. https://www.rti.org/service-capability/survey-methodologies.

(14) Russ-Eft, D.F. & Preskill, H.S. (2001). Evaluation in organization: A systematic approach to enhancing learning, performance, and change. Cambridge, MA: Perseus Publishing.