Module 9: Measuring Surgical Outcomes

Surgery, like any other international development program, requires careful impact assessment. Monitoring and evaluation has several important benefits, including its ability to assess whether the program is having its desired effect. This information allows surgeons and administrators to identify and improve upon weaknesses in the healthcare delivery system, resulting in better patient care. This assessment also provides effectiveness data to stakeholders. With proper reporting, funders can be confident in the reach of their investment, governments can place their trust in the organization, and patients can rest assured that their doctors are delivering quality care.(1)

While the importance of assessing program effectiveness is clear, the methodology of doing so is far more intricate. Measuring surgical outcomes is especially difficult, “as intervention efficacy is not always immediately apparent, complications are often delayed, and follow-up of patients is limited.”(2) However, with thorough planning and careful execution, surgery providers can achieve high-quality program evaluation, thereby equipping themselves with an indispensable tool for patient care improvement.

Measuring Surgical Quality

Birkmeyer, Dimick and Birkmeyer outline three ways to measure surgical quality:(3)

Structural Measures

Structural measures, or outputs, reflect characteristics of the health care delivery system—namely the hospital’s physical resources and staff organization. Procedure volume is the most common of these variables and is often used as a proxy for surgical quality, due the loose correlation between high procedure volume and improved long-term survival. Other common structural measures include the staff’s level of training, the organization of hospital personnel, and the availability of up-to-date technology and financial resources. The main advantage of using these metrics to assess an organization’s impact is that they can be measured inexpensively and relatively quickly using existing administrative data. The problem with this method is that these variables are “imperfect proxies for outcomes.”(4) Providing patients with surgery, staff and high-tech equipment does not necessarily mean that those outputs will always have the desired impact. As discussed in Outcomes Are Essential in Global Health, “when assessing the impact of an organization, it is critical to keep in mind that distribution does not equal value.”(5)

Process Measures

Process variables indicate the type of treatment that patients receive, and are routinely used in nonsurgical fields. For example, primary care physicians are often scored according to the fraction of their patients that receive appropriate mammograms, retinal exams and vaccinations. This methodology might be applied to surgery by tracking the specific type of procedure used, the kind of anesthesia applied, or the combination of postoperative medications provided. The advantages of this method are that it directly reflects patient care and that it provides a clear link to quality improvement strategies. The downside is that it is not known which processes are most important. Medical care is comprised of hundreds of processes, many of which are often taken for granted, making it difficult to discern which activities are most associated with positive outcomes.

Outcome Measures

Monitoring direct outcomes is by far the most robust method of measuring impact, as it measures not only the hospital’s distribution of care, but also the impact of that care on patients’ quality of life. Surgical institutions often use “operative mortality . . . complication rates, length of stay, readmission rates, patient satisfaction, functional health status, and other measures of health-related quality of life.”(6) Outcome measurement is considered the gold standard, as it assesses the value of the services provided.

The Importance of Measuring Surgical Outcomes

The research studies discussed below illustrate the necessity of measuring not only outputs, but also outcomes.

Case Study: Shunyi County, China

A study of vision-related quality of life in rural China tracked the impact of cataract surgery on patients’ visual acuity. The authors found that of 87 operated patients, 12% had postoperative visual acuity of 6/18 (equivalent to 20/60) or better in both eyes, and 24% had visual acuity worse than 6/60 (20/200, or legally blind). Of 116 operated eyes, 25% had visual acuity of 6/18 or better and 44.8% had worse than 6/60. This case clearly demonstrates the importance of going beyond structural and procedural measures to focus on surgical outcomes. Simply counting the number of surgeries performed—87 patients, 116 eyes—does not accurately portray the impact that this medical care had on patients, as nearly a quarter of the study’s subjects had poor outcomes. Equipped with these outcome measures, the researchers can work to improve patient care and ensure that future services have the desired impact on patients’ quality of life.(7)

Case Study: Kikuyu Eye Unit, Kenya

A year-long observational study of the eye unit of Kikuyu Hospital outside Nairobi, Kenya, measured whether the introduction of prospective outcome monitoring influenced surgical outcomes. The study authors found that the proportion of patients who achieved a positive outcome following cataract surgery increased from 77% in the first quarter, when the monitoring system was first being introduced, to 89% in the fourth quarter, when the system had been in place for nearly a year. While the complication rate remained unchanged, the surgeons’ responses to complications improved. For example, the proportion of patients achieving a good visual outcome after vitreous loss, a surgical complication, increased from 47% to 71%. This study demonstrates that the very process of monitoring outcomes improves these outcomes above any improvements made to healthcare delivery systems following outcome analysis. This “Hawthorne Effect” is likely due to a change in surgeons’ attitudes, leading to meticulous patient care.(8)

Using Outcome Measurement to Improve Institutional Effectiveness

Incorporating routine outcome measurement into an entire organization can seem far more daunting than tracking the outcomes of a small sample of subjects for a research project. Key issues to consider when designing a monitoring and evaluation system include:

• Operational Definitions: What defines a successful surgery? Establishing measurable standards is essential for quantifying outcomes. For emergency surgery, the operation is considered successful simply if the patient lives. However, defining a positive outcome for procedures that treat non-life-threatening conditions requires a bit more nuance. For example, what change in visual acuity score is required for cataract surgery to be considered helpful? What degree of closure is necessary for obstetric fistula repair to be successful?

• Timescale: How long does it take for the true outcome of a surgery to become apparent?(9) Reaching patients for follow-up care is difficult anywhere, but is especially challenging in developing countries, where financial and geographic barriers often limit healthcare access. Devising healthcare systems that facilitate post-operative care—through outreach trips, a network of community health workers, mHealth applications, etc.—is essential for proper outcome measurement.

• Patient Reported Wellbeing and Clinical Analysis: Both patient perceptions of wellbeing and clinical analysis of health should be included in outcome measurement. Monitoring and evaluation systems must be able to analyze both subjective, nuanced patient accounts, and scientific, standardized clinical measurements.(10)

• Data Quality: It is important to standardize the way in which data is initially recorded and subsequently transcribed in order to minimize error. This will likely require extensive training, but is essential to produce valid results.(11)

• Publishing Results: Once data has been analyzed, organizations can use the results to report to patients, staff, donors, colleagues, governmental bodies, and other partners. Given the complexity of the statistics, it is important to plan the best way to present the data to each audience so that the information is both understandable and comprehensive.(12)

The following organizations demonstrate various methods of applying these principles to patient care.

Case Study: Aravind Eye Care System

With the goal of eradicating blindness in India, the Aravind Eye Care System has created a revolutionary model for delivering healthcare to the poor. In addition to the hospitals at the core of the organization, the eye care system consists of a manufacturing center that produces lenses, sutures, and medications at a low cost, as well as a training institute, a research institute, an eye bank, and an outreach program.(13) As of March 2011, the hospital’s physicians had treated over 2.6 million outpatients and performed 300,000 surgeries.(14)

Rather than simply tallying the number of surgeries completed, the hospitals monitor outcomes by keeping careful records of any complications, to ensure that each patient’s vision is truly restored.(15)“Cataract Surgery Performance Evaluation Meetings” are held every week,(16) and each month, the hospital calculates the complication rate for both individual surgeons and the entire facility.(17) They also track how well patients comply with treatment recommendations, and how often patients need re-prescription of glasses and re-surgery.(18) This quality assurance process has resulted in high success rates. A 2001 study of adverse events found that when compared to the Royal College of Ophthalmologists in the U.K, Aravind had lower or comparable rates of every type of complication.(19) By investing in outcome measurement, Aravind ensures that their patients receive the highest quality care.

Case Study: ReSurge International

ReSurge International provides reconstructive surgeries to the world’s poor by both building the capacity of the local hospital infrastructure and by dispatching teams of volunteer physicians to supplement the existing medical personnel. In 2019 alone, the organization treated over 2,300 burn victims, children with cleft lips and cleft palates, and patients with hand deformities in 12 countries, and 86% of the surgeries were performed by developing partners.(20) In doing so, ReSurge empowers these individuals to pursue educational, career, and other opportunities that would be otherwise unattainable.

While ReSurge carefully tracks the number of surgeries that they have sponsored, they also focus on outcomes. ReSurge has ensured that their surgeries have the desired impact on patients by instituting a continuum of care that monitors patients as they recover.(21) Children receiving cleft palate repairs are evaluated by a multi-disciplinary panel consisting of speech, orthodontic, and dentistry professionals.(22) ReSurge typically holds week-long speech camps, in which children and their families are provided with speech therapy, dental hygiene, and psychological counseling.(23) Furthermore, occupational and physical therapists work with patients who have undergone burn and hand surgeries to maximize post-operative range of motion.(24)

Having determined that they have provided 110,000 successful reconstructive surgeries since their inception in 1969, ReSurge estimates that, together, these patients will require fewer sick days from school and work.(25) Recognizing the connection between healthcare and poverty is important, but it is essential that this data be collected on the ground, since theoretical calculations often make assumptions about patients that may not necessarily be accurate. Without gathering information directly from patients, the exact social and economic impact of the organization remains unknown.

Innovations for Improving Surgical Outcomes(26)

As the demand and volume of surgery has increased dramatically in all parts of the world, ensuring the safe delivery of high quality surgical care is a major public health concern.

Surgical Apgar Score

The Surgical Apgar score was developed as a simple, low-cost metric to provide rapid feedback to surgery teams in diverse settings. The Surgical Apgar Score is a 10-point tally based on three parameters: the estimated intraoperative blood loss, the lowest heart rate, and the lowest mean arterial pressure. The score is designed to provide rapid feedback for clinical teams, and is predictive of morbidity and mortality, even after controlling for preoperative patient factors. A study evaluating the Surgical Apgar Score in eight countries found that a higher score was correlated with lower rates of complications. For example, those who scored 0-4 had complication rates of 32.9% and death rates of 7.9%, while those who scored a 10 (the best score) had complication rates of 3% and death rates of 0.5%.(27) This study corroborates the importance of measuring outcomes, and supports the Surgical Apgar Score as an accurate predictive measure for doing so.

WHO Surgical Safety Checklist

As part of the Safe Surgery Saves Lives program, the World Health Organization has developed a 19-item checklist to improve surgical outcomes. The list consists of routine activities that should be part of every operation, but are often forgotten.

An 8-country pilot study of the checklist found that the complication rate decreased from 11% to 7%, and the in-hospital death rate dropped from 1.5% to 0.8%.(28) Such remarkable progress in surgical outcomes not only indicates the success of this particular intervention, but also points to the broader fact that improvement in surgical quality is possible. Surgery can be an effective tool for improving health and wellbeing around the world, but maximizing its benefit requires more rigorous and routine measurement of outcomes.

Footnotes

(1) "Measuring Surgical Outcomes." The Royal College of Surgeons of England. Accessed on 29 June 2012.

(2) Ozgediz, Doruk, Renee Hsia, Thomas Weiser, Richard Gosselin, David Spiegel, Stephen Bickler, Peter Dunbar, and Kelly McQueen. "Population Health Metrics for Surgery: Effective Coverage of Surgical Services in Low-Income and Middle-Income Countries." World Journal of Surgery 33.1 (2009): 1-5.

(3) Birkmeyer, John D., Justin B. Dimick, and Nancy J.O. Birkmeyer. "Measuring the Quality of Surgical Care: Structure, Process, or Outcomes?" Journal of the American College of Surgeons 198.4 (2004): 626-32.

(4) Ibid.

(5)“Outcomes Are Essential in Global Health.”Unite for Sight.

(6) Birkmeyer, John D., Justin B. Dimick, and Nancy J.O. Birkmeyer. "Measuring the Quality of Surgical Care: Structure, Process, or Outcomes?" Journal of the American College of Surgeons 198.4 (2004): 626-32.

(7) Zhao, Jialiang, Ruifang Sui, Lijan Jia, Astrid E. Fletcher, and Leon B. Ellwein. "Visual Acuity and Quality of Life Outcomes in Patients with Cataract in Shunyi County, China.”American Journal of Ophthalmology 126.4 (1998): 515-23.

(8) Yorston, D., S. Gichuhi, M. Wood, and A. Foster. "Does Prospective Monitoring Improve Cataract Surgery Outcomes in Africa?" British Journal of Ophthalmology 86 (2002): 543-47.

(9) "Measuring Surgical Outcomes." The Royal College of Surgeons of England.

(10) Ibid.

(11) Ibid.

(12) Ibid.

(13) Prahalad, C. K. The Fortune at the Bottom of the Pyramid. Upper Saddle River, NJ: Wharton School, 2005. 265-86.

(14) "Hospital Services." Aravind Eye Care System.

(15) Prahalad, C. K. The Fortune at the Bottom of the Pyramid. Upper Saddle River, NJ: Wharton School, 2005. 265-86.

(16) Aravind Eye Care System. “Clinical Quality Assessment at Aravind Eye Hospitals.”

(17) Prahalad, C. K. The Fortune at the Bottom of the Pyramid. Upper Saddle River, NJ: Wharton School, 2005. 265-86.

(18) Aravind Eye Care System. “Clinical Quality Assessment at Aravind Eye Hospitals.”

(19) Prahalad, C. K. The Fortune at the Bottom of the Pyramid. Upper Saddle River, NJ: Wharton School, 2005. 265-86.

(20) ReSurge International. “The Solution.”https://www.resurge.org/solutions/. Accessed on 1 May 2020.

(21) ReSurge International. “International Medical Partners.”https://www.resurge.org/international-medical-partners/. Accessed on 1 May 2020.

(22) ReSurge International. "Surgical Mission Trips." 'https://www.resurge.org/volunteer-surgical-mission-trips/. Accessed on 1 May 2020.

(23) Ibid.

(24) Corlew, S., & Fan, V. Y. (2011). A model for building capacity in international plastic surgery: ReSurge International. Annals of plastic surgery, 67(6), 568-570.

(25) ReSurge International. “50 Years of Experience.”https://www.resurge.org/50-years-of-experience/. Accessed on 1 May 2020.

(26) "Outcomes Are Essential in Global Health." Unite For Sight. https://www.uniteforsight.org/global-health-university/outcomes. Accessed 1 May 2020.

(27) Haynes, A. B., S. E. Regenbogen, T. G. Weiser, S. R. Lipsitz, G. Dziekan, W. R. Berry, and A. A. Gawande. "Surgical Outcome Measurement for a Global Patient Population: Validation of the Surgical Apgar Score in 8 Countries." Surgery 149.4 (2011): 519-24.

(28) Haynes, A., Weiser, T., Berry, T., et al. “A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population.”The New England Journal of Medicine 360 (2009): 491-499.