Module 4: Testing and Implementation

STAGE 1: Prototype Evaluation by Decision Matrix

While creating prototypes, innovators may discover that a seemingly successful brainstorm is not adaptable to concrete design. Although this simplifies the process of narrowing down brainstorms to an ideal solution, there may still be a handful of substantial prototypes to consider. As such, another decision matrix is employed.

This matrix should use the same design criteria previously identified, and should give the same weight of significance to each element. Maintaining this consistency in evaluation is critical for the validity of the design process. After completing the matrix, total scores can be compared for objective ranking among prototype options.

In this stage, it is possible that, despite the results of the decision matrix method, innovators or other stakeholders in the technology have predetermined preferences for a particular idea. In many cases, these preferences are not aligned with the highest scoring option from the decision matrix. Whether other factors will override the analysis presented by the decision matrix is to be determined on a case-by-case basis. In some cases, decisions in development may be guided by donor interest or investment by stakeholders in public-private partnerships. To avoid barriers to the development of appropriate designs, there are various precautions that can be taken to ensure compatible interests among innovators and stakeholders. For instance, donors should demonstrate an understanding of the global health challenge, technology performance specifications that would meet needs, a commitment to providing available resources, potential obstacles to design, and the realistic value of benefits offered by the technology-in-development.(1) Although the prototype chosen for continuation at this stage is not the be-all, end-all of the design process, a carefully selected idea (ideally with appropriate justification) can prevent future delays in technology implementation and delivery.

Stage 2: Testing Design Criteria

Although the decision matrix methodology assesses how well the idea or prototype fulfills each design criterion, further development requires evidence-based measures of success. In a way, the testing phase serves to justify scores achieved in decision matrix evaluation. Accordingly, each criterion should incorporate a system of quantitative testing – made possible since criteria are measurable.

Plans for testing vary according to the criterion under consideration. Testing may involve physical testing of technology hardware or components, as well as the administration of surveys to assess technology interface with human factors. The only requirement is that descriptions or instructions should be standardized to maintain consistency and control during experimentation. Regardless of the testing style chosen for this phase, the goal of testing is to determine if the selected design has weaknesses that may be improved upon.

Examples of testing plans for measurable design criteria include:

• Accuracy of intervention in prevention, diagnosis, or treatment

  • Run trials of the technological intervention and traditional standard of care. Compare results.

• Economic feasibility

  • Conduct cost-effectiveness analyses, comparing costs of development (including required consumables) and consequences against the valued benefits.

• Durability

  • Test operation of device or intervention before and after representative environmental conditions, i.e. exposure to heat, water, or turbulence.

• Sustainability

  • Assess the maximum length of time that a device can continue operation until necessary replacement or new parts, i.e. battery life or count of consumable parts.

• Ease of use

  • Administer surveys to sample demographic. Use a scale (0 to 10) to provide quantitative results of measurement.

• Adherence to quality and safety standards

  • Compare features of use to existing standards from regulatory institutions.

Stage 3: Reiterations

• Using findings from the testing phase, adapt prototype designs accordingly. In this process of developing new or improved iterations of design, it may be beneficial to draw from previously eliminated ideas (in the brainstorming or decision matrix stages) for creative ways to re-iterate.

  The opportunity to create new iterations of a design also may be used to elicit more feedback. For instance, it may be easier to find weaknesses in technology design by comparing and contrasting the iterations along tests outlined in the previous stage. Findings from this process would support the creation of even more iterations in technology.

Stage 4: Implementation, In Stages

Technology implementation is the application of the innovation to real-life settings of use. It is suggested that implementation occur in phases, beginning with small pilots prior to broad scale-up. Pilots may vary in location of use, age of user, or gender of user, providing insight beyond simply whether the technology is appropriate in practice. By using this step-wise progression, it is possible to continue the “positive-feedback loop” of assessment and modification. The goal is to develop the most appropriate technology design, which requires different perspectives and settings for fine-tuning.

For each pilot, it is recommended to consider the application of use in terms of context (i.e., What questions can we answer from this constituent group? What can we discover in [x] weeks?), resources (i.e., Are allocated people and funds representative of actual use?), questions to answer (i.e., Do key issues arise about this concept and its desirability?), and how to measure success (i.e., What are the key expectations in the various phases of implementation?).(2) These questions are intended to make innovators mindful of the limitations imposed by implementation; this process is only the first step to bringing a technology to scale. At this stage, phases of implementation are still trials in a continuous process of technology design and development.

Stage 5: Gathering Empirical Evidence

In this stage of development, all data is crucial to the success of the chosen technology. It is best to integrate both qualitative and quantitative methods of learning in an approach that will best collect information about the device or program, as viewed from various stakeholders’ perspectives.

IDEO suggests a “Learning Loop,” shown below. At this point, the focus is on Indicators, to track progress, choose ideas, iterate solutions, and identify unintended consequences.(3)

Specifically, these indicators may be positive or negative, and intended or unintended. It is important to view all categories to determine the full scale of impact. The following chart lists types, descriptions, and examples of indicators, as taken from the IDEO Toolkit:

Focus on each stakeholder and ask for feedback regarding each type of indicator: Are thereleading indicators that we should be tracking? Are there analogous indicators that we can track? How can we measure awareness and engagement? How will we track and understand the dynamics of transformations that are occurring?(4) If it is difficult to immediately jump into these types of indicators, it is helpful to begin simply with the broad question: What can I expect to see happen if solutions are improving people’s lives?

Although the importance of gathering feedback is highlighted explicitly in this stage, the learning process is ongoing. Design does not end with implementation; implementation is simply another phase of the development cycle. Additional methods of evaluating program success are outlined in the Course on Evaluation.

Stage 6: Reiterate, Reiterate, Reiterate

It is never too late to make improvements to design. Although this module followed the selection of an idea for prototyping through testing, re-iteration, implementation, and data collection, modifications or new iterations from this point forward are to be expected in the development process. Feedback in the form of success or failure is informative to the creation of appropriate and effective technology, and innovators of global health technology should welcome continued opportunities to improve upon their designs.

Footnotes

(1) Free, M. “Achieving appropriate design and widespread use of health care technologies in the developing world.” International Journal of Gynecology and Obstetrics. 85.1(2004): S3-S13.

(2) IDEO. “Human-Centered Design Toolkit, 2nd Edition.” https://www.ideo.com/post/design-kit.

(3) Ibid.

(4) Ibid.