Testing for Improvement

Part 1: Introduction

Part 2: Methods for Testing Changes

Part 3: Successful Tests of Change

Part 4: Test for Improvement

Part 5: Implementation

Part 6: Supporting Information

Part 2: Methods for Testing Changes 

The Managing Data for Performance Improvement module explains that most QI work involves closing the gap between a system's actual and ideal operation. An organization's goal is to apply changes that result in improvement to close the gap. To achieve this goal, a QI team understands what change is needed and how to implement it. The Core Clinical Measures (CCMs) modules provide recommendations of changes to improve patient care and the improvement models guide a team on how to implement those changes.

The Model for Improvement, developed by Associate in Process Improvement, provides a framework for developing, testing, and implementing change, and it is a powerful tool for accelerating improvement. The model shown in Figure 2.1 is meant to augment an organization's existing change model—not replace its current model. The Model for Improvement is used to successfully improve care processes and outcomes by numerous health care organizations. The model comprises two equally important parts: ⁽¹⁾

• Part 1 presents three fundamental questions that are essential for guiding improvement work:
  1. What are we trying to accomplish? An organization's response to this question helps to clarify which improvements it should target and their desired results.
  2. How will we know that a change is an improvement? Actual improvement can only be proven through measurement. An organization should think about how it wants things to be different when it has implemented a change and agree on what data needs to be collected for measuring. A measureable outcome that demonstrates movement toward the desired result is considered an improvement. For example, showing how the service that patients receive will improve, or how an organization's processes might change.
  3. What changes can we make that will result in improvement? Improvement occurs only when a change is implemented, but not all changes result in improvement. One way to identify which change will result in improvement is to test the change before implementing it.
• Part 2 involves the Plan-Do-Study-Act (PDSA) cycle that tests and implements a change in real-work settings.

One of the most common tools for improvement is the Deming (or Shewhart) Cycle. This method is also known as Plan-Do-Check-Act (PDCA) or Plan-Do-Study-Act (PDSA), and it is well suited for many improvement projects. The PDSA cycle is shorthand for testing a change - by planning it, trying it, observing the results, and acting on what is learned. This is the scientific method used for action-oriented learning. Many quality improvement practitioners believed that the Check stage of the process meant to simply measure the improvement and move forward to the Act stage. Figure 2.2 below shows how the PDSA cycle operates:

The PDSA Cycle starts at the Plan stage. When a QI team o understands the nature of the current problem, the process that underpins the problem, and has specific ideas about what would mitigate the problem, it is ready to test changes to that process. The Plan stage helps the QI team to determine this by working through the following questions:

• Which process needs improvement?
• How much improvement is required?
• What change should be implemented?
• When should the change be implemented?
• How should the effect of the change be measured?
• What does the change affect (such as, documents or procedures)?

Before changes are tested, the team should secure the buy-in of those that will be affected. Whether the reason for change is due to patient challenges, unreliability, or a continual improvement opportunity, it is important to keep people informed. This ensures their cooperation and results in an effective test of change.

Testing the change occurs during the Do stage. The QI team tests the change and collects the required data to evaluate the change. In addition, any problems and observations during the test are documented. An analysis of the data naturally occurs during the next stage, Study.

In the Study stage, the QI team learns all it can from the data collected during the Do and considers the following:

• Is the process improved?
• If improved, by how much?
• Is the objective for improvement met?
• Is the process more difficult using new methods?
• Did anything unexpected happen?
• Is there something else to learn?

The responses derived from the Study stage define the QI team's tasks for the Act stage. For example, if the process is not improved, the team may review the change tested to determine why, then further refine it, or plan another test cycle.

The QI team may choose to start again with a new test cycle based on the analysis. If the problem is unsolved, the team may return to the Plan stage to consider new options.

If the process improves, the team should determine if the improvement is adequate. For example, if the improvement speeds up the process, the team should evaluate it to see if it is fast enough to meet its requirements. If not, the team may consider additional methods to tweak the process until its improvement objectives are met. It also may consider testing the same step of the process, or possibly a different step in the process, to reach its overall goal. Again, the QI team is back at the Plan stage of the PDSA cycle. For most system changes in health care, multiple small tests of change are needed to improve one system. Fortunately, these tests are performed in a very short time so overall improvements can be accomplished efficiently.

The case study continues with an example of RHC's approach to the PDSA process:

The case study may be read in its entirety by clicking here.

The Case Study: The Approach

The concept of linking PDSAs to accomplish a change is covered in the next section, Using Multiple Test Cycles.

The use of multiple test cycles helps a QI team improve upon each test of change, as shown in Figure 2.3. Linked PDSAs focus on improving one process, but they also reflect the complexity that those in health care encounter. Sometimes linked PDSAs show that multiple interventions need to be applied at one step. Each test is a learning experience and identifies if the team needs to make additional steps or changes to improve the targeted process. For example, one team evaluates the benefits of group visits as a means to improve diabetes care. It performs PDSAs around three factors-group size, the weekday and meeting time, and the content-to-conversation ratio. The team considers each factor and tests it over time, but the results of the three sets of PDSA cycles show that a group-visit strategy enhances diabetes care for a subset of patients.

An organization may learn from linked PDSAs of some unintended consequences and issues that it did not consider initially. For example, a clinic follows HIV patients on Highly Active Antiretroviral Therapy (HAART) and decides to test an aggressive approach for retaining patients in care. The QI team designs an outreach system to contact patients 75 days after their last visit and to remind them of their need for care. When the clinic tests the system, however, it learns that a small percentage of patients dislike receiving reminders about their illness; they pointed out their good track record by using their own systems. The team recognizes its approach is not patient centered and consults with the consumer advisory board before embarking on additional PDSAs to improve retention.

A QI team may use linked PDSAs to broaden the test of a change and ensure that special conditions or sets of patients are not missed. For example, a change may be tested on different clinic days or with other personnel. The team may continue linking tests in this way and refining the change until it is ready for a broader implementation. People are more willing to test a change when they know the changes can and will be modified as needed. Linking small tests of change helps an organization to overcome its natural resistance to change and helps to promote buy-in from the staff.

Tips for Successful Linked Tests of Change ⁽²⁾

1. Plan multiple cycles for testing a change.
2. Think ahead through a couple of cycles.
3. Scale down the test size (the number of patients or location).
4. Test with volunteers.
5. Focus on getting consensus or buy-in from management, staff who will do the work, and patients and families.
6. Be innovative to make the test feasible.
7. Collect useful data during each test.
8. Test over a wide range of conditions.
9. Use a quick test; for example, ask "What change can we test by next Tuesday?"

The case study continues below and exemplifies RHC's use of repeated PDSA cycles to improve adult influenza vaccination rates:

The case study may be read in its entirety by clicking here.

The Case Study

Most QI literature references the PDSA process-small sequential tests of change to reassure the team that change will result in improvement when tested before implementation.