Testing for Improvement

Part 1: Introduction

Part 2: Methods for Testing Changes

Part 3: Successful Tests of Change

Part 4: Test for Improvement

Part 5: Implementation

Part 6: Supporting Information

Part 4: Test for Improvement 

Quality is directly linked to an organization's service delivery approach or underlying systems of care (see Quality Improvement module). To achieve a different level of performance and improve quality, an organization's current system needs to change. When planning a test of change, a successful QI program uses four key principles, which are described in the following subsections: ⁽³⁾

• QI work as systems and processes
• Focus on patients
• Focus on team approach
• Focus on data use

QI Work as Systems and Processes

A QI team must know the organization's delivery system and key processes to make improvements. In a health care organization, there are generally two major activities or processes: 1) what is done, that is, what care is provided, and 2) how it is done, that is, when, where, and by whom care is delivered. Improvement is achieved by addressing either function; however, the greatest impact for QI is when both are addressed simultaneously. A QI team usually considers both when planning a test of change.

Process mapping is a tool used for understanding a health care system's processes better. The map is a visual diagram of sequential events that cause a particular outcome. A QI team uses the process mapping tool to evaluate or redesign a current process. By reviewing the sequential steps, who performs each step, and the overall efficiency of the process, the team can identify opportunities for improvement. When the team compares its map to one that shows optimal service care, other opportunities to improve care are presented. The process map in Figure 4.1 is an example of how RHC, in the case study improved vaccination rates after its process was changed.

The Root Cause Analysis method is used to identify or get to the "root cause" of a problem by correcting or eliminating it and preventing its recurrence. The philosophy of Root Cause Analysis is each problem presents an opportunity, because it shows how and why the problem occurred. An organization must understand the true cause of a problem in order to correct it. If a problem's root cause is unknown, an organization wastes time and resources attempting quick fixes. Root Cause Analysis delves into why the problem occurred until the root cause or failed process is identified. It assumes systems and events are interrelated, and an action in one area triggers action in another. An organization traces the actions and unveils the problem's origin and progression to its current level of symptoms. There are three basic types of causes:

1. Physical causes are tangible material items that failed in some way.
2. Human causes are people not performing a requirement or performing it incorrectly. Human causes typically lead to physical causes.
3. Organizational causes are faulty systems, processes, or policies used for decision making.

Focus on Patients

An important quality measure for an organization to study is the extent its patients' needs and expectations are met. Patient-focused services include:

• Systems affecting patient access
• Evidence-based provision of care
• Patient safety
• Patient engagement
• Coordination of care with other parts of the larger health care system
• Cultural competence
• Assessment of patients' health literacy
• Patient-centered communication
• Linguistically appropriate care

Focus on Team Approach

A team approach is critical when testing change. The collaboration of knowledge, skills, experiences, and perspectives of different team members cause lasting improvements. A team approach is most effective when the system is complex and involves multiple disciplines or work areas. One person in an organization cannot know all the dimensions of a complicated process or issue. It requires innovative solutions developed in a team environment, staff commitment, and buy-in. In the RHC case study, the team approach to testing change resulted in less burden on individual members. The QI team gained confidence and moved quickly to the next test cycle with little disruption to the delivery system, where the process of educating patients on the importance of influenza vaccination was tested.

Each individual is responsible for being an active and contributing member of the team and bringing a unique perspective to the process; i.e., how things work; what happens when changes are made, and how to sustain improvements during daily work. Contributions are made from an individual's skill set and the team's synthesis of ideas. Additional information, tools, and resources for developing and supporting an organization's QI team are in the Improvement Teams module.

Focus on Data Use

Data is the cornerstone of QI and an important component when testing a change. Data demonstrates how well an organization's current systems are working, what happens when changes are applied, and documents a successful performance. Data separates what is thought to be happening from what is really happening. It establishes a baseline, reduces placement of ineffective solutions, and indicates whether changes lead to improvements.

Both quantitative and qualitative methods of data collection are useful when planning tests of change. Quantitative methods use numbers and frequencies to develop measurable data. In the RHC case study, quantitative data was used to determine an area for improvement. The RHC QI team noted only 50 percent of its adult population received influenza vaccination. This concerned the team, and they decided to target influenza vaccination rates for improvement. Qualitative methods collect data with descriptive characteristics, rather than numeric values that draw statistical inferences. Qualitative data is observable but not measurable, and it provides important information about patterns, relationships between systems, and often provides context for needed improvements. In the RHC case study, the team conducted outreach to patients to determine why its adult patients did not receive influenza vaccination. It also collected qualitative data and noted an overarching need to develop a communication and education plan.

Documenting the testing process is crucial, because an organization can study the data (or documented test process) before, during, and after the testing process. It provides qualitative data needed to make improvements to an organization's care systems. Documentation is collected during the test cycle, which enables an organization to evaluate and understand the test process. The PDSA Worksheet is useful for documenting a test of change. This tool guides an improvement team in developing a plan to test the change (Plan), carrying out the test (Do), observing and learning from the consequences (Study), and determining needed modifications for the test (Act). The PDSA process and tool also help to develop additional test cycles, which build knowledge in a sequential step.

An organization's ability to develop, test, and implement change is essential for improvement. Many types of changes lead to improvement, but are developed from a limited number of change concepts. A change concept is a useful approach for developing specific change ideas that lead to improvement. Ideas for tests of change are generated by subject-specific experts on the team who creatively combine change concepts. After generating ideas, a QI team runs PDSA cycles to test a change or group of changes on a small scale to see if improvement occurs. If a test of change causes an improvement, the team may expand the tests and gradually incorporate larger samples until it is confident that changes can be widely adopted. Listed below are a few techniques for the QI team to consider when developing a test of change:

1. Creative thinking techniques generate new ideas, different perspectives, and break down large amounts of information quickly. These techniques involve participation of the entire team to develop multiple solutions to problems.
2. Brainstorming is a team process for generating creative ideas for improvement. It involves everyone on the team, encourages creative thinking, and results in a comprehensive list of ideas. Brainstorming allows ideas to build upon each other as the team works together. General tips for brainstorming include: 1) ensure everyone on the team understands the outcome, and 2) set ground rules, such as, all ideas are welcome and listen to others without comments or criticism. Ideally, brainstorming ideas are written on a flip chart and read aloud to ensure everyone on the team understands the ideas or has an opportunity to request clarification.
3. Affinity analysis helps a QI team gather and organize a large amount of information. It uses brainstorming and organizes information and ideas into categories. The team sees the big picture of a problem and understands how it relates to other areas within the organization. Affinity analysis is useful when the problem needs resolution quickly or it is overwhelming.
4. Steal shamelessly means there is no need to reinvent a process. A QI team is encouraged to "steal" what worked in other organizations by using existing knowledge of evidenced-based processes and modifying them for its own system.

A prediction is a statement about a system or process' expected performance. With each change cycle, the QI team usually predicts the change will result in improvement. A QI team's degree of confidence in its prediction is based on two considerations: 1) extent of evidence supporting the prediction, and 2) similarity between the conditions under which evidence was obtained and conditions that apply to the prediction. A QI team's confidence increases in the change will result in improvement, as its ability to predict a test's outcome over various conditions improves. When a team incorrectly predicts its test results; however, it questions the validity of the prediction theory. The reassessment process provides an opportunity for QI teams to explore new strategies for improvement through continued and refined PDSA cycles.

Measuring the actual change process is the only way to know if change results in an improvement. A QI team's actions are determined by what it learns from the change. For example, an organization's action to implement the change may involve refining and testing the change again. This stage includes analyzing the test cycles, reflecting on what was learned, comparing predictions to the data collected, and making decisions. Since change in one area of the organization can impact another, it is important to review the entire system and ensure another area is not adversely affected.

In many situations, the results of quality improvement work change the way an organization thinks about and manages data. While QI is not solely about measurement, an organization needs a process in place to manage and interpret data to determine if a change actually results in an improvement. Key outcome measures are required to assess an organization's progress in achieving its aim. Additional information on performance measures in quality improvement is in the Managing Data for Performance Improvement module.