Managing Data for Performance Improvement

Part 1: Overview

Part 2: Using the Data for Analysis and Interpretation

Part 3: Related Resources

Part 1: Overview 

Effective data management plays an important role in improving the performance of an organization’s health care systems.  Collecting, analyzing, interpreting, and acting on data for specific performance measures allows health care professionals to identify where systems are falling short, to make corrective adjustments, and to track outcomes.  This module is designed to help users understand the relationship between quality improvement and data management and to provide information on how to gather, analyze, interpret, and act on data for a specific performance measurement.

How do I identify what data I have?

Simple data collection planning is a process that ensures that the data you gather are useful, reliable, and resource-efficient. Designing a data collection plan will help to prevent errors in the data collection process.

There are several factors to consider when planning for data collection:

• What information needs to be collected in order to address each quality measure?
  The data that you need to collect will be influenced by the areas where you are seeking improvement and the measures you intend to use. As the types of questions differ, so will the kinds of data best suited for use in the evaluation of your program.
• What are the information sources?
  You must determine where to find the best source of data to answer each of your evaluation questions.  Possible sources of data include people (e.g. program staff, clinicians, or patients), records, or clinical observations.
• How should information be collected (methodology)?
  Surveys, interviews, focus groups, literature reviews, and record analysis (e.g. chart audits) are just a few examples of data collection methods.  Registries have offered an important opportunity for tracking quality measures. As providers move to adopt EHRs and as e-specifications for quality measures get developed and EHR-based quality reporting becomes standard practice. It is important to choose the data collection method that is best suited to your evaluation questions.  There is often more than one way to collect data to answer a given question.  Some questions are best answered by using more than one data collection method.  For example, you may want to do a chart review to understand practice patterns and then conduct interviews with a smaller number of providers to understand more detailed information about the observed practice patterns.
• How much data should be collected?
  It is not always necessary to collect all of the data available to you.  If the data on the full population you are looking at is very large, evaluating a subset, or sample, may be sufficient. On the other hand, if you are interested in using QI reports to create “profiles”, or snapshots of provider performance measures, and manage the performance of providers, then you may want comprehensive data.
• What timeline is being followed to meet task deadlines?
  The structure of your data capture timeline will depend on the resources available to you, as well as logistical program considerations.  Before beginning data collection, it is helpful to determine how often data will be collected and what deadlines need to be met.  It is a good idea to allow enough time for unforeseeable problems with the data capture process.  

How do I use these data to construct measures?

Quality measures are constructed using a variety of methods, including proportions, ratios, means, medians, and counts. Which method you choose depends on which quality measures you have selected and which evaluation questions you are trying to answer. Proportions or percentages with a numerator and a denominator are the most common way to construct quality measures. The proportion’s denominator represents the number of patients at risk of or eligible for the numerator event. For example, the denominator might consist of the total number of patients undergoing surgery, and the numerator might represent the total number of those patients who experienced respiratory failure after the procedure. For more information on how quality measures are constructed, visit the website of the Agency for Healthcare Research and Quality, which provides an introduction to measures of quality.

There are many existing quality measures available to you.  Measures endorsed by the National Quality Forum (NQF) can be found on the NQF website in a searchable directory categorized by measure type, measure steward (entity that designed and maintains the measures), or care settings.  (If you wish to develop a measure and submit it for endorsement at the NQF, you must complete the consensus development process, which consists of nine principal steps.)

Through its National Quality Measures Clearinghouse, the Agency for Healthcare Research and Quality (AHRQ) also provides a searchable database  of quality measures categorized by various patient and provider characteristics, as well as a current list of the top ten most-viewed measures during the last 30 days.

When and why do I sample? 

There are times when you will want to look at quality measures that represent all eligible patients within an organization. For example, if you are trying to track the performance of specific providers or sites and each provider does not have a large enough set of patients to allow for sampling. In many other cases, sampling allows you to make inferences about a large group (e.g., all diabetic patients in your organization) based on observations of a smaller subset of that group (a smaller subset of those patients). Sampling your data saves time and resources while still accurately evaluating performance.  

Random sampling is a technique used to reduce the likelihood of bias when collecting samples.  The main benefit of random sampling is that it ensures that the sample you choose will be representative of the larger population that it was drawn from.  This is important because any conclusions you draw from a representative sample can be extrapolated to the entire population.  In other words, the results generated by your sample can be accurately projected onto the larger population.  In random sampling, each subject is selected entirely by chance.  There are several different methods for selecting random samples. For quality improvement, a simple sample is often sufficient.

How do we calculate measures?

Quality measures should be specific in terms of time frames and patient/program characteristics (i.e. gender, provider type, age, etc.) and based on guidelines or standards of care when possible.  When calculating measures, there are five steps to follow:

1. Define the measure.
2. Define eligibility in the measurement population
3. Set the denominator
4. Set the numerator
5. Divide the numerator by the denominator to get the performance percentage

Example: Calculating a Pap test measure for an HIV clinic:

1. Measure: Percent of females  age 18 and older who received an annual Pap test
2. Eligibility: HIV+ ambulatory patients who have had at least 2 primary care visits in the last year: 380 patients
3. Denominator: Female patients, age 18 and older: 190 patients
4. Numerator: Number of female patients  age 18+ who received a Pap test at this clinic within the last 12 months (and have the test results documented in the medical record): 115 patients
5. Measure: (# of female patients 18 and older who received Pap test) divided by (total # of female patients 18 and older): 115/190 = .605 or 60.5%

When calculating measures such as this one, it is important to bear in mind that there are many factors that influence the results.  In this example, it is possible that some female patients are declining to have a Pap test or receiving the test elsewhere.  These are important factors to bear in mind when making quality improvement decisions based on quantitative measures.

Electronic health records (EHRs) can be helpful in constructing and tracking quality measures.  EHRs aid in quality improvement by improving clinical documentation and by making standardized patient data readily available for collection. EHRS are a source of detailed, current patient information for use in constructing performance measures.

What is the best way to standardize data processes?

A standardized data collection procedure is essential for successful quality improvement.  A standardized data process will simplify the task of quality improvement by allowing you to collect accurate and consistent data and generate reliable information to act upon.  

When collecting data from medical charts, standardized chart audit forms are a simple way to ensure that data collection is consistent and thorough.  The Migrant Clinicians network offers a series of standardized chart audit forms for quality management including well-child care evaluations, medical chart reviews for individual providers, and WIC chart audits.  You can also create your own standardized chart audit forms to facilitate data collection.  
When collecting data from patients or providers in the form of survey or interviews, standardized questionnaires and interview scripts will increase the likelihood that each subject interprets the questions in a similar way and that the data you collect is accurate and complete.