The javascript used on this site for creative design effects is not supported by your browser. Please note that this will not affect access to the content on this web site.
Skip Navigation
H H S Department of Health and Human Services
U.S. Department of Health and Human Services
Health Information Technology and Quality

A-Z Index  |  Questions?  |  Order Publications  |  HRSA Mobile

Appendix: Sample QI Tools Developed by HRSA Grantees

This appendix includes a brief description of the QI process and links to QI tools developed by HRSA grantees.  The purpose of the section is to provide some examples of how EHRs are used to support QI initiatives in these organizations.   

I. Alliance of Chicago

The Alliance of Chicago utilizes EHR technologies at all of their four core health centers.  In addition to an EHR, The Alliance has created reporting dashboards; the Health Outcomes Dashboard and the Performance Excellence Dashboard . The Health Outcomes Dashboard reports on clinical measures and outcomes, whereas The Performance Excellence Dashboard reports on user and patient satisfaction.  A Key Performance Indicator Dashboard that includes selected measures from the Health Outcomes Dashboard and the Performance Excellence Dashboard.

Creating a culture of quality.  The Alliance had two main goals when they embarked on the process of implementing health IT at their health centers.  The first goal was to improve the quality of care.  Health centers felt that health IT would allow them to, among other things, compare their clinical outcomes to national benchmarks and enable them to better adhere to guidelines in treating chronically ill patients.  The second goal that the Alliance had was to increase administrative efficiency.  The Alliance felt that implementing an EHR would help them to have more efficient workflow at clinics, eliminating the burden of locating paper charts, and bettering the referral process.

Creating and utilizing clinical decision support (CDS) functionality in the EHR is one way that the Alliance has promoted clinician involvement in quality improvement activities on a daily basis, thus adding to the culture of quality. This CDS has very few hard stops, and allows clinicians to set the prompts to be more passive or active.  

Additionally, the Alliance employs a network-wide QI director. This has allowed the Alliance to take a more hands-on approach to assisting health centers in incorporating a culture of quality into their workflow and has aided health centers in making efficient use of QI tools. The Alliance rolled out their EHRs and Dashboard Reporting formats at one health center at a time, allowing for lessons learned during one implementation to be used during the next implementation.

The Alliance has worked closely with their vendor to incorporate key decision support and QI elements into their EHR. The vendor helped the Alliance to create disease management screens, listing preventative and treatment services relevant to particular disease populations, and to set-up provider reminders and alerts. The Alliance also worked closely with their vendor to utilize its clinical data warehouse that allows the Alliance to produce multiple reports from the EHR data.

Selecting measures.  Clinical measures being tracked on the Health Outcomes Dashboard are national measures that are based on HRSA's core clinical measures, Health Disparities Collaborative measures (diabetes),  and other measures from the NCQA and for Pay-4-Performance incentives (hypertension, coronary artery disease, HIV/AIDS). They chose to use these measures because the numerator and denominator of these measures are well specified, and feel that the promotion of use of national measures is important.  User and patient satisfaction measures being tracked on the Performance Excellence Dashboard include: access to care metrics, patient satisfaction, EHR system use, and EHR system user satisfaction.

Determining data sources.  The data in the Health Outcomes Dashboard comes from clinical documentation (e.g., vital signs, smoking status) and orders (e.g., procedural orders, medication orders) entered by clinical staff or from results (e.g., lab or radiology results) directly reported into the Alliance’s EHR.  The warehouse can de-identify data from their aggregate pool of customers. The data included in the Performance Excellence Dashboard comes from administrative data from the EHR and surveys issued to assess user satisfaction ,as well as patient satisfaction. These surveys and subsequent reports are issued on a quarterly basis.   

Collecting and analyzing data and reporting.  Clinical data from the Alliance’s EHR is sent to the vendor’s data warehouse where it is de-identified and aggregated.  The Health Outcomes Dashboard is then generated applying the Alliance’s reporting specifications to allow for assessing individual health center performance as well as allow for comparison of data across all of the Alliance’s health centers or against national benchmarks, when available, over specific periods of time. These comparisons are issued on a monthly basis.

  • The Performance Excellence Dashboard is generated by examining administrative data from the EHR as well as analyzing survey results from both patients and clinical users.  
  • Additionally, the Alliance’s health centers can run their own detailed reports on clinical measures, as well as on the performance excellence measures. These reports can be organized in many ways, including by site, disparity population, or provider.  The health centers are trained to run these reports on their own.  Some health centers are running reports for incentive payments, and some are providing data to the clinicians regularly so they can follow QI progress.  

Sharing results.  The Alliance produces reports on a quarterly or monthly basis, depending on the report.  Producing and disseminating reports at regular intervals allows physicians, administrators and the quality improvement team to track the health centers progress. The Alliance continually monitors the progress of the technologies and QI efforts through regular reporting and feedback from clinic staff.

II. Blackstone Valley Community Health Care

Blackstone Valley Community Health Care (BVCHC) is utilizing the Crystal Reports offering from SAP Business Objects, in conjunction with their EMR, as a tool to track patient care and to generate quality reports.   BVCHC also utilizes clinical decision support (CDS) technology with their EMR to help clinicians provide the highest quality of care possible.

Create a culture of quality.  To create a culture of quality at their health centers, BVCHC introduced their reporting tool to improve direct patient care, as well as for incentive payment programs with various payers. They have added a CDS system, also called a health maintenance alert system), to help ensure that clinical guidelines are being followed at the point of care.  Alerts show up in several places in the EMR and display items appropriate to age and gender based on national standards.  Items that need attention are highlighted in red, alerting providers that a certain procedure or measure is out of date for a particular patient.

Additionally, staff members are encouraged to practice at the top of their license to streamline work flow processes and ensure patients receive high quality and efficient care.  MAs regularly order tests in the EMR at the point of care, and have test results printed and in the hands of the clinician when he/she sees a patient.  This ensures that clinicians are receiving up to the minute data.

Selecting measures. Prior to implementing their reporting framework, the QI  team at BVCHC worked with the IT team to find the group of metrics that were most closely aligned with existing national recommendations and that would be the most beneficiary at the health center.  The measures currently being tracked are based on HEDIS go to exit disclaimer, HRSA’s core clinical measures and other national metrics and federal mandates.

Determining data sources. The data in the reports comes directly from BVCHC’s EMR.  Diagnosis information is coded to ICD-9 and is used to generate diagnosis-based patient registries to monitor health information, such as patients with diabetes (i.e., patients with ICD-9 code 250.XX).  In other circumstances, such as primary prevention, other demographic and clinical variables may be used to generate a patient registry.  For example, gender and age are the data sources used to define a panel report to identify appropriate patients for female health management. 

Collecting and analyzing data and reporting.  BVCHC generates clinical data reports for regulatory and compliance reporting, as well as for internal quality checks. The data comes from the EMR and a business intelligence application is used to generate the reports.  The data in the reports can be organized in many ways, including by timeframe, by patient registry, or by provider, allowing BVCHC to analyze their data for many different reports and purposes. Lab results, ordering information for follow-up tests and exams, and other data documented in structured formats in the EMR are used to populate reports. All HRSA measures are highlighted in yellow in the reports.  BVCHC noted the difficulty of exclusion criteria and indicated, for the most part, all patients are included in the report.  Some report examples include:

  • Asthma Report.  Reports on patients in the asthma registry (i.e., patients with an ICD-9 code of 493.XX) are generated over a specified timeframe by provider.  The date self-management goals were reviewed and the date that symptom-free days were recorded are both documented in the EMR.  When the symptom free days are recorded, the number of symptom free days is also entered into the EMR. The patient’s tobacco exposure status and the date this was reviewed are also documented.  A summary of a particular provider’s performance related to recording and monitoring patient’s self-management goals, symptom free days and tobacco exposure status can be generated. 
  • Diabetes Report.  For diabetes, a report can be created to monitor patients in the diabetes registry (i.e., patients with an ICD-9 code of 250.XX) by provider and timeframe.  The report includes lab results (HbA1c, LDL) and blood pressure measurements, including the date and value, as well as the date that ophthalmologic exams were ordered and completed.  In addition, the report notes the patients who do not have a documented HbA1c in the past three months.  The provider’s performance is monitored based on the measures tracked in this report.
  • Immunization Report.  Immunization reports are generated for patients who turned 2 during this timeframe with at least one visit between ages 1 and 2.
  • Well Child Report.  A report can also be created that shows if a child has been seen in the clinic over the past year.  

Sharing results. BVCHC’s quality reports are generated on a monthly basis and are organized by provider allowing providers to see how they are doing in terms of meeting quality improvement and patient care goals.  The reports are presented to providers in three different ways: 1) the reports are shared in the monthly provider meetings, 2) the QI director meets with each provider individually to go over his/her report, 3) the report is automatically emailed to each provider on the 1st of the month for providers to review at their convenience.  There is a Provider Report Card at the end of the report which shows the average of each metric for each provider.  Providers are able to see each other’s Report Card, which fosters healthy competition and allows for the sharing of best practices.

III. Boston HealthNet

Established in 1995, Boston HealthNet (BHN) is a coordinated, integrated health care delivery network comprised of Boston Medical Center, Boston University School of Medicine, and 15 community health centers, which serves Boston’s underserved and working class neighborhoods.  BHN has initiated two Health Information Technology projects, which focus on using health IT innovations as a tool to drive a Network-wide Continuous Quality Improvement (CQI) initiative. The projects have a direct impact on the clinical and business objectives of participating CHCs by using rapid cycle improvement activities to improve the safety, quality, efficiency and effectiveness of health care delivery.

  1. The Community Health Automated Record Technology (CHART)-Plus project aims to vertically integrate the Centricity EMR of eight Boston HealthNet community health centers  utilizing the same vendor product, with that of Boston Medical Center (BMC).  The goals of the CHART- Plus project are to: eliminate the need for separate log-in and search to access laboratory results, medication lists, allergy lists, problem lists and consult notes from BMC and other CHCs.  The long term goals of the project are to  provide BHN clinicians with access to other clinical information such as discharge summaries, immunizations and imaging documents from BMC and other CHCs, and to integrate CHCs that use another vendor’s EMR in data exchange and quality improvement programs.   
  2. The CHART eReferral project will implement an electronic referral management system that will link primary care providers and referral coordinators at the ten BHN primary partner CHCs to specialists in the BMC Departments of Medicine.  The eReferral system will make it possible for the CHCs to make or propose appointments, electronically submit required clinical information to the specialist with regards to the referral, be alerted and receive electronic information about the referral visit, track patients and receive notification of missed and kept appointments and specialist reports through the local EMR or by opening a web-based practice portal.

Creating a culture of quality. The two new projects that Boston HealthNet implemented will enhance a culture of quality at the health centers. For example, integration of BHN’s EMRs in the CHART Plus project will allow BHN clinicians to view patients’ clinical information aggregated from separate EMR’s across the network.  The information will be displayed within the local EMR providing real-time access to important clinical information needed to make effective care decision.

The BHN CHC’s who use Centricity EMR are hosted at BMC and benefit from their strong relationship with their vendor in order to ensure that any problems are addressed in a timely manner.  The vendor has also helped BMC and BHN to implement many QI aspects of the EMR, including an automating routing of prescription requests to pharmacies through e-Rx.  The vendor has also been integral in setting up the vertical integration of the EMRs as part of the CHART-Plus project.

Selecting measures.  BHN is tracking a number of metrics based on HEDIS measures  to assess the success of these projects.  For the CHART-Plus project, BHN is currently tracking the care of diabetics (improved LDL and A1C), the care of patients with cardiovascular disease (improved LDL) and patient safety (improving creatinine monitoring for patients on metformin, and improved liver function tests for patients on statins).  To measure success on the CHART eReferral project, BHN is tracking colon cancer screening, cardiology diagnostic testing, and provider and referral coordinators satisfaction.  Some of the denominators for the HEDIS measures have been modified for these projects.  To meet inclusion criteria, BHN noted that the patient needs to have been seen within the last year and to have the diagnosis entered in the problem list (for diagnosis-based measures).

Determining data sources. To monitor the measures, clinical data is pulled directly from the EMR.  BHN has made the decision to include all patients who meet the inclusion criteria regardless of the type and level of services provided.

Collecting and analyzing data and reporting.  While reports can be generated in real-time, reports for the CHART Plus project are being run using Access and are generally printed at the end of the month. Some health centers are also pulling reports comparing provider performances to each other, and to the practice as a whole.  Some reports can also be run prospectively in conjunction with the appointment scheduling system that can provide information regarding what care should be provided during the upcoming visit to improve the quality measure.

Sharing results.  BHN has a feedback loop between users and the QI/IT teams in place to facilitate rapid cycle quality improvement. Clinicians and users give regular feedback to quality coordinators at the health centers.  Coordinators and technology analysts also have monthly meetings to share this feedback in order to monitor the projects and to make adjustments as needed. Logs of additional comments are also kept for future reference.

IV. Institute for Family Health

The Institute for Family Health (IFH) utilizes an EHR at all of their health centers. IFH regularly runs reports using a business intelligence application on the data in the EHR to monitor progress. Additionally, IFH has implemented a clinical decision support (CDS) system to help them reach their quality improvement goals. The CDS system is one of the cornerstones of IFH’s Quality Improvement effort.  

Creating a culture of quality. The Institute is controlled centrally and the deployment of all clinical decision support tools were implemented across sites consistently.  All sites started with a basic set of CDS alerts, and new alerts were added over time.

When IFH first decided to develop CDS capabilities they had several goals.  Two of their top goals were to bring public health and epidemiologic factors to providers’ attention at the point of care, and to create alerts about the conditions that are leading causes of morbidity and mortality among New York City residents. Currently, IFH has built tools and alerts to support the goals of the New York City Department of Health and Mental Hygiene’s Take Care New York (NCNY) initiative to promote preventive health.

IFH utilizes their CDS to bring quality improvement to the attention of clinicians during every patient visit. Although providers can bypass some alerts, they are encouraged to satisfy all alerts to ensure that clinical guidelines are being met.  

Selecting measures.  In developing the measures for the TCNY initiative, IFH worked with the New York City Department of Health and Mental Health. The ten TCNY goals for patients are:

  1. Have a regular doctor or other health care provider.
  2. Be tobacco-free.
  3. Keep your heart healthy
  4. Know your HIV status.
  5. Get help for depression.
  6. Live free of alcohol and drugs.
  7. Get checked for cancer.
  8. Get the immunizations that you need.
  9. Make your home safe and healthy
  10. Have a healthy baby.

For each goal, measures were defined that included appropriate numerators and denominators.  When possible, national measures (HEDIS and NCQA go to exit disclaimer, NQF go to exit disclaimer) served as the basis.  However, for some of the measures, such as having a regular doctor, an expert panel from within the NYC DOHMH and the Institute defined the appropriate measures.     

Determining data sources.  IFH went through a year long process to define the data sources for the measures and to decide on the most useful alerts to be included in the CDS to support data collection. Data generated from demographics, vitals, lab results, and other clinical information is often used to define measures.  Measures that are associated with a particular diagnosis (e.g., hypertension) often derive from national efforts and, in general, ICD-9-CM codes are used to define the inclusion criteria.  Similarly, for measures that involve both a particular diagnosis and a diagnostic test (e.g., diagnosis of coronary artery disease and at least one recorded LDL-C test), the inclusion criteria often includes a combination of ICD-9-CM or CPT codes.  Additional codes may also be used to establish the numerator or denominator of a measure often based on the definition established by national organizations (Measures List - Colorectal cancer screening, breast cancer screening).  Unless measures specifically noted that billing codes should be used, clinical data is used to generate reports to the extent possible.  

Collecting and analyzing data.  In conjunction with the measures, CDS tools were also designed to best support data collection and analysis.  IFH had a panel of experts who worked on this process to ensure that the language in the alerts was appropriate. As part of the initiative, at least two CDS tools were implemented for each TCNY objective. Some attempts at alerts failed and needed to be reworked.  During this process, IFH had to work to get clinician buy-in to the system to ensure that the tool would, in fact, be used.  In cases where a site was unable to successfully apply a clinical decision tool into its current workflow, the senior clinical team met with the site to redesign how the tool can best fit into the workflow. Additionally, the system monitors providers’ reactions to different alerts and aspects of the CDS, allowing for adjustments to be made if a particular alert is not being used.  For example, IFH previously employed a smoking alert that was found to be too wordy and was often bypassed by clinicians.  This alert was subsequently dropped.  

The Institute utilizes its data warehouse to generate reports on each measure.  The data in the warehouse comes from the EHR and a business intelligence application is used to create the reports.  The data in the data warehouse is downloaded daily from the EHR.  IFH generates and monitors reports monthly to capture the impact of CDS alerts on provider performance and clinical outcomes. Baseline data was captured initially for the measures before initiating the CDS alerts to determine their clinical effectiveness. A separate report is prepared to monitor clinician performance on CDS alert measures.     

Sharing results.  IFH regularly produces and disseminates reports to share the impact of the CDS system on clinical outcomes and provider adherence with clinical guidelines.  The QI team also seeks feedback from clinicians regarding the usability of CDS alerts and the appropriateness of measures.  If a particular alert does not work appropriately, the QI team consults with experts and tries to resolve the problem.  The Institute has multiple reports available to clinicians, including one that captures data by provider, giving each clinician a type of “report card”.

V. Health Center Network of New York

The Health Center Network of New York (HCNNY) utilizes an EMR and registry function to generate quality reports from each of their four implemented health centers, Open Door Family Medical Centers, Hudson River Healthcare, Whitney M. Young Jr. Health Center and Westside Health Services.  Additionally, HCNNY is currently in the process of developing a data warehouse for enhancing EMR data collection and reporting capabilities, and creating a chronic disease dashboard to further enhance reporting abilities.

Create a culture of quality.  HCNNY’s original QI program was built around improving clinical outcomes for their diabetic population.  They began by tracking and reporting on their diabetic population, but they are now identifying new populations and measures to track as well.  HCNNY looks at successful quality programs that are in place at the individual clinics, and then attempts to establish a protocol to implement the program on a network wide scale.

In addition to utilizing their clinical decision support (CDS) functionality within their EMR to integrate quality improvement into daily workflow, each of the four health centers has a QI Director and QI staff member in charge of clinical quality.  These efforts ensure that quality improvement is one of the Centers’ top priorities.   The data warehouse and chronic disease dashboard will further add to HCNNY’s culture of quality once fully implemented.

Selecting Measures. HCNNY uses many of the canned reports and the registry functionality included in their EMR.  The majority of the measures currently tracked and reported are based on HRSA's UDS measures.

Determine data sources.  The data used in HCNNY’s reports comes directly from the health centers’ EMRs at two centers, and from their emerging data warehouse at the remaining two centers. For example, diabetes patients are identified based on information entered into the EMR that is coded in ICD-9 (e.g., 250.xx).   Other information documented in the EMR in structured formats, such as gender, age, and clinical data, including vitals and lab results, also serve as data sources for numerators and denominators of the identified measures.  

Collect and analyze data.   The QI team works with each health center to educate staff about extracting data from their EMR and utilizing the registry function to generate reports on clinical quality.  Each of the four health centers in their network are  live on one single vendor EMR product, making it easier for each of the clinics to use the same reports.  HCNNY worked together with their vendor to troubleshoot and to ensure that staff understood the information that was contained in the canned reports.  In addition, they utilize the registry functionality of their EMR to identify sets of patients based on various parameters, such as demographics, vitals, labs, and diagnosis. The patient set can then be further filtered based on the above parameters.   Each site sends their reports, saved in a spreadsheet format to the network.  The data reported from each of the centers can then be aggregated using the spreadsheet application to provide a network-level view.   HCNNY expects to be able to aggregate their quality data on-demand from their data warehouse very soon, but continues to rely on the spreadsheet until all centers are populating the data warehouse.

The data is analyzed in a number of ways in order to compare the performance of the four health centers in the network.  The data is compared by site for each given report and an outcome progression analysis report is then run to see if each of the four sites are improving upon the measures over time.   Performance over time is measured from an initial pre-EMR baseline through previous year’s outcomes and all current year quarterly outcomes.  

Share results. Once the network-level analysis has been completed, the results are shared with the health centers.  Monitoring reports at regular intervals allow HCNNY to track progress at all health centers.   HCNNY’s QI team has regular meetings to look at outcomes and to discuss trends in the reported measures.  They discuss past interventions that have been the most successful. Each member of the QI team is able to share his/her perspective, leading to open conversations about the QI effort at HCNNY. Often new ideas about interventions training and analyses come out of these meetings.  An effort is underway to establish a formal Quality Collaborative program fashioned after HRSA’s Health Disparities Collaboratives of the past.

Developed by the Health Resources and Services Administration as a resource for health centers and other safety net and ambulatory care providers who are seeking to implement health IT.
Health Information Technology Toolboxes help health centers, safety net providers, and ambulatory care providers with electronic and online resources and technical assistance to improve patient care.  More>
Stay Informed