• Printer-friendly Performance Review Protocol (Acrobat/pdf)

The Health Resources and Services Administration (HRSA) is the principal Federal agency within the Department of Health and Human Services (DHHS) charged with increasing access to basic health services for the nation’s underserved, vulnerable, and special needs populations. With a budget of over $6 billion, HRSA provides direct financial assistance, in the form of grants or cooperative agreements, to over 3,000 organizations (i.e., grantees) for the purpose of carrying out 80 different programs that improve and expand access to quality health care and reduce disparities in health status.

To assure that all HRSA funded programs are accomplishing their intended purposes, HRSA continuously tracks and analyzes the performance of its grantees. In 1993, the Government Performance and Results Act (GPRA) reinforced these agency-wide efforts by requiring all Federal agencies to submit annual reports on program performance. Agency program performance reports describe the actual program performance achieved, compared with the performance goals expressed in the agency’s performance plan.

For HRSA, seven strategic goals serve as the organizing framework for the agency’s program performance measures:

1. Improve Access to Health Care
2. Improve Health Outcomes
3. Improve the Quality of Health Care
4. Eliminate Health Disparities
5. Improve the Public Health and Health Care Systems
6. Enhance the Ability of the Health Care System to Respond to Public Health Emergencies
7. Achieve Excellence in Management Practices

Role of the Office of Performance Review
The Office of Performance Review (OPR) plays a central role in achieving the HRSA mission and its program performance goals. With a headquarters office in Rockville, Maryland; ten regional divisions in Boston, New York, Philadelphia, Atlanta, Chicago, Dallas, Kansas City, Denver, San Francisco, and Seattle; as well as a sub-regional office in Puerto Rico, OPR serves as the agency’s focal point for reviewing and enhancing the performance of HRSA funded programs within communities and States. On a regularly scheduled basis, HRSA grantees are reviewed by a team from one of the ten OPR regional divisions.

Purpose of Performance Reviews
The purpose of performance reviews is to improve the performance of HRSA funded programs. All HRSA grant programs (including cooperative agreements) eligible for a review are designated by OPR as either a Category 1 or Category 2 grant program for the purpose of conducting a performance review.

• Category 1 includes: health services delivery activities and professional training programs with measurable data.
• Category 2 includes: grant programs funding professional training and public health infrastructure development that do not lend themselves to using the same approach used for Category 1 grant programs typically due to small numbers of recipients/clients, limited term funding, or the scope of the activity.

Through systematic pre-site and on-site analysis using the Performance Review Protocol, OPR works collaboratively with grantees and HRSA Bureaus/Offices to assess the performance of HRSA funded programs. The review of Category 1 grant programs measures program performance, analyzes the factors impacting performance, and identifies effective strategies and partnerships to improve program performance. The review of Category 2 grant programs analyzes the factors impacting the ability of the grantee to meet selected goals and objectives of the grant program and/or achieving sustainability beyond the end of HRSA support, and identifies effective strategies and partnerships to improve performance/progress on program goals and objectives. Performance reviews also provide direct feedback to the agency about the impact of HRSA policies on program implementation and performance within communities and States. From this analysis and feedback, OPR tracks key program performance issues, identifies effective practices and outstanding program outcomes, provides technical assistance, and, when appropriate, develops recommendations for changes to HRSA policies to further enhance the performance of HRSA funded programs.

Performance Review Protocol
The Performance Review Protocol is a performance improvement tool used with each grantee to review its HRSA funded programs. When a grantee receives more than one HRSA grant, the grantee’s HRSA funded programs selected for review are optimally reviewed during the same onsite visit. In cases where it is not possible to schedule reviews of all the grants during one onsite visit, separate reports of the individual reviews may be issued. In such cases, each of the performance review reports will provide standard background information on the grantee organization, a description of all HRSA grants received by the grantee, a description of the grant programs being reviewed, and specific information on the scope of the report.

For Category 1 grant programs, the four primary components of the protocol are:

• Performance Review Measures
• Performance Analysis and Key Area Discussion
• Performance Report
• Action Plan/Technical Assistance/Follow-up

For Category 2 grant programs, the four primary components of the protocol are:

• Goals and Objectives
• Discussion about Progress/Impact/Sustainability
• Performance Report
• Action Plan/Technical Assistance/Follow-up

I. Review of Category 1 Grant Programs

A. Performance Review Measures
Effective performance measures describe the population served by each program, the level and scope of program services or resources provided to this population, how well the program is providing those services or resources, and the impact of the services or resources on the population served. The most important types of performance measures for a program or set of programs are: (1) outcome measures (i.e., measures that track the impact or effectiveness of a program or set of programs); and (2) effort measures (i.e., measures that assess the grantee’s efforts to implement a program or set of programs).

Performance reviews begin with a pre-site review of established program performance measures (i.e., HRSA GPRA measures http://www.hrsa.gov/perplan, and HRSA program performance reporting requirements) and the grantee’s program goals and objectives as identified in their grant application(s). From this pool of established measures, the review team, in collaboration with the grantee and through consultation with the HRSA project officer(s) and other appropriate HRSA staff, selects a set of Performance Review Measures for each funded program. These measures define the scope and focus of the performance review.

Selection of Performance Review Measures

For each funded program:

• Does the grant have specific program measures?
• What are the grant’s goals and objectives?
• Who is the population served (e.g., patients, students, customers)?
• What are the specific services or resources provided to this population?
• From the pool of measures, what are the key outcome and effort measures?
  • Are these measures of outcomes and efforts communicated easily?
  • Do they measure the most important outcomes and efforts?
  • Is the data available and credible?

Once the Performance Review Measures are selected for each grant, the review team, in collaboration with the grantee and the HRSA project officer(s) and other appropriate HRSA staff, determines performance achieved on each measure. Progress is assessed by comparing actual performance with established HRSA program and/or grantee performance goals.

Progress on Performance Review Measures

• How is the grantee performing on each Performance Review Measure?
• What are the grantee’s performance trends for each measure?
• What is the gap from current to desired performance on each measure?

B. Performance Analysis and Key Area Discussion

After the Performance Review Measures have been selected and data has been collected and graphed, the Performance Analysis portion of the review examines the factors contributing to and restricting the grantee’s performance on the selected measures. The analysis and discussion for a grant includes reviewing the grantee’s internal systems and processes, external issues (including HRSA policies and environmental, political, demographic, and economic trends) that may contribute to or restrict the grantee’s performance on the Performance Review Measures, and the grantee’s successes or challenges in forming and sustaining successful performance partnerships.

Starting with a pre-site visit review of the grantee’s program applications, progress and data reports, and applicable HRSA program guidance/expectations (http://www.hrsa.gov/grants/preview), the review team, in collaboration with the grantee and through consultation with the HRSA project officer(s) and other appropriate HRSA staff, analyzes the following performance factors for each performance review measure through its pre-site and on-site work.

Performance Analysis and Key Area Discussion

• What are the key factors behind the grantee’s performance on each selected Performance Review Measure?
  
  • How do the grantee’s internal systems and processes contribute to or restrict the grantee’s performance on each measure? (See Appendix A)
  • What external issues, including HRSA policies, are influencing the grantee’s performance on each measure?
  • What are the grantee’s successes and challenges in forming and sustaining partnerships that support its performance on each measure?
    
• What is the order of priority among the factors examined – which are the most important to address to improve the grantee’s performance on each measure?
• What additional information about the grantee and its implementation of the grant program enhances an understanding of the challenges and opportunities the grantee faces in achieving its mission?
• What significant accomplishments and/or best/promising practices should be highlighted?

II. Review of Category 2 Grant Programs

A. Goals and Objectives

A performance review of a Category 2 grant program begins with a pre-site visit examination of a grantee’s goals and objectives as described in its application. From this set of goals and objectives, the review team in collaboration with the grantee and HRSA project officer(s) and other appropriate HRSA staff, selects a set of three to five significant goals and objectives on which to focus. The grantee’s goals and objectives should be aligned with HRSA grant program goals and objectives. If the grantee has sufficient and reliable data regarding their pursuit of a goal or objective, the review team should present it as an indicator of progress towards a goal or objective.

Identification of Goals and Objectives

For each funded program:

• What are HRSA’s stated goals and objectives for the grant program?
• What are the grantee’s stated goals and objectives in the grant application?
• Who is the population served (e.g., patients, students, consumers)?
• What are the specific services or resources provided to this population?

B. Discussion about Progress/Impact/Sustainability

After goals and objectives have been selected, the review team and the grantee conduct an on-site facilitated discussion regarding the progress the grantee has made in achieving the selected goals and objectives. Issues to consider are factors contributing and restricting attainment of the selected goals and objectives, the grantee’s impact on its target population/community, the structure and functioning of the grantee’s internal systems, and external issues (including HRSA policies, and environmental, political, demographic, and economic trends) which may affect performance. If applicable, the discussion should identify the grantee’s plans and efforts to sustain its project after support ends. The grantee also may share lessons learned in implementing its project and best or promising practices that may be beneficial to other similar grantees.

Performance Analysis and Key Area Discussion

• What are the key factors behind the grantee’s performance on selected goals and objectives?
  
  • How do the grantee’s internal systems and processes contribute to or restrict the grantee’s performance? (See Appendix A)
  • What external issues, including HRSA policies, influence the grantee’s performance?
  • What are the grantee’s successes and challenges in forming and sustaining partnerships that support its performance?
    
• What additional information about the grantee and its implementation of the grant program enhances an understanding of the challenges and opportunities the grantee faces in achieving its goals and objectives?
• What significant accomplishments and/or best/promising practices should be highlighted?

III. Performance Report

Upon completing the analysis of the factors impacting the grantee’s performance on each of the Performance Measures and/or its achievement of program goals and objectives, the review team, working on-site with the grantee, develops a preliminary set of Performance Improvement Options and Actions that the grantee can include in the Action Plan to improve performance. The review team assists the grantee in identifying evidence-based strategies, no-cost/low-cost options, as well as partners who may have a role to play in improving the grantee’s program performance.

After the on-site portion of the performance review is completed, the review team presents its findings, including any identified Program Requirement issues, in a draft Performance Report, and requests feedback on the report from the grantee, HRSA project officer(s), and other appropriate HRSA staff. The review team reviews the feedback received and develops the final Performance Report.

Performance Report

• Executive Summary describes the grantee organization, HRSA grants received and reviewed, key findings of the performance review, and special accomplishments (not to exceed 2 pages).
• Table of Contents lists six major sections and parts of Section IV, Grantee Performance
• Grantee Information provides grantee contact information.
• HRSA Grant Program Information identifies HRSA grants received.
• Grantee Overview describes the grantee organization; the HRSA grants received and reviewed, the scope of the report, population served, and the services/ resources provided by the HRSA funded program(s) reviewed.
• Performance Measures and/or Goals and Objectives present trend data on each of the selected Performance Review Measures and/or Program Goals and Objectives. A graph and trend analysis is required for review of Category 1 grants.
• Performance Analysis and Key Area Discussion summarizes the factors (including HRSA policies) contributing to and restricting the grantee’s performance with respect to each measure and/or impacting the grantee’s ability to achieve program goals and objectives. The discussion of the broader key areas that are not directly aligned to performance measures should also be captured in this section, as well as promising practices.
• Performance Improvement Options provides options to improve performance with respect to each measure and/or program goal or objective, and, when necessary, identifies technical assistance needs. Options include: evidence-based strategies, no-cost/low-cost options, as well as forming partnerships with those who may have a role to play in improving program performance.
• Program Requirements identifies, when necessary, a requirement of applicable program statutes, regulations, and/or grant award conditions.

IV. Action Plan/Technical Assistance/Follow-up

If the project officer was not part of the on-site review team, the on-site visit is followed by a call to debrief the project officer(s) and a subsequent three-way conference call among the review team, the project officer(s), and the grantee to discuss the development of an Action Plan. Within the Action Plan, the grantee is asked to describe the specific actions to be completed, and when necessary, to describe the specific actions to be completed to address any identified Program Requirement issues. The grantee submits the draft Action Plan, along with the grantee’s feedback on the draft Performance Report, to the review team. The review team and project officer(s) provide feedback on the alignment, specificity, feasibility and potential impact of the actions, and offer recommendations on any follow up technical assistance that may be appropriate. The grantee will finalize the Action Plan based upon this feedback and send the Action Plan to the review team and project officer. The review team will share the final Performance Report with the grantee and project officer. The performance review concludes with the adoption of an agreed upon Action Plan to be implemented and reviewed by the grantee and the project officer over time, as appropriate.

Action Plan/Technical Assistance Request

• Action Items describe the actions to be completed by the grantee.
• Key Steps specify where, how, who, and by when each action will be accomplished.
• Technical Assistance Request identifies, when necessary, technical assistance to support the grantee’s implementation of the action plan.

V. Sample Performance Review Timeline

Week 1-8 Prior to the Site Visit: Review of key grantee materials (i.e., grant application(s), progress reports, program guidance/expectations, and performance data submissions.), consultation with HRSA Project Officer(s) and other appropriate HRSA staff, and discussions with the grantee regarding measures or goals and objectives and factors impacting performance.

Week 9 Site Visit * (Note: Includes on-site performance analysis, key area discussions, and development of performance improvement options. Review Team advises grantee to begin drafting the Action Plan)

Week 13 Draft Performance Report sent to Grantee with request for feedback and an Action Plan. Feedback is also requested from HRSA Project Officer(s) and Other Appropriate HRSA Staff.

Week 15 The grantee sends feedback on the draft Performance Report and submits the draft Action Plan to the review team and the Project Officer.

Week 18 Final Performance Report with approval of Action Plan sent by OPR to grantee. Final Performance Report and Action Plan are distributed within HRSA for Follow-up Actions/Technical Assistance with grantee.

* The site visit is scheduled with the Grantee at least 12 weeks in advance.

For additional information on Performance Reviews, please visit:
http://www.hrsa.gov/performancereview.

APPENDIX A – Internal Grantee Systems and Processes

Below is a series of questions review teams should consider when exploring factors impacting performance and examining the broader implications of the grantee’s efforts on its overall performance.

Capacity and Systems

Structure, Function, and Capacity

• How does the grantee’s structure and capacity support the HRSA funded program, including subcontractor performance and financial arrangements?
• Is the management structure appropriate for program implementation and performance, including any governing board or planning coalitions? Are there members on the governing board or planning coalition who are representative of the population groups in the service area and are knowledgeable of their health care needs?
• How does management assure that the program is implemented in accordance with stated program requirements, including any grant award conditions?
• Is there integration and coordination of programs across the grantee?
• Are there any emerging issues that could impact on the grantee’s capacity and/or structure (e.g. demographic, economic, environmental shifts)?
  Internal Quality Improvement (QI) Program
• Has the grantee implemented a QI program? Throughout the organization?
• How has the grantee used this QI program to improve performance, increase access, and work to reduce health disparities? How does the QI program identify new and emerging issues that need to be addressed?
• How and from whom is feedback obtained and incorporated into the QI program?
• Has the grantee undergone outside, independent QI reviews (e.g., accreditation)? How have the results of these outside QI reviews been addressed?

Staff Performance Systems

• How are staff performance and productivity reviewed?
• What efforts are made to support staff to improve outcomes and eliminate health disparities (e.g., training in cultural competency, access to interpretation services, continuation education, etc.)?
• How are staff orientation and training assessed?
  Data Capacity and Systems
• How does the grantee collect and analyze data for tracking program performance (including subcontractor performance)?
• How does the grantee use their data and information to support program decision-making and innovation?
• Does the grantee have the capability to have changes made to their data system to respond to new and emerging changes to submission of required data?
• How does the grantee’s data capacity support the timely submission of reports required by HRSA and the grantee? Has the grantee invested in quality hardware and software as well as staff to meet their data needs?
• Does the grantee have the capacity to collect data by race, ethnicity, and primary language?
• Is there a process in place for building consumer trust to facilitate the collection of race/ethnic data?
  Clinical Systems (For Grantees Involved in Patient Care)
• Does the grantee provide the required and additional services necessary (including referral arrangements and after hours/weekend coverage) to achieve program goals and objectives?
• Does the grantee tailor services to meet the needs of its consumers, including those with limited reading skills, with diverse cultural and ethnic backgrounds, and who speak a primary language other than English? How does the grantee assure medical interpreter services to these consumers?
• How does the grantee assure that preventive and emerging clinical care guidelines are updated and followed
• What quality assurance systems are in place? What is the membership of their Quality Assurance Committee? What process does the QA Committee use in selecting clinical performance measures? How does the QA Committee address the identified clinical issues and deficiencies?
• Does the grantee have clinical tracking capability for referrals, ancillary and laboratory results and no-shows?
• How does the grantee assure patient confidentiality, informed consent, and Human Subjects/IRB protections where research is conducted? How has the grantee implemented Health Information Privacy Act (HIPAA) requirements?
• What systems are in place to provide for ongoing contact with patients after the face-to-face encounter to help them adhere to treatment plans, self-manage their medical conditions, and keep follow-up appointments?
• Are patient records signed and dated with progress notes from all providers who see the patient? Are patient records maintained with the ongoing listing of medications in an orderly chart, which indicates the name of the medication, dose, and date prescribed, and if applicable, date ended, for all medications?
• What are the grantee’s efforts around risk management? Does the grantee have written policies and procedures for risk management (e.g., security of medical records, pharmaceuticals and bio-hazardous materials)? How is staff informed of these policies and procedures as well as the potential risks? How are the grantee and staff implementing risk management procedures and policies?
• Has the grantee evaluated their information system’s capacity for automated billing, scheduling, and recall to produce data for program analysis purposes?
  Program Location and Facility
• Does the location and equipment support the implementation and performance of the intended program?
• If applicable, is the facility accessible to the population targeted for services?
• Is the facility’s environment welcoming to the consumer population (e.g., layout and design are not in conflict with cultural practices of client population, signage is displayed in appropriate languages, etc)?
• For programs that provide services of a sensitive nature, such as HIV/AIDs or sexually transmitted infections (STI), is patient confidentiality maintained internally by not indicating the services provided in signs or other information posted outside of the building?
• Does the facility location impact other entities serving similar populations?

Outreach and Consumer Feedback

Reaching the Target Population

• How does the grantee identify the needs of its target population?
• How does the grantee market its program to the target population?
• How does the grantee build trust among emerging populations?
• How does the grantee identify barriers to program services/resources and reduce/remove these barriers for the target population, including barriers for persons with disabilities, language and/or cultural differences?
• How does the grantee use information and data to track changes in the persons/populations being served by the program?
• Does the grantee have on staff, as paid workers or as volunteers, individuals who are consumers or would represent the consumer’s interest?
• How does the grantee assess and track changes in health care environment and marketplace? What approaches are used to address these changes?
  Use of Feedback How has the grantee demonstrated that they successfully implement programs that respond to the identified needs of the population served?
• How does the grantee solicit feedback (e.g., consumer satisfaction surveys) from those served by the program?
• How does the grantee use this feedback within its program?
• Is feedback provided to providers on their performance?
• What is the “feedback loop” between the grantee and those being served, as well as the larger community/population groups targeted for services?

Business and Financial Management

Program Budget Aligned with Goals and Objectives

• Does the budget reflect the priority program goals and objectives and is it aligned with the priority needs of the consumer population?
• Is the budget well justified, including any maintenance of effort and/or Federal matching fund requirements?
• How does the grantee demonstrate that fiscal resources are allocated properly to complete the program goals and objectives?
• Do expenses follow the program purpose and priorities?
  Financial Management
• Does the grantee have an adequate plan for the use of HRSA funds, including systems in place to reduce erroneous or improper payments?
• Does the grantee draw down HRSA funds from the Payment Management System (PMS) using a reasonably apportioned rate? How has the grantee performed in the oversight of funds and contracted services and in its overall fiduciary responsibilities? Are funds disbursed in a timely manner?
• Does the grantee have incentives and procedures to measure and achieve efficiencies and cost effectiveness in program execution? What results has the grantee achieved in effecting economies of scale and cost efficiencies in administrative functions?
• Does the grantee use strong financial management practices? Are financial systems maintained which provide internal controls (i.e., no co-mingling of funds), safeguard assets, maintain adequate cash flow, and maximize revenue from all sources? Are required audits conducted in a timely manner? Are employee payroll taxes paid on a timely basis?
• Has the grantee taken meaningful steps to address its identified financial management deficiencies, including any audit findings?
  Reimbursement for Patient Services (For Grantees Involved in Patient Care)
• What steps is the grantee taking to maximize available third party reimbursement? Is there a system in place for timely third-party billing?
• How does the grantee enroll patients in applicable insurance programs?
• Does the grantee have policy, structures, and procedures in place that enable staff to collect sensitive financial information from consumers?
• Does the grantee provide Medicaid/Medicare reimbursable services? Is the grantee a certified Medicaid/Medicare provider? Does the grantee appropriately negotiate with, bill, and collect funds from Medicaid/Medicare?
• Does the grantee have an established billing system in place, including a schedule of charges (caps) and sliding fee scale (discounts), where appropriate?
• How often does the grantee provide in-service training for employees with responsibility for coding and billing to assure that the latest information is appropriately utilized? Does the grantee have a system in place to "spot check" their coding efforts?
• Does the organization have adequate systems in place to operate effectively under managed care?
• How are other funding sources coordinated with the HRSA program, including participation in the Section 340B Drug Pricing Program?
• To what extent does the grantee collaborate with other organizations to address the needs of consumers?

Leadership and Staffing

Leadership and Strategic Priorities

• How does the grantee leadership team promote the vision, mission, values, and strategic priorities of the program internally and externally? Are other HRSA funded programs aware of the grantee’s mission and strategic priorities?
• How do consumers and employees provide input to the strategic program priorities?
• How does leadership address internal and external forces (market, community, and State) that may impact on performance?
• How does the grantee assure that strategic program priorities and emerging issues are addressed?
• What does the leadership see as their most effective practices and outstanding program outcomes? How do these practices/program outcomes compare to other like organizations and are they worthy of replication?
• How does the leadership draw upon the expertise of community leaders, stakeholders, and community constituency groups?
  Staff Qualifications, Characteristics, and Skills
• How does the grantee assure that staff possesses the appropriate skill sets, competencies, and credentials required to perform their assigned duties?
• What commitment has the grantee made to improving staff skills and competencies?
• How does the grantee assure that the staff is culturally competent and responsive to the needs of the population being served?
• Does the staff reflect the diversity (in terms of culture, race, ethnicity, and language spoken) of the population served?
• Is there sufficient staff in place, with appropriate training and credentials, to effectively provide the services the grantee has been funded to provide?

Workforce Stability

• Has there been frequent turnover in program leadership and key management positions?
• Has there been frequent turnover in program leadership?
• What are the staff turnover patterns? Does the grantee consistently and quickly fill vacant positions?
• How is staff turnover tracked and analyzed?
• Does the grantee have a current Recruitment and Retention Plan (development tools on the NHSC website) for all positions, especially clinical, and do they utilize available recruitment tools, i.e. 3RNet, Practice Sights, etc., to their maximum?