Dear Colleagues:

• Essential Characteristics of a Good Performance Review System
  
  A good performance review system should have the following characteristics:
  • Useful
  • Common sense
  • Simple
  • Uses plain language
  • Uses minimum paper
    
• Data-driven Decision Making
  Effective performance management systematically accesses and uses the best quantitative and qualitative data and data analysis available at every stage of decision making. For example, the ability to manage performance and, in particular, to achieve an impact for the population served by a program is inseparable from the ability to assess whether such an impact has been achieved. Progress on performance measures and/or goals and objectives, provide such an assessment – a “bottom line” – in managing for performance.
  
• Language Discipline
  Too often terms are used interchangeably and with a multiplicity of definitions in talking about performance. The Protocol recognizes the importance of the disciplined (clear, consistent, and jargon-free) use of language.
  
• Population-level Accountability vs. Program Performance Accountability
  In thinking about accountability, it is important to distinguish between accountability for achieving a condition of well being for (1) a whole population (e.g., all of the children in a county who are under six years old) and (2) the population or clients served by a program, agency or service system (e.g., the clients of a HRSA grantee). The significance of this distinction lies in the recognition that a single program, agency or even service system cannot take sole responsibility (or credit) for achieving a desired condition of well being for a whole population.
  
• Talk to Action
  Importantly, the Protocol gets from talk to action quickly.
  
  (1) Adapted from materials authored by Mark Friedman
  
• Ends to Means
  The Protocol starts with ends defined as either Performance Review Measures or selected Program Goals and Objectives and works backwards to means (Performance Analysis, Key Area Discussion, Performance Improvement Options, and Action Items). Starting with ends – asking what we want for whom at the outset – saves countless hours and reams of paper typically expended in most review processes. It also provides focus: we know why we are here. The progression of the Protocol from ends to means involves four basic steps (these steps mirror the four components of the Protocol) which should first be understood in general terms:
  
  Step 1. First, define what we want for whom (e.g., babies born healthy) in measurable terms when there are adequate data (e.g., % of prenatal clients who had babies with low birth weights) or what goal or objective is desired. In the cases where a measure is chosen, the data for the measure is graphed to show both historic and projected trends. Following a review of the performance trend, the question becomes one of means: how to change (or accelerate, if positive or the same) the trend line (i.e., how to “turn (improve) the curve”) of the trend line. In the cases where there is a dearth of measurable data or an inability to trend data, an examination of the grantee’s application should yield a set of three to five significant goals and objectives on which to focus the performance review.
  
  Step 2. To turn the curve for a specific performance measure or goal or objective, start by analyzing the performance of the grantee with respect to the particular performance measure. Identify the most important factors that must be addressed to improve the performance trend. This is also an opportunity to explore what other issues that may not be directly aligned with the performance measure but are important to relevant to the grantee’s overall performance and operations.
  
  Step 3. Once factors associated with performance have been identified, determine what works to address those factors, and what the available options are for actions that will turn the curve.
  
  

  Step 4. Finally, deciding on a specific action plan to turn the curve and improve performance.

• Broadening the Focus

  The protocol encourages the focus of the analysis be broadened beyond the individual performance review measure or goal or objective. This should include a discussion of “Key Areas” relating to grant implementation, best or promising practices, or the challenges and opportunities the grantee faces in achieving its mission.

• Collaboration

  Finally, before moving ahead with a detailed discussion of the four components of the Protocol, it is important to note that the Protocol – and the performance review – is designed to be conducted in collaboration. The Protocol is, from this perspective, a tool for collaboration: it provides a framework from which the review team, in collaboration with the HRSA project officer(s) and the grantee work together in selecting key Performance Review Measures or Goals and Objectives, analyzing the factors affecting performance, and developing options and actions to improve the grantee’s performance.

  More specifically, the Protocol requires systematically working through each of the four components, making performance data and analysis explicit, applying certain decision making criteria, and ultimately developing a set of actions that will improve performance. The Protocol both provides for efficient participation and garners the best thinking of those involved.

• Performance Review Approach

  The Performance Review Protocol is a performance improvement tool used with each grantee to review its HRSA funded programs. Due to the wide variability in the scope and purpose of HRSA grant programs, the Protocol provides for two approaches to accommodate these variances and best facilitate improvement for each type of grant program.

  All HRSA grant programs (including cooperative agreements) eligible for a review are designated by OPR as either a Category 1 or Category 2 grant program for the purpose of conducting a performance review. (See Matrix of HRSA Grant Programs for listing of grant programs by category).

  • Category 1 includes: health services delivery activities and professional training programs with measurable data.
  • Category 2 includes: grant programs funding professional training and public health infrastructure development that do not lend themselves to using the same approach used for Category 1 grant programs typically due to small numbers of recipients/clients, limited term funding, or the scope of the activity.

The review of Category 1 grant programs measures program performance, analyzes the factors impacting performance, and identifies effective strategies and partnerships to improve program performance. The review of Category 2 grant programs analyzes the factors impacting the ability of the grantee to meet selected goals and objectives of the grant program and/or achieving sustainability beyond the end of HRSA support, and identifies effective strategies and partnerships to improve performance/progress on program goals and objectives.

For Category 1 grant programs, the four primary components of the protocol are:

• Performance Review Measures
• Performance Analysis and Key Area Discussion
• Performance Report
• Action Plan/Technical Assistance/Follow-up

For Category 2 grant programs, the four primary components of the protocol are:

• Goals and Objectives
• Discussion about Progress/Impact/Sustainability
• Performance Report
• Action Plan/Technical Assistance/Follow-up

1. How much service (or resources) is delivered?
   
2. How well is it delivered? In the Protocol, these kinds of measures are called “effort measures.”
   
3. What impact (3) is produced? In the Protocol, these kinds of measures are called “outcome measures.

(3) The term “impact” is not meant to imply an “impact analysis,” which would involve isolating variables to scientifically verify a causal relationship between a specific activity and a corresponding result. Such impact can, at best, be inferred from the knowledge that certain activities occurred and certain results followed.

• What are the grantee’s goals and objectives for the activity?
  
  When selecting Performance Review Measures or a grantee’s Goals or Objectives for review, it is important to consider the goals and objectives and program priorities of the awarding HRSA program.
  
  
• Who is the population served (e.g., patients, students, customers)?
  
  For the purposes of selecting a Performance Review Measure, the “population served” will differ depending upon the program or major activity being measured. For example, if outreach activities are being measured for a health center (the term “outreach” is used here to mean activities designed to increase access to health care by underserved populations), then the “population served” will typically be the entire underserved population within the geographic area served by the program. If the outreach activities are for a certain type of care, such as a diabetes clinic, then the “population served” might be all underserved diabetics within the geographic area served by the clinic. (4)
  
  

  By contrast, if the delivery of health services is being measured, then the “population served” will typically be the clients actually receiving services and/or enrolled in a program (e.g., the diabetics enrolled in a diabetes clinic). In each case, if the program does not collect exact data on the number of people being served, then the number should be estimated based upon the best available data.
  
  
  
  
• What are the specific services or resources provided to this population?
  
  This question refers to the first type of performance measures described above. (“How much service or resources is delivered?”). It should, therefore, be answered in concrete, quantifiable terms, indicating not only what are the services or resources provided to the “population served”, but also how much of those services are provided. For example, a financial aid program for nursing students might provide 100 students annually a total of $200,000 in financial aid and 2 hours per student of financial aid counseling. Note that the description of services or resources provided will often include an indication of the number of clients to whom the services or resources are being provided.

Outcome Measures and Effort Measures

Performance Review Measures should, where such measures are available, consist of a combination of outcome measures and effort measures, for several reasons. First and foremost, not all performance measures are created equal. It is more important to measure impact (outcome measures) than it is to measure how well a service is delivered (effort measures), and it is more important to measure how well a service is delivered (effort measures) than it is to measure how much of a service is delivered. The HRSA Performance Plan not only states that outcomes are the most important performance measures at HRSA, it also states that HRSA will work to increase the use of such measures.

1. HRSA GPRA measures
2. HRSA program performance reporting requirements; and
3. The grantee’s own measures and data, identified as program (including business management) goals and objectives in grant applications and progress reports, and often connected to a grantee’s strategic plans, annual plans, and/or work plans.
4. Performance Review Measures required by several HRSA Bureaus and Offices

• Are these measures of outcomes and effort communicated easily? Would those who pay attention to your work (e.g., taxpayers, legislators, HRSA project officers, clients) understand what this measure means?

• Do they measure the most important outcomes and efforts? Is this a measure of something of central importance with respect to the program’s goals and objectives (e.g., the health of a baby at birth would be of central importance to a prenatal clinic)? Is this measure a good corollary for the many other measures that might also be used (i.e., would the trend represented by the data collected for this measure be similar to the trend represented by data collected for those other measures)?

• Are the data available and credible? To be credible, the data must be consistent and reliable. In addition, timeliness is necessary to track progress.

Criteria for Selecting Performance Review Measures Worksheet

A set of national performance review measures have been identified for use with several HRSA programs. For a listing of these programs as well as additional information on the measures, please visit http://www.hrsa.gov/performancereview.

Identifying Goals or Objectives for Category 2 Grant Programs

Similar to identifying Performance Measures for Category 1 Grant Programs, the identification of selected Goals or Objectives for the review of Category 2 Grant Programs should consider the communication power and the central importance of the goal or objective, i.e., by reviewing the grantee’s progress in attaining the stated goal or objective the public, the project officer, and stakeholders could generally determine the grantee’s success in implementing the approved grant program.

Graphing Performance Review Measures

Once the Performance Review Measures have been selected, the next step is to assess performance. The first step in assessing performance is to graph the data for a Performance Review Measure (5) .

In grant types where goals and objectives will be used in lieu of performance review measures, typically no data or very limited data will be available. Therefore, graphs will not be created.

Once the Performance Review Measures have been selected, the next step is to assess performance. The first step in assessing performance is to graph the data for a Performance Review Measure.

Performance Review Measure #1 Data Source: < >

The graph provides a concise, visual representation of a grantee’s performance on each selected Performance Review Measure. Each graph should include units of the measure on the vertical axis and units of time on the horizontal axis. The graph should also include a title and legend. When available and appropriate as comparisons, the graph may include the grantee’s goal/target, HRSA standards and/or national performance data. The graph should include at least three actual data points. Each graph should also provide the source of the Performance Review Measure data.

Assessing Performance

Once the data are graphed for a Performance Review Measure, performance may be assessed. Assessing performance means answering the following questions:

Performance Review Measures should be used first and foremost to assess the improvement of the grantee’s program against its own history. To this end, the graph should also include a trend line of the history of the program’s performance and, to the extent possible, a projection of future performance (i.e., the projected trend assuming there is no change in the grantee’s management of the program). These projections of future performance should be based on facts (i.e., past performance history) or informed opinions based on facts (i.e., similar grantee performance trends; etc.) – not just the grantee/review team’s specified goal.

Trend lines may also be used to compare the grantee’s performance with (1) a goal, (2) a standard in the field, or (3) the performance of other similar programs. These uses, however, should be approached with caution. Goals are often important for managing a program, but are not inherently a reliable reflection of a grantee’s performance potential; they are sometimes set unrealistically high or low based on reasons other than an objective assessment of a grantee’s potential performance. The reliability of standards also varies. Standards for effort measures are typically more well established and, therefore, more reliable than standards for outcome measures. Examples of standards for effort measures include: staffing ratios; application processing time; handicap accessibility; and waiting time. Standards for outcome measures are almost always more experimental, such as hospital recovery rates and morbidity/mortality rates.

Similarly, comparisons to other programs can lead to unfair and misleading conclusions when such comparisons are cases of “apples and oranges.” In the case of complex service systems, for example, the variables are usually so dynamic that it is hard to imagine a scenario in which the comparison of outcome measures between two such systems would, alone, be a reliable indication of program performance. Similar programs may have significantly different clients or environmental circumstances.

In the final stage of this component, the performance review answers the question “Is this acceptable?” The Performance Analysis that follows will either focus on improving performance if it is “not acceptable” or sustaining performance if it is “acceptable.” To the extent that a performance review can reach such a judgment about the grantee’s performance, the basis for that judgment, such as a comparison to a goal or standard should be stated clearly. By contrast, at this point in the Protocol, it is not necessary or even desirable to explain the underlying reasons for a grantee’s performance, which is the subject of the second component of the Protocol: Performance Analysis.

Performance Analysis

In order to develop a plan to improve performance, it is necessary to understand the factors that impact performance (i.e., those factors that contribute to and/or restrict performance with respect to each Performance Review Measure), answering the question:

In answering this question, the grantee, in particular, has the opportunity to step back from its day-to-day operations and consider the larger issues it must deal with to improve performance. This question also presents the grantee with the opportunity to explain to others (including funders) the nature of the challenges it faces.

It should be noted that this analysis is important whether or not a grantee’s performance has met expectations. The analysis will: (1) identify those contributing factors that should be preserved to sustain performance; (2) identify restricting factors that are hindering performance; (3) identify factors beyond the control of the grantee that might have influenced performance (e.g., economic trends); (4) identify those effective elements of the grantee’s program that may be useful to other grantees engaged in similar work; and (5) determine whether performance can still be improved (for example, if a goal was, in retrospect, too low).

The worksheet below is a “force field analysis,” which can be used to organize the factors according to their contributing and restricting influences on efforts to turn or improve the curve of the performance trend.

Force Field Analysis Worksheet

In analyzing performance, the performance review should consider the full range of factors that might influence performance. The following questions address different possible factors: