0
Articles |

Performance Measurement Through Audit, Feedback, and Profiling as Tools for Improving Clinical Care* FREE TO VIEW

Kevin B. Weiss, MD; Robin Wagner, RN, MHSA
Author and Funding Information

*From the Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, Chicago, IL.

Correspondence to: Kevin B. Weiss, MD, Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, 1653 West Congress Parkway, Chicago, IL 60612



Chest. 2000;118(2_suppl):53S-58S. doi:10.1378/chest.118.2_suppl.53S
Text Size: A A A
Published online

Clinical audits and practice profiling have become popular tools in the attempt to change physician behavior to improve quality of care. Unfortunately, the growing need for information on quality of care has often outpaced the development of standard, valid, and reliable approaches to using these tools. The studies of performance measurement published in the literature to date demonstrate varying impact on ability to improve clinical care; few are randomized controlled trials. While performance measurement has become a common practice, the science surrounding this field is still in its early stages of development; while it seems promising, it should be viewed as largely experimental.

Figures in this Article

Since the seminal work of Wennberg and Gittelsohn1in the 1970s, numerous studies have sought to understand the reasons for large geographic variations in the use of clinical services, as well as variations in patient outcomes.2For much of the 1980s and early 1990s, these studies have served to indicate that there might be important differences in the quality of medical care as a result of overuse, underuse, and/or misuse of medical interventions.34 In response, the health-care system has begun to explore ways to use this information to influence changes in provider behavior to improve care.

Use of clinical data to improve outcomes is not a new concept. For decades, hospitals have routinely held surgical morbidity and mortality reviews as a means by which to learn from experience. More recently, the focus has shifted to clinical data review across multiple settings, and among differing types of provider groups. This report presents an overview of the use of clinical audits/practice feedback, practice profiling/benchmarking, and regulatory oversight as tools for changing physician behavior to improve outcomes. Before exploring these different tools, it may be useful to review a couple of the basic concepts of performance measurement.

Individual case review provides different data from population-based reviews, and both types of review have their strengths and weaknesses. Individual case review is principally used to explore concerns that are associated with rare, but sentinel, events. An example would be the review of an incident in which a patient received mismatched blood products. This type of review is best suited for events that are infrequent but important enough to warrant the use of resources so as to minimize the chances of a similar error in the future. Yet, one may or may not be able to generalize from the knowledge gained from such a review, and the infrequent nature of such events makes it infeasible to examine multiple similar events. There is also little opportunity to use epidemiologic and statistical tools to assist in judging the degree of certainty of the findings from individual case reporting.

The alternative to individual case review is the population-based approach. Aggregated experience from multiple cases can provide insights to patterns of clinical behavior for more common conditions that affect many more patients. An example would be a measure of the proportion of patients within a particular health plan who received the flu vaccine. With population-based assessment, it is possible to use standard epidemiologic and statistical techniques to help assess the degree of certainty of the conclusions drawn from the observed clinical experiences.

Donabedian5 offered the concept that quality could be measured based on structure, process, and outcomes. Structure encompasses physical factors, such as buildings, as well as professional and institutional factors, such as the regulatory and financing environments in which health care is delivered. Process refers to the actions that health-care providers take in delivering medical care, such as performing examinations, ordering tests, and prescribing medications. Outcomes are the end result of the process interventions: the effects on the patient’s health and well being. While early attempts at measuring the quality of health care focused on the structure, much of the current focus relates to exploring clinical processes and outcomes.

Although patient outcomes are the ultimate judge of the quality of health care, there are several advantages to using process measures instead of outcome measures for purposes of performance evaluation. Most notably, it is much easier for physicians or other health-care providers to accept responsibility for their actions in providing care than to accept responsibility for their patients’ outcomes, because there are many factors affecting patient outcomes that are not directly under the control of providers. For example, while a provider might make a concerted effort to ensure that a patient has been offered the flu vaccine, the patient may choose not to take the vaccine and may subsequently develop influenza. In this situation, performance evaluation will produce very different results depending on whether it is the process (providing access to the vaccine) or the outcome (influenza) that is measured.

Process measures are also useful in evaluating the quality of care for common chronic conditions for which the ultimate outcomes may take years to determine, such as hypertension and stroke, or glycemic control and complications from diabetes. For these reasons, it is attractive to focus on using process measures rather than outcomes measures for performance measurement. However, it would seem that the best measure of health-care performance rests with patient outcomes, including physiologic status, health-related quality of life, and satisfaction with the health-care system.

A third central concept for performance measurement relates to how the data are used. Formative data are gathered for immediate use, to guide clinical decisions affecting ongoing patient care.6 As such, this type of information is different from the kind used for evaluation. Although evaluative data may be collected at any time in the process of care, they are generally examined retrospectively in an attempt to evaluate good vs bad quality health care, overuse vs underuse of services, or perhaps to compare one type of service to another. Currently, there are several major types of performance measurement in use. These include clinical audits/practice feedback, practice profiling/benchmarking, and regulatory oversight of performance indicator systems.

For decades, health-care systems have used clinical audits as a tool for quality assessment. Audits of this type usually seek to characterize care through the systematic review of a series of patient experiences. Most often, the information is obtained by examining charts or medical records for documentation of specific clinical practices/procedures. Since the 1970s, the British have used audits to examine issues of quality surrounding clinical management of minor acute problems or preventive health practices,78 chronic disease management (eg, diabetes,9 and asthma10), and the use of specialty consultations.11

While clinical audits are widely used to assess performance, there is conflicting evidence regarding whether or not they are effective in changing provider behavior. For example, a study at one hospital demonstrated significant improvements in preventive health processes that were audited vs other health-care processes that were not monitored.12Two small studies, examining the quality of Papanicolaou smears, demonstrated that performance of both residents and faculty physicians substantively improved after they received feedback from clinical audits.1314 By contrast, a study by Reilly and Patten15 demonstrated little change in targeted prescribing patterns for various clinical conditions as a result of audit and feedback.

The Ambulatory Care Medical Audit Demonstration Project16 is the largest formal study of the use of audit information in the United States. The project was designed as a randomized controlled clinical trial of the use of quality-improvement techniques to improve clinical performance in areas of primary care. Although audit information was only one element in a multidimensional intervention, this study demonstrated that it is possible to improve the quality of care with feedback of audit information.

Unfortunately, there has been no formal synthesis of studies on the use of audits to affect clinical performance. Many of the studies conducted to date were not well controlled and did not include a strategy for randomizing the physicians who were given feedback. Rather, most were preevaluation/postevaluation designs, based on interventions conducted at a single site or with a small number of practices. Therefore, while clinical audit with feedback is an attractive approach to changing physician behavior, its efficacy is unclear.

Another approach to performance measurement compares the performance of a single provider against that of a panel of similar providers. This type of measurement is often referred to as practice profiling or benchmarking.17In practice profiling the performance of a single physician or a group is expressed as a rate, a measure of resource use during a defined period for the population served. A profile is created by comparing this rate to that of a community norm based on the practices of other physicians or on other standards such as guidelines.18Figure 1 provides an example of the differences in the case mix-adjusted number of relative value units per hospital admission by physician specialty among physicians providing care to Medicare patients in Oregon vs Florida.19

As with audits, there is also little known about how profiling affects clinical performance. A meta-analysis of randomized trials of profiling revealed only 12 eligible trials; many of the studies under evaluation had notable design flaws.20 The analysis found that while profiling had a statistically significant positive effect on utilization, the effect was of minimal clinical importance.

Nevertheless, practice profiling is widely used and has attracted much attention and controversy.2123 In a 1994 American Medical Association survey of practicing physicians, over half of the respondents reported that they were subject to clinical or economic practice profiling.24Much of the controversy rests on the quality, validity, and reliability of the profiling data. For performance assessments to provide useful information, they must meet certain methodologic criteria. Among these is the need for well-defined, similar patient populations. It is important that practice data are adjusted for case mix severity and for other nonmedical care factors that are known to affect clinical performance, and that sufficient numbers of events are measured to ensure that differences are not due to chance alone.2526

This last issue is particularly problematic. A study by Hofer et al27 examined the usefulness of physician profiling for patients with diabetes, one of the most prevalent conditions in clinical practice. The authors conducted a study of approximately 3,600 patients with type II diabetes, under the care of 232 different physicians. Yet, as Figure 2 illustrates, they were unable to reliably detect any true differences in care among the physicians. They observed that “ … a physician would need to have more than 100 patients with diabetes in their panel for the profiles to have a reliability of 0.80 or better (while more than 90% of all primary care physicians in the health maintenance organization had fewer than 50 patients with diabetes).”,27

Therefore, the studies of physician profiling as a tool for changing practice behavior present a very mixed picture. The randomized, controlled trial literature suggests that profiling can produce a modest, but statistically significant effect on improving physician behavior.28 However, more recent studies on the validity and reliability of this measurement technique have opened up new questions about its usefulness.

With such increased interest in attempting to improve the quality of care through feedback of clinical data, it is perhaps no surprise that there have been efforts to create complex systems to evaluate clinical performance. The apparent premise behind such performance measurement systems is to use them as administrative tools, either voluntary or regulatory, to broadly measure quality-improvement activities.

In the United States, one of the prototypes of these systems is the Health Plan Employer Data Information Set (HEDIS). The HEDIS was developed in the early 1990s by the National Committee on Quality Assurance, a not-for-profit organization committed to evaluating and reporting the quality of care delivered by managed care plans. Using standardized methodology, HEDIS data are gathered from several sources within each health plan, including administrative claims and encounter information, medical records, and survey information. The National Committee on Quality Assurance, which uses the information from HEDIS as part of its accreditation program, makes the results publicly available through a national database of HEDIS information and accreditation results.29 Employers and consumers alike can use this information about quality of care to make choices among health plans. Figure 3 is an example of HEDIS data comparing mammography rates for Medicare managed-care organizations in Orange County, CA.,29

In addition to HEDIS, the United States has developed or proposed several other performance indicator systems.3033 Unlike HEDIS, the role and utility of these other systems have not yet fully evolved. Many of these newer systems do not clearly specify the primary audience for their information. Is the primary audience the provider, the health plan, the employer, or the consumer? Is the primary focus on process measures or outcomes measures? What are the costs and burdens of collecting such complex and comprehensive data? Perhaps most importantly, what effect will such data have on changing the quality of health care in the future?

What Is the Future for Using Performance Measurement To Evaluate Quality Improvement Activities?

Although not always approached with rigorous research methods, health-care performance measurement is now pervasive, and it seems likely that these activities will continue, if not increase, in the future. On the positive side, these efforts have helped to focus attention on the overall importance of evaluating the quality of health care; for the public, they have removed some of the mystery surrounding the delivery of care. As performance measurement continues to evolve nationally, clearer standards will emerge to define the types of measures that are most appropriate for this field, and valid and reliable methods will emerge for the collection, analysis, and reporting of data.

Alternatively, there are many limitations to this evolving practice of performance measurement that, if not adequately addressed, will undermine its long-term credibility. There are still many unanswered questions, such as the appropriate population size to study and the types of data adjustments (eg, case mix, severity, sociodemographic) that need to be applied in order to be able to make accurate comparisons. Also, the literature has yet to determine which clinical conditions or administrative issues benefit the most from these types of data collection and feedback, and which methods work best to produce positive changes in the delivery of care.

Performance measurement appears to be most useful when it is used as a formative tool as part of a more complex set of quality-improvement activities.34 However, this field has yet to determine which types of quality-improvement efforts will lead to better care. Until such questions are adequately addressed, performance measurement should be considered to be still in the “experimental stage” in the challenge toward reducing unintended variations in the quality of health care.

Abbreviation: HEDIS = Health Plan Employer Data Information Set

Figure Jump LinkFigure 1. Case mix-adjusted number of relative value units (RVUs) per hospital admission in Florida and Oregon according to medical specialty.Grahic Jump Location
Figure Jump LinkFigure 2. Comparison of visit rate profiles for 232 physicians in three health plans, showing the relative visit rate by physician (with 1.0 being an average profile) after adjustment for patient demographics and detailed case mix measures. The error bars represent a 1.4 SE confidence interval, so that overlapping confidence intervals suggest that the overall difference between two physician visit rates is not statistically significant (p > 0.05). In this graph, although the overall physician effect on visits is statistically significant, it is not possible to say that the physicians at the extremes are significantly different in their visit rates from any of the other physicians.Grahic Jump Location
Figure Jump LinkFigure 3. Medicare compare graph showing mammography rates for Medicare managed-care organizations in Orange County, CA.Grahic Jump Location
Wennberg, J, Gittelsohn, A (1973) Small area variations in health care delivery.Science182,1102-1108
 
Detsky, AS Regional variation in medical care.N Engl J Med1995;333,589-590
 
Wennberg, DE Variation in the delivery of health care: the stakes are high.Ann Intern Med1998;128,866-868
 
Brook, RH Appropriate and necessary.Health Manage Q1991;13,8-11
 
Donabedian, A Basic approaches to assessment: structure, process and outcome.The definition of quality and approaches to its assessment: explorations in quality assessment and monitoring (vol 1)1980,77-128 Health Administration Press. Ann Arbor, MI:
 
Goodman, RM, Wandersman, A FORECAST: a formative approach to evaluating community coalitions and community-based initiatives.J Commun Psychol1994;special issue,6-25
 
Ryan, MP, Buchan, IC, Buckley, EG Medical audit: a preliminary report from general practice.J R Coll Gen Pract1979;29,719-722
 
LeBaron, CW, Chaney, M, Baughman, AL, et al Impact of measurement and feedback on vaccination coverage in public clinics, 1988–1994.JAMA1997;277,631-635
 
Doney, BJ An audit of the care of diabetics in a group practice.J R Coll Gen Pract1976;26,734-742
 
Hewer, SM, Hambleton, G, McKenzie, S, et al Asthma audit: a multicentre pilot study.Arch Dis Child1994;71,167-169
 
Verby, JE, Holden, P, Davis, RH Peer review of consultations in primary care: the use of audiovisual recordings.BMJ1979;1,1686-1688
 
Holmboe, E, Scranton, R, Sumption, K, et al Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines.Acad Med1998;73,901-903
 
Curtis, P, Skinner, B, Varenholt, J, et al Papanicolaou smear quality assurance: providing feedback to physicians.J Fam Pract1993;36,309-312
 
Norton, PF, Shaw, PA, Murray, MA Quality improvement in family practice: program for Pap smears.Can Fam Physician1997;43,503-508
 
Reilly, PM, Patten, MP An audit of prescribing by peer review.J R Coll Gen Pract1978;28,525-530
 
Palmer, RH, Hargraves, JL The Ambulatory Care Medical Audit Demonstration Project: research design.Med Care1996;34,S12-S28
 
Lasker, RD, Shapiro, DW, Tucker, AM Realizing the potential of practice pattern profiling.Inquiry1992;29,287-297
 
Lee, PR, Shapiro, DW, Lasker, RD, et al Managed care: provider profiling.J Insur Med1992;24,179-181
 
Welch, HG, Miller, ME, Welch, WP Physician profiling: an analysis of inpatient practice patterns in Florida and Oregon.N Engl J Med1994;330,607-612
 
Balas, EA, Boren, SA, Brown, GD, et al Effect of physician profiling on utilization: meta-analysis of randomized clinical trials.J Gen Intern Med1996;11,584-590
 
Krakauer, H, Bailery, RC, Skellan, KJ, et al Evaluation of the HCFA model for the analysis of mortality following hospitalization.Health Serv Res1992;27,317-335
 
Green, J, Wintfeld, N How accurate are hospital discharge data for evaluation effectiveness of care?Med Care1993;31,719-731
 
Kassirer, JP The use and abuse of practice profiles.N Engl J Med1994;330,634-636
 
Emmons, DW, Wozniak, GD Profiles and feedback: who measures physician performance? Gonzalez, ML eds.Socioeconomic characteristics of medical practice1994,11-16 American Medical Association. Chicago, IL:
 
Physician Payment Review Commission Conference on Profiling Washington, DC: Physician Payment Review Commission, May 1992; Publication No. 92–2.
 
Orav, EJ, Wright, EA, Palmer, RH, et al Issues of variability and bias affecting multisite measurement of quality of care.Med Care1996;34,SS87-SS101
 
Hofer, TP, Hayward, RA, Greenfield, S, et al The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease.JAMA1999;281,2098-2105
 
Kim, CS, Kristopaitis, RJ, Stone, E, et al Physician education and report cards: do they make the grade? Results from a randomized controlled trial.Am J Med1999;107,556-560
 
HEDIS 2000 narrative: what’s in it and why it matters. Washington, DC: National Committee for Quality Assurance, 1999; Publication No.10234–100-00.
 
Palmer, RH, Peterson, LE Development and testing of performance measures for quality improvement in clinical preventive services.Am J Prev Med1995;11,402-406
 
Lawtheres, AB, Palmer, RH, Edwards, FE, et al Developing and evaluating measures of ambulatory care quality: a preliminary report of the DEMPAY Project.J Qual Improv1993;19,552-565
 
Lansky, D Measuring what matters to the public.Health Aff1998;17,40-41
 
Braun, BI, Koss, RG, Loeb, JM Integrating performance measure data into the Joint Commission accreditation process.Eval Health Prof1999;22,283-297
 
Thompson, RS Systems approach and the delivery of health services.JAMA1997;277,668-671
 

Figures

Figure Jump LinkFigure 1. Case mix-adjusted number of relative value units (RVUs) per hospital admission in Florida and Oregon according to medical specialty.Grahic Jump Location
Figure Jump LinkFigure 2. Comparison of visit rate profiles for 232 physicians in three health plans, showing the relative visit rate by physician (with 1.0 being an average profile) after adjustment for patient demographics and detailed case mix measures. The error bars represent a 1.4 SE confidence interval, so that overlapping confidence intervals suggest that the overall difference between two physician visit rates is not statistically significant (p > 0.05). In this graph, although the overall physician effect on visits is statistically significant, it is not possible to say that the physicians at the extremes are significantly different in their visit rates from any of the other physicians.Grahic Jump Location
Figure Jump LinkFigure 3. Medicare compare graph showing mammography rates for Medicare managed-care organizations in Orange County, CA.Grahic Jump Location

Tables

References

Wennberg, J, Gittelsohn, A (1973) Small area variations in health care delivery.Science182,1102-1108
 
Detsky, AS Regional variation in medical care.N Engl J Med1995;333,589-590
 
Wennberg, DE Variation in the delivery of health care: the stakes are high.Ann Intern Med1998;128,866-868
 
Brook, RH Appropriate and necessary.Health Manage Q1991;13,8-11
 
Donabedian, A Basic approaches to assessment: structure, process and outcome.The definition of quality and approaches to its assessment: explorations in quality assessment and monitoring (vol 1)1980,77-128 Health Administration Press. Ann Arbor, MI:
 
Goodman, RM, Wandersman, A FORECAST: a formative approach to evaluating community coalitions and community-based initiatives.J Commun Psychol1994;special issue,6-25
 
Ryan, MP, Buchan, IC, Buckley, EG Medical audit: a preliminary report from general practice.J R Coll Gen Pract1979;29,719-722
 
LeBaron, CW, Chaney, M, Baughman, AL, et al Impact of measurement and feedback on vaccination coverage in public clinics, 1988–1994.JAMA1997;277,631-635
 
Doney, BJ An audit of the care of diabetics in a group practice.J R Coll Gen Pract1976;26,734-742
 
Hewer, SM, Hambleton, G, McKenzie, S, et al Asthma audit: a multicentre pilot study.Arch Dis Child1994;71,167-169
 
Verby, JE, Holden, P, Davis, RH Peer review of consultations in primary care: the use of audiovisual recordings.BMJ1979;1,1686-1688
 
Holmboe, E, Scranton, R, Sumption, K, et al Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines.Acad Med1998;73,901-903
 
Curtis, P, Skinner, B, Varenholt, J, et al Papanicolaou smear quality assurance: providing feedback to physicians.J Fam Pract1993;36,309-312
 
Norton, PF, Shaw, PA, Murray, MA Quality improvement in family practice: program for Pap smears.Can Fam Physician1997;43,503-508
 
Reilly, PM, Patten, MP An audit of prescribing by peer review.J R Coll Gen Pract1978;28,525-530
 
Palmer, RH, Hargraves, JL The Ambulatory Care Medical Audit Demonstration Project: research design.Med Care1996;34,S12-S28
 
Lasker, RD, Shapiro, DW, Tucker, AM Realizing the potential of practice pattern profiling.Inquiry1992;29,287-297
 
Lee, PR, Shapiro, DW, Lasker, RD, et al Managed care: provider profiling.J Insur Med1992;24,179-181
 
Welch, HG, Miller, ME, Welch, WP Physician profiling: an analysis of inpatient practice patterns in Florida and Oregon.N Engl J Med1994;330,607-612
 
Balas, EA, Boren, SA, Brown, GD, et al Effect of physician profiling on utilization: meta-analysis of randomized clinical trials.J Gen Intern Med1996;11,584-590
 
Krakauer, H, Bailery, RC, Skellan, KJ, et al Evaluation of the HCFA model for the analysis of mortality following hospitalization.Health Serv Res1992;27,317-335
 
Green, J, Wintfeld, N How accurate are hospital discharge data for evaluation effectiveness of care?Med Care1993;31,719-731
 
Kassirer, JP The use and abuse of practice profiles.N Engl J Med1994;330,634-636
 
Emmons, DW, Wozniak, GD Profiles and feedback: who measures physician performance? Gonzalez, ML eds.Socioeconomic characteristics of medical practice1994,11-16 American Medical Association. Chicago, IL:
 
Physician Payment Review Commission Conference on Profiling Washington, DC: Physician Payment Review Commission, May 1992; Publication No. 92–2.
 
Orav, EJ, Wright, EA, Palmer, RH, et al Issues of variability and bias affecting multisite measurement of quality of care.Med Care1996;34,SS87-SS101
 
Hofer, TP, Hayward, RA, Greenfield, S, et al The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease.JAMA1999;281,2098-2105
 
Kim, CS, Kristopaitis, RJ, Stone, E, et al Physician education and report cards: do they make the grade? Results from a randomized controlled trial.Am J Med1999;107,556-560
 
HEDIS 2000 narrative: what’s in it and why it matters. Washington, DC: National Committee for Quality Assurance, 1999; Publication No.10234–100-00.
 
Palmer, RH, Peterson, LE Development and testing of performance measures for quality improvement in clinical preventive services.Am J Prev Med1995;11,402-406
 
Lawtheres, AB, Palmer, RH, Edwards, FE, et al Developing and evaluating measures of ambulatory care quality: a preliminary report of the DEMPAY Project.J Qual Improv1993;19,552-565
 
Lansky, D Measuring what matters to the public.Health Aff1998;17,40-41
 
Braun, BI, Koss, RG, Loeb, JM Integrating performance measure data into the Joint Commission accreditation process.Eval Health Prof1999;22,283-297
 
Thompson, RS Systems approach and the delivery of health services.JAMA1997;277,668-671
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Find Similar Articles
CHEST Journal Articles
PubMed Articles
  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543