0
Original Research: Lung Cancer |

Computerized Triggers of Big Data to Detect Delays in Follow-up of Chest Imaging Results OPEN ACCESS

Daniel R. Murphy, MD, MBA; Ashley N.D. Meyer, PhD; Viraj Bhise, MBBS; Elise Russo, MPH; Dean F. Sittig, PhD; Li Wei, MS; Louis Wu, PA; Hardeep Singh, MD, MPH
Author and Funding Information

FUNDING/SUPPORT: This project was funded by a Veteran Affairs Health Services Research and Development Collaborative Research to Enhance and Advance Transformation and Excellence grant [CRE-12-033] and partially funded by the Houston Veteran Affairs Health Services Research and Development Center for Innovations in Quality, Effectiveness and Safety [CIN 13–413]. Dr Murphy is additionally funded by an Agency for Healthcare Research and Quality Mentored Career Development Award [K08-HS022901], and Dr Singh is additionally supported by the VA Health Services Research and Development Service [CRE 12-033; Presidential Early Career Award for Scientists and Engineers USA 14-274], the VA National Center for Patient Safety, and the Agency for Healthcare Research and Quality [R01HS022087].

aHouston VA Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, TX

bDepartment of Medicine, Baylor College of Medicine, Houston, TX

cUniversity of Texas Health Science Center at Houston’s School of Biomedical Informatics, Houston, TX

dUT–Memorial Hermann Center for Healthcare Quality & Safety, Houston, TX

CORRESPONDENCE TO: Daniel R. Murphy, MD, MBA, Michael E. DeBakey Veterans Affairs Medical Center, Houston Center for Innovations in Quality, Effectiveness and Safety (152), 2002 Holcombe Blvd, Houston, TX 77030


Copyright 2016, . All Rights Reserved.


Chest. 2016;150(3):613-620. doi:10.1016/j.chest.2016.05.001
Text Size: A A A
Published online

Background  A “trigger” algorithm was used to identify delays in follow-up of abnormal chest imaging results in a large national clinical data warehouse of electronic health record (EHR) data.

Methods  We applied a trigger in a repository hosting EHR data from all Department of Veterans Affairs health-care facilities and analyzed data from seven facilities. Using literature reviews and expert input, we refined previously developed trigger criteria designed to identify patients potentially experiencing delays in diagnostic evaluation of chest imaging flagged as “suspicious for malignancy.” The trigger then excluded patients in whom further evaluation was unnecessary (eg, those with terminal illnesses or with already completed biopsies). The criteria were programmed into a computerized algorithm. Reviewers examined a random sample of trigger-positive (ie, patients with trigger-identified delay) and trigger-negative (ie, patients with an abnormal imaging result but no delay) records and confirmed the presence or absence of delay or need for additional tracking (eg, repeat imaging in 6 months). Analysis included calculating the trigger’s diagnostic performance (ie, positive predictive value, negative predictive value, sensitivity, specificity).

Results  On application to 208,633 patients seen between January 1, 2012, and December 31, 2012, a total of 40,218 chest imaging tests were performed; 1,847 of the results were suspicious for malignancy, and 655 (35%) were trigger-positive. Review of 400 randomly selected trigger-positive patients found 158 (40%) with confirmed delays and 84 (21%) requiring additional tracking (positive predictive value, 61% [95% CI, 55.5-65.3]). Review of 100 trigger-negative patients identified 97 without delay (negative predictive value, 97%; [95% CI, 90.8-99.2]). Sensitivity and specificity were 99% (95% CI, 96.2-99.7) and 38% (95% CI, 32.1-44.3), respectively.

Conclusions  Application of triggers on “big” EHR data may aid in identifying patients experiencing delays in diagnostic evaluation of chest imaging results suspicious for malignancy.

Figures in this Article

Wider use of electronic health records (EHRs) has created vast amounts of digitally stored data to track a patient’s journey through the continuum of health care. Although other industries have rapidly advanced methods to analyze “big data,” mining of health-care data has been slow, and few, if any, efforts have translated this information into improving patient care. Better use of large clinical data repositories could provide useful information on missed opportunities in patient care at the population level and create knowledge to improve care beyond the traditional patient–provider encounter.

Follow-up of abnormal imaging test results remains a problem despite communication facilitated by the EHR.,,,,,,,, For example, despite the presence of an “inbox” in most EHRs where providers receive electronic messages about abnormal test results, we found that patients did not receive appropriate and timely follow-up of abnormal imaging results, such as lung nodules or masses identified by a radiologist,, in nearly 8% of abnormal imaging results. These “missed findings” have been associated with increased malpractice litigation and poorer patient outcomes.,,,, We also found that 38% of patients with a diagnosis of lung cancer had missed opportunities in follow-up of their imaging (chest radiograph or CT scan) test results. Among other factors, information overload from EHR data,, and lack of resilient communication processes of important but not immediately life-threatening results,, can contribute to these preventable delays in follow-up of important information. A recent Institute of Medicine report (“Improving Diagnosis in Health Care”) suggests new measurement approaches, including the use of health information technology, to identify and reduce these delays.

Triggers offer one method to use EHR data to prevent and mitigate the impact of delays in care related to missed test results, and preliminary research by our team and other investigators has shown promise in achieving this goal.,,, These triggers consist of computerized algorithms that can scan hundreds of thousands of patient records to flag those with clues suggestive of patient safety events. The goal of running such triggers is to identify delays in care and provide this information to clinicians or other quality and safety personnel to take action to mitigate patient harm or prevent similar events in the future,,, in a more efficient manner than what is feasible through nonselective manual chart review alone.

Building on our pilot research, the goal of the present study was to test the application of a trigger to big data as a first step in creating a large-scale surveillance system to identify and enable action on delays in care. The Department of Veterans Affairs (VA) has successfully developed a large national database that contains clinical data on inpatient, outpatient, mental health, rehab, and long-term care services collected from all 144 VA facilities serving > 6 million veterans. This database provides an opportunity to develop and test the performance (eg, sensitivity, specificity, positive predictive value [PPV]) of the triggers to identify delays in follow-up of abnormal lung imaging results in a large dataset.

Setting

We developed and tested triggers in the VA's national database of EHR data with a specific focus on patients seen at a large VA network of seven hospitals and associated clinics in the Midwestern United States. The VA primarily serves a male population (92%) aged ≥ 18 years. Both the Baylor College of Medicine institutional review board and the VA Research Office approved this study (H-30995).

Trigger Refinement and Application

In previous research, we developed a basic trigger algorithm using literature reviews, input from specialists and primary care providers, and information about existing lung mass and nodule follow-up processes.,, During interpretation of imaging results at all VA facilities, radiologists electronically assign numerical codes for certain high-risk findings. One such code is a “suspicious for malignancy” code applied to imaging results when radiologists believe there is a reasonable likelihood that a malignancy exists or the suspicion deserves follow-up action. Our basic trigger identified these numerical codes as “red flags.” Of these, the trigger subsequently excluded patients in whom follow-up evaluation was unnecessary, such as patients with terminal illnesses or for whom follow-up action had already occurred. To develop the computer algorithm, we converted each criterion into a set of International Classification of Diseases, Ninth Revision, codes; Current Procedural Terminology codes; and numerical “suspicious for malignancy” codes. Because no standard definition of a delay exists for timely follow-up of chest imaging, 30 days was chosen to complete follow-up action. This interval would allow sufficient time for clinicians to follow up without significant progression of disease. We then programmed these factors into a structured query language search algorithm and applied the trigger to the national EHR data warehouse.

Several additional changes were made to the trigger compared with our previous research. First, we accounted for differences in use of structured data fields across multiple facilities (eg, different sites referred to “suspicious for malignancy” results using different numerical codes, although, clinically, these were identical). Second, changes were made to increase the sensitivity and specificity of triggers based on what we learned from preliminary research, including lowering age to >18 years, standardizing the list of excluded terminal illnesses to encompass all diseases with a 5-year survival rate < 50%, and adding multidisciplinary tumor board discussions as expected follow-up. Expert clinicians in primary care, pulmonology, and oncology then reviewed and approved criteria before application.

Each criterion was initially evaluated individually by iteratively performing 20 record reviews (10 records meeting the criteria and 10 that did not). We then made modifications to the program code to extract data more accurately and reviewed additional records. Once we validated all the criteria individually (Table 1), the full trigger algorithm was applied to a 1-year set of data in the warehouse. Figure 1 displays the technical steps to operationalizing the trigger. Pilot reviews were conducted to test the record review instrument, and two clinician reviewers were rigorously trained. We discussed cases within the research team to develop consensus and to resolve areas of disagreement and ambiguity, made refinements to the instrument to clarify when needed, and iteratively reviewed 10 additional records after each modification. Overall, 310 charts were reviewed during efforts to adapt the trigger algorithm to function on the data warehouse, including 260 reviews of individual criteria and 50 reviews to pilot test the complete algorithm and chart review form. Once 80% interrater reliability was reached between the two reviewers, triggers were applied to a separate validation cohort to evaluate performance.

Table Graphic Jump Location
Table 1 Trigger Algorithm Logic
Figure 1
Figure Jump LinkFigure 1 Technical steps to operationalize trigger.Grahic Jump Location
Testing Trigger Performance

The final trigger algorithm was applied to all patient records in which a chest radiograph or CT scan was performed during the 1-year study (“the validation cohort”). Clinician reviewers, blinded to the trigger status, manually reviewed a randomly selected sample of 400 trigger-positive and 100 trigger-negative records (ie, patients with abnormal imaging results with no evidence of delays detected by the trigger) and classified whether each record truly experienced a delay. At times, we found inaction within 30 days because of patient factors, nodule follow-up guidelines, or radiologists’ recommendations that necessitated longer follow-up periods. In these situations, reviewers identified whether there was a clinician-documented plan to follow up on the imaging at a later date. If so, these records were considered as a separate category that needed tracking. We then reviewed documentation up to 30 days beyond the clinician-documented follow-up date to confirm if follow-up occurred or whether a different course of action was documented. Each reviewer evaluated 240 trigger-positive and 60 trigger-negative records, such that there was a 10% overlap to enable calculation of interrater reliability.

Statistical Analysis

The study was powered to determine the number of records needed to identify a two-sided 95% CI with a width of 10% at any PPV. Because the largest sample size occurs at 0.5 when using a binomial distribution, a target PPV of 50% was used for PPV calculations, and the need to review at least 384 trigger-positive records was identified. Thus, at any given PPV, (eg, 65%), we would be able to construct a CI of ±5% around the obtained point value (eg, 60%-70%) and be 95% confident that the true PPV would be contained in the interval. An identical calculation was used to determine the negative predictive value (NPV). However, a conservative point estimate of 95% was used for calculation of the NPV because the detection of follow-up by the trigger during pilot reviews achieved nearly perfect accuracy. This calculation yielded a sample size of at least 73 trigger-negative charts, which was rounded to 100 to allow a margin of error.

Analysis included calculating the trigger’s diagnostic performance (ie, PPV, NPV, sensitivity, specificity). PPV was the percentage of the 400 trigger-positive records with delays in diagnostic evaluation, whereas NPV was the percentage of the 100 trigger-negative patients reviewed who truly did not have delays in diagnostic evaluation.

SPSS version 22 (IBM SPSS Statistics, IBM Corporation) was used to analyze trigger performance, time to follow-up, and reasons for lack of follow-up; results were reported by using descriptive statistics. Breakdown of patients with and without follow-up across tracked vs delayed cases was compared by using a Fisher exact test.

Trigger Performance

The trigger was applied to 208,633 patients seen at the seven sites between January 1, 2012, and December 31, 2012, and we identified 40,218 chest imaging tests performed. A total of 1,847 results were flagged by radiologists with a numerical “suspicious for malignancy” code; 655 (35%) were trigger-positive after clinical and expected follow-up exclusions (Fig 2). Of these, 400 records were randomly selected for manual review, and 242 (PPV, 61% [95% CI, 55.5-65.3]) records were identified for which follow-up diagnostic evaluation was not performed within 30 days. Of the randomly selected 100 trigger-negative records (suspicious imaging results but follow-up detected or not needed), 97 were found to truly not require follow-up (NPV, 97% [95% CI, 90.8-99.2]). Sensitivity and specificity of the trigger were 99% (95% CI, 96.2-99.7) and 38% (95% CI, 32.1-44.3]), respectively. Of the 10% of records studied by both reviewers, interrater reliability achieved a kappa of 0.74, indicating good agreement.

Figure 2
Figure Jump LinkFigure 2 Trigger validation process flow.Grahic Jump Location
Follow-up Outcomes

Of 242 high-risk records in which no follow-up was detected, 158 (65%) had no documented plan in response to the abnormal result. Table 2 lists the provider, patient, and system factors contributing to delays identified during chart reviews. In the remaining 84 (35%) records in which a documented plan for future action was identified (ie, needed tracking), 22 patients (26%) did not receive the expected follow-up at 30 days after the clinician-documented date nor was a change in follow-up plan documented. The remaining 62 patients (73%) underwent the expected diagnostic evaluation within the planned time frame. Within 2 years of the abnormal imaging study, 93% (78 of 84) of trigger-positive patients with a documented plan on record review received follow-up action, whereas only 79% (125 of 158) of the patients without a documented plan received follow-up action (P < .01). Median time to action for those who received follow-up was 107 days and was not dependent on whether a plan was documented (P = .63).

Table Graphic Jump Location
Table 2 Reasons and Contributory Factors for Delays

Among the 158 records that were falsely flagged positive by the trigger, the most common cause was that the suspicious finding was not located within the lungs, such as a mediastinal or chest wall mass (51%). Although these findings require follow-up, expected actions for nonlung malignancies are different and could not be accounted for by our trigger, thus serving as a topic for future research. Additional reasons for false-positive findings are displayed in Table 3 and include the inability of the trigger to detect certain evidence of follow-up due to missing or incorrectly coded data in the data warehouse (20%) or when patients received follow-up outside the VA system (8%).

Table Graphic Jump Location
Table 3 Reasons for False-Positive Results

VA = Department of Veterans Affairs.

Reviewers identified three false-negative records. One occurred when the trigger detected a completed tumor board consultation, but this consultation involved discussion of resectable bladder cancer, whereas the patient’s lung findings were not mentioned. Another false-negative record resulted when a chest radiograph was ordered to evaluate cough in a patient with nodules seen on a previous chest CT scan; pneumonia was identified and treated, but the clinician did not mention the lung nodules or plan for their subsequent evaluation. The third false-negative record resulted from a procedure incorrectly coded as complete in the EHR. In all three cases, patients received subsequent follow-up of the suspicious finding within 2 years (median time, 111 days).

Application of electronic trigger algorithms to big EHR data can aid clinicians in identifying patients experiencing delays in diagnostic evaluation of abnormal lung imaging results. Our trigger achieved a diagnostic accuracy suitable for future practical use, with a sensitivity of 99%, a specificity of 38%, an NPV of 97%, and a PPV of 61%. These findings suggest that triggers are able to identify almost all delays related to abnormal lung imaging follow-up (high sensitivity and NPV) and cost-effectively minimize the amount of effort providers spend reviewing false-positive results (PPV > 50%). Although 30 days was chosen as an appropriate cutoff for the facilities in this study, other institutions considering such triggers could increase or decrease this time frame based on clinical resources available and thus increase or decrease PPV, respectively. These triggers have demonstrated the potential for successful use in large datasets across multiple sites. Such large-scale application could leverage economies of scale by allowing multiple sites to use a centralized team to monitor and act on potential delays. The new Institute of Medicine report (“Improving Diagnosis in Health Care”) also recommends methods to identify and reduce delays in care from lack of timely follow-up of abnormal diagnostic test results.

In the majority of confirmed delays identified, we were unable to determine reasons for the delay during chart reviews, suggesting that providers likely just missed EHR notifications. However, despite documented plans for follow-up action in 84 instances, we found that 26% did not adhere to their plan. This result confirms the findings from multiple international studies that breakdowns in test result communication and follow-up remain a problem for health-care systems and carry the potential for poor patient outcomes and malpractice litigation.,,,,,, Existing health-care systems and EHRs do not provide sufficient tools and workflows to enable tracking of the plans that providers put into place, such as completion of test orders, referrals, and follow-up visits. Our research suggests that use of big EHR data can identify at-risk patients, allowing mitigation of delays. Future efforts are needed to develop tracking tools and to iteratively refine workflows to support the transformation of health care from a series of visit-based episodes of care to a comprehensive visit-agnostic health care system., Subsequent use of such triggers and their performance may benefit from the American College of Radiology’s effort to standardize lung imaging reporting via the Lung Imaging Reporting and Data System program.

Despite the performance achieved by use of the trigger, several limitations of our study merit discussion. First, although the study was conducted at several health-care institutions, these sites belonged to a single national health care system that used the same EHR. Thus, the findings are not necessarily generalizable to commercial EHRs and non-VA institutions. However, our previous research suggests trigger portability given use of standardized codes (eg, those from Current Procedural Terminology and International Classification of Diseases), which could be particularly effective in the large data repositories of other health systems, both within and outside the United States,,, or with health information exchanges that are becoming more common. Furthermore, if American College of Radiology recommendations for standardizing lung imaging reports are adopted, triggers could become more widely used. Second, retrospective chart reviews relied on text within the EHR, which may not always describe actual care delivered or the rationale for not taking action. However, we previously found a high correlation between documentation and action. Third, the study was not designed to evaluate the clinical or economic impact of prospective trigger use on morbidity, mortality, and stage at diagnosis. Nevertheless, our recent clinical trial revealed that prospective triggers can reduce delays in follow-up at minimal personnel costs when information about trigger-flagged records is provided to patients’ care teams, and we plan to evaluate these outcomes in future studies.

An algorithm designed to identify patients at risk for delays in follow-up of abnormal imaging from a large national dataset performed with reasonable accuracy for use in the clinical setting. Future research to develop and refine similar algorithms more widely can potentially reduce delays in diagnostic evaluation and improve quality and safety of patient care.

Author contributions: D. R. M. and A. N. D. M. had full access to all data of the study and take responsibility for the integrity of the data and accuracy of the data analysis. D. R. M., A. N. D. M., D. F. S., E. R., and H. S. contributed to the conception and design of the study; D. R. M., V. B., L. Wei, L. Wu, and H. S. assisted in the acquisition of data; and D. R. M., A. N. D. M., V. B., D. F. S., and H. S. assisted in the analysis and interpretation of data. All authors contributed to the drafting and revision of the article and have provided approval for submission and publication.

Financial/nonfinancial disclosures: None declared.

Role of sponsors: These funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, and approval of the manuscript.

Other contributions: The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Manyika J, Chui M, Brown B, et al. Big data: the next frontier for innovation, competition, and productivity.http://www.citeulike.org/group/18242/article/9341321. Accessed August 3, 2015.
 
Belle A. .Thiagarajan R. .Soroushmehr S.M.R. .Navidi F. .Beard D.A. .Najarian K. . Big data analytics in healthcare. BioMed Res Int. 2015;2015:370194- [PubMed]journal. [PubMed]
 
Murphy D.R. .Reis B. .Sittig D.F. .Singh H. . Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012;125:209.e1-209.e7 [PubMed]journal. [CrossRef]
 
Murphy D.R. .Reis B. .Kadiyala H. .et al Electronic health record-based messages to primary care providers: valuable information or just noise? Arch Intern Med. 2012;172:283-285 [PubMed]journal. [CrossRef] [PubMed]
 
McDonald C.J. .McDonald M.H. . Electronic medical records and preserving primary care physicians’ time: comment on “Electronic health record-based messages to primary care providers.”. Arch Intern Med. 2012;172:285-287 [PubMed]journal. [CrossRef] [PubMed]
 
Poon E.G. .Kachalia A. .Puopolo A.L. .Gandhi T.K. .Studdert D.M. . Cognitive errors and logistical breakdowns contributing to missed and delayed diagnoses of breast and colorectal cancers: a process analysis of closed malpractice claims. J Gen Intern Med. 2012;27:1416-1423 [PubMed]journal. [CrossRef] [PubMed]
 
Graber M.L. .Franklin N. .Gordon R. . Diagnostic error in internal medicine. Arch Intern Med. 2005;165:1493-1499 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Giardina T.D. .Petersen L.A. .et al Exploring situational awareness in diagnostic errors in primary care. BMJ Qual Saf. 2012;21:30-38 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Singh H. .Berlin L. . Communication breakdowns and diagnostic errors: a radiology perspective. Diagnosis. 2014;1:253-261 [PubMed]journal. [PubMed]
 
Menon S. .Smith M.W. .Sittig D.F. .et al How context affects electronic health record-based test result follow-up: a mixed-methods evaluation. BMJ Open. 2014;4:e005985- [PubMed]journal. [CrossRef] [PubMed]
 
Al-Mutairi A. .Meyer A.N. .Chang P. .Singh H. . Lack of timely follow-up of abnormal imaging results and radiologists’ recommendations. J Am Coll Radiol. 2015;12:385-389 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Sethi S. .Raber M. .Petersen L.A. . Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol. 2007;25:5009-5018 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Hirani K. .Kadiyala H. .et al Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record–based study. J Clin Oncol. 2010;28:3307-3315 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Thomas E.J. .Mani S. .et al Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med. 2009;169:1578-1586 [PubMed]journal. [PubMed]
 
Gandhi T.K. .Kachalia A. .Thomas E.J. .et al Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med. 2006;145:488-496 [PubMed]journal. [CrossRef] [PubMed]
 
Tørring M.L. .Frydenberg M. .Hansen R.P. .Olesen F. .Hamilton W. .Vedsted P. . Time to diagnosis and mortality in colorectal cancer: a cohort study in primary care. Br J Cancer. 2011;104:934-940 [PubMed]journal. [CrossRef] [PubMed]
 
Phillips R.L. Jr..Bartholomew L.A. .Dovey S.M. .Fryer G.E. Jr..Miyoshi T.J. .Green L.A. . Learning from malpractice claims about negligent, adverse events in primary care in the United States. Qual Saf Health Care. 2004;13:121-126 [PubMed]journal. [CrossRef] [PubMed]
 
Brenner R.J. .Lucey L.L. .Smith J.J. .Saunders R. . Radiology and medical malpractice claims: a report on the practice standards claims survey of the Physician Insurers Association of America and the American College of Radiology. Am J Roentgenol. 1998;171:19-22 [PubMed]journal. [CrossRef]
 
Berlin L. .Murphy D.R. .Singh H. . Breakdowns in communication of radiological findings: an ethical and medico-legal conundrum. Diagnosis. 2014;1:263-268 [PubMed]journal. [PubMed]
 
Murphy D.R. .Meyer A.N. .Russo E. .Sittig D.F. .Wei L. .Singh H. . The burden of inbox notifications in commercial electronic health records. JAMA Intern Med. 2016;176:559-560 [PubMed]journal. [CrossRef] [PubMed]
 
Lacson R. .O’Connor S.D. .Sahni V.A. .et al Impact of an electronic alert notification system embedded in radiologists’ workflow on closed-loop communication of critical results: a time series analysis. BMJ Qual Saf. 2016;25:518-524 [PubMed]journal. [CrossRef] [PubMed]
 
Litchfield I. .Bentham L. .Lilford R. .McManus R.J. .Hill A. .Greenfield S. . Test result communication in primary care: a survey of current practice. BMJ Qual Saf. 2015;24:691-699 [PubMed]journal. [CrossRef] [PubMed]
 
Litchfield I. .Bentham L. .Hill A. .McManus R.J. .Lilford R. .Greenfield S. . Routine failures in the process for blood testing and the communication of results to patients in primary care in the UK: a qualitative exploration of patient and provider perspectives. BMJ Qual Saf. 2015;24:681-690 [PubMed]journal. [CrossRef] [PubMed]
 
National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care.  2015;:- [PubMed] The National Academies Press Washington, DCjournal
 
Murphy D.R. .Laxmisan A. .Reis B.A. .et al Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf. 2014;23:8-16 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Thomas E.J. .Meyer A.N. .Singh H. . Development and validation of electronic health record-based triggers to detect delays in follow-up of abnormal lung imaging findings. Radiology. 2015;277:81-87 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Wilson L. .Petersen L.A. .et al Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication. BMC Med Inform Decis Mak. 2009;9:49- [PubMed]journal. [CrossRef] [PubMed]
 
Kidney E. .Berkman L. .Macherianakis A. .et al Preliminary results of a feasibility study of the use of information technology for identification of suspected colorectal cancer in primary care: the CREDIBLE study. Br J Cancer. 2015;112:S70-S76 [PubMed]journal. [CrossRef] [PubMed]
 
Szekendi M.K. .Sullivan C. .Bobb A. .et al Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care. 2006;15:184-190 [PubMed]journal. [PubMed]
 
Classen D.C. .Resar R. .Griffin F. .et al “Global trigger tool” shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581-589 [PubMed]journal. [CrossRef] [PubMed]
 
Jha A.K. .Classen D.C. . Getting moving on patient safety—harnessing electronic data for safer care. N Engl J Med. 2011;365:1756-1758 [PubMed]journal. [CrossRef] [PubMed]
 
Zalis M. .Harris M. . Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol. 2010;7:625-633 [PubMed]journal. [CrossRef] [PubMed]
 
Fihn S.D. .Francis J. .Clancy C. .et al Insights from advanced analytics at the Veterans Health Administration. Health Aff Proj Hope. 2014;33:1203-1211 [PubMed]journal. [CrossRef]
 
Gould M.K. .Donington J. .Lynch W.R. .et al Evaluation of individuals with pulmonary nodules: when is it lung cancer? Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2013;143:e93S-120S [PubMed]journal. [CrossRef] [PubMed]
 
Wang Y.X. .Gong J.S. .Suzuki K. .Morcos S.K. . Evidence based imaging strategies for solitary pulmonary nodule. J Thorac Dis. 2014;6:872-887 [PubMed]journal. [PubMed]
 
MacMahon H. .Austin J.H. .Gamsu G. .et al Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology. 2005;237:395-400 [PubMed]journal. [CrossRef] [PubMed]
 
Altman D.G. . Practical Statistics for Medical Research.  1990;:- [PubMed] Chapman and Hall/CRC Press Londonjournal
 
Callen J.L. .Westbrook J.I. .Georgiou A. .Li J. . Failure to follow-up test results for ambulatory patients: a systematic review. J Gen Intern Med. 2012;27:1334-1348 [PubMed]journal. [CrossRef] [PubMed]
 
Callen J. .Georgiou A. .Li J. .Westbrook J.I. . The safety implications of missed test results for hospitalised patients: a systematic review. BMJ Qual Saf. 2011;20:194-199 [PubMed]journal. [CrossRef] [PubMed]
 
Kwan J.L. .Cram P. . Do not assume that no news is good news: test result management and communication in primary care. BMJ Qual Saf. 2015;24:664-666 [PubMed]journal. [CrossRef] [PubMed]
 
Andrews E. .Toubman S. . Patient-centered medical home: improving health care by shifting the focus to patients. Conn Med. 2009;73:479-480 [PubMed]journal. [PubMed]
 
Davis K. .Schoenbaum S.C. .Audet A.M. . A 2020 vision of patient-centered primary care. J Gen Intern Med. 2005;20:953-957 [PubMed]journal. [CrossRef] [PubMed]
 
Pinsky P.F. .Gierada D.S. .Black W. .et al Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med. 2015;162:485-491 [PubMed]journal. [CrossRef] [PubMed]
 
Hazlehurst B.L. .Kurtz S.E. .Masica A. .et al CER Hub: an informatics platform for conducting comparative effectiveness research using multi-institutional, heterogeneous, electronic clinical data. Int J Med Inf. 2015;84:763-773 [PubMed]journal. [CrossRef]
 
Paulus R.A. .Davis K. .Steele G.D. . Continuous innovation in health care: implications of the Geisinger experience. Health Aff (Millwood). 2008;27:1235-1245 [PubMed]journal. [CrossRef] [PubMed]
 
Adler-Milstein J. .Bates D.W. .Jha A.K. . Operational health information exchanges show substantial growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32:1486-1492 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Arora H.S. .Vij M.S. .Rao R. .Khan M.M. .Petersen L.A. . Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc. 2007;14:459-466 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Wu L. .Thomas E.J. .Forjuoh S.N. .Meyer A.N. .Singh H. . Electronic trigger-based intervention to reduce delays in diagnostic evaluation for cancer: a cluster randomized controlled trial. J Clin Oncol. 2015;33:3560-3567 [PubMed]journal. [CrossRef] [PubMed]
 

Figures

Figure Jump LinkFigure 1 Technical steps to operationalize trigger.Grahic Jump Location
Figure Jump LinkFigure 2 Trigger validation process flow.Grahic Jump Location

Tables

Table Graphic Jump Location
Table 1 Trigger Algorithm Logic
Table Graphic Jump Location
Table 2 Reasons and Contributory Factors for Delays
Table Graphic Jump Location
Table 3 Reasons for False-Positive Results

VA = Department of Veterans Affairs.

References

Manyika J, Chui M, Brown B, et al. Big data: the next frontier for innovation, competition, and productivity.http://www.citeulike.org/group/18242/article/9341321. Accessed August 3, 2015.
 
Belle A. .Thiagarajan R. .Soroushmehr S.M.R. .Navidi F. .Beard D.A. .Najarian K. . Big data analytics in healthcare. BioMed Res Int. 2015;2015:370194- [PubMed]journal. [PubMed]
 
Murphy D.R. .Reis B. .Sittig D.F. .Singh H. . Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012;125:209.e1-209.e7 [PubMed]journal. [CrossRef]
 
Murphy D.R. .Reis B. .Kadiyala H. .et al Electronic health record-based messages to primary care providers: valuable information or just noise? Arch Intern Med. 2012;172:283-285 [PubMed]journal. [CrossRef] [PubMed]
 
McDonald C.J. .McDonald M.H. . Electronic medical records and preserving primary care physicians’ time: comment on “Electronic health record-based messages to primary care providers.”. Arch Intern Med. 2012;172:285-287 [PubMed]journal. [CrossRef] [PubMed]
 
Poon E.G. .Kachalia A. .Puopolo A.L. .Gandhi T.K. .Studdert D.M. . Cognitive errors and logistical breakdowns contributing to missed and delayed diagnoses of breast and colorectal cancers: a process analysis of closed malpractice claims. J Gen Intern Med. 2012;27:1416-1423 [PubMed]journal. [CrossRef] [PubMed]
 
Graber M.L. .Franklin N. .Gordon R. . Diagnostic error in internal medicine. Arch Intern Med. 2005;165:1493-1499 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Giardina T.D. .Petersen L.A. .et al Exploring situational awareness in diagnostic errors in primary care. BMJ Qual Saf. 2012;21:30-38 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Singh H. .Berlin L. . Communication breakdowns and diagnostic errors: a radiology perspective. Diagnosis. 2014;1:253-261 [PubMed]journal. [PubMed]
 
Menon S. .Smith M.W. .Sittig D.F. .et al How context affects electronic health record-based test result follow-up: a mixed-methods evaluation. BMJ Open. 2014;4:e005985- [PubMed]journal. [CrossRef] [PubMed]
 
Al-Mutairi A. .Meyer A.N. .Chang P. .Singh H. . Lack of timely follow-up of abnormal imaging results and radiologists’ recommendations. J Am Coll Radiol. 2015;12:385-389 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Sethi S. .Raber M. .Petersen L.A. . Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol. 2007;25:5009-5018 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Hirani K. .Kadiyala H. .et al Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record–based study. J Clin Oncol. 2010;28:3307-3315 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Thomas E.J. .Mani S. .et al Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med. 2009;169:1578-1586 [PubMed]journal. [PubMed]
 
Gandhi T.K. .Kachalia A. .Thomas E.J. .et al Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med. 2006;145:488-496 [PubMed]journal. [CrossRef] [PubMed]
 
Tørring M.L. .Frydenberg M. .Hansen R.P. .Olesen F. .Hamilton W. .Vedsted P. . Time to diagnosis and mortality in colorectal cancer: a cohort study in primary care. Br J Cancer. 2011;104:934-940 [PubMed]journal. [CrossRef] [PubMed]
 
Phillips R.L. Jr..Bartholomew L.A. .Dovey S.M. .Fryer G.E. Jr..Miyoshi T.J. .Green L.A. . Learning from malpractice claims about negligent, adverse events in primary care in the United States. Qual Saf Health Care. 2004;13:121-126 [PubMed]journal. [CrossRef] [PubMed]
 
Brenner R.J. .Lucey L.L. .Smith J.J. .Saunders R. . Radiology and medical malpractice claims: a report on the practice standards claims survey of the Physician Insurers Association of America and the American College of Radiology. Am J Roentgenol. 1998;171:19-22 [PubMed]journal. [CrossRef]
 
Berlin L. .Murphy D.R. .Singh H. . Breakdowns in communication of radiological findings: an ethical and medico-legal conundrum. Diagnosis. 2014;1:263-268 [PubMed]journal. [PubMed]
 
Murphy D.R. .Meyer A.N. .Russo E. .Sittig D.F. .Wei L. .Singh H. . The burden of inbox notifications in commercial electronic health records. JAMA Intern Med. 2016;176:559-560 [PubMed]journal. [CrossRef] [PubMed]
 
Lacson R. .O’Connor S.D. .Sahni V.A. .et al Impact of an electronic alert notification system embedded in radiologists’ workflow on closed-loop communication of critical results: a time series analysis. BMJ Qual Saf. 2016;25:518-524 [PubMed]journal. [CrossRef] [PubMed]
 
Litchfield I. .Bentham L. .Lilford R. .McManus R.J. .Hill A. .Greenfield S. . Test result communication in primary care: a survey of current practice. BMJ Qual Saf. 2015;24:691-699 [PubMed]journal. [CrossRef] [PubMed]
 
Litchfield I. .Bentham L. .Hill A. .McManus R.J. .Lilford R. .Greenfield S. . Routine failures in the process for blood testing and the communication of results to patients in primary care in the UK: a qualitative exploration of patient and provider perspectives. BMJ Qual Saf. 2015;24:681-690 [PubMed]journal. [CrossRef] [PubMed]
 
National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care.  2015;:- [PubMed] The National Academies Press Washington, DCjournal
 
Murphy D.R. .Laxmisan A. .Reis B.A. .et al Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf. 2014;23:8-16 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Thomas E.J. .Meyer A.N. .Singh H. . Development and validation of electronic health record-based triggers to detect delays in follow-up of abnormal lung imaging findings. Radiology. 2015;277:81-87 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Wilson L. .Petersen L.A. .et al Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication. BMC Med Inform Decis Mak. 2009;9:49- [PubMed]journal. [CrossRef] [PubMed]
 
Kidney E. .Berkman L. .Macherianakis A. .et al Preliminary results of a feasibility study of the use of information technology for identification of suspected colorectal cancer in primary care: the CREDIBLE study. Br J Cancer. 2015;112:S70-S76 [PubMed]journal. [CrossRef] [PubMed]
 
Szekendi M.K. .Sullivan C. .Bobb A. .et al Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care. 2006;15:184-190 [PubMed]journal. [PubMed]
 
Classen D.C. .Resar R. .Griffin F. .et al “Global trigger tool” shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581-589 [PubMed]journal. [CrossRef] [PubMed]
 
Jha A.K. .Classen D.C. . Getting moving on patient safety—harnessing electronic data for safer care. N Engl J Med. 2011;365:1756-1758 [PubMed]journal. [CrossRef] [PubMed]
 
Zalis M. .Harris M. . Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol. 2010;7:625-633 [PubMed]journal. [CrossRef] [PubMed]
 
Fihn S.D. .Francis J. .Clancy C. .et al Insights from advanced analytics at the Veterans Health Administration. Health Aff Proj Hope. 2014;33:1203-1211 [PubMed]journal. [CrossRef]
 
Gould M.K. .Donington J. .Lynch W.R. .et al Evaluation of individuals with pulmonary nodules: when is it lung cancer? Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2013;143:e93S-120S [PubMed]journal. [CrossRef] [PubMed]
 
Wang Y.X. .Gong J.S. .Suzuki K. .Morcos S.K. . Evidence based imaging strategies for solitary pulmonary nodule. J Thorac Dis. 2014;6:872-887 [PubMed]journal. [PubMed]
 
MacMahon H. .Austin J.H. .Gamsu G. .et al Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology. 2005;237:395-400 [PubMed]journal. [CrossRef] [PubMed]
 
Altman D.G. . Practical Statistics for Medical Research.  1990;:- [PubMed] Chapman and Hall/CRC Press Londonjournal
 
Callen J.L. .Westbrook J.I. .Georgiou A. .Li J. . Failure to follow-up test results for ambulatory patients: a systematic review. J Gen Intern Med. 2012;27:1334-1348 [PubMed]journal. [CrossRef] [PubMed]
 
Callen J. .Georgiou A. .Li J. .Westbrook J.I. . The safety implications of missed test results for hospitalised patients: a systematic review. BMJ Qual Saf. 2011;20:194-199 [PubMed]journal. [CrossRef] [PubMed]
 
Kwan J.L. .Cram P. . Do not assume that no news is good news: test result management and communication in primary care. BMJ Qual Saf. 2015;24:664-666 [PubMed]journal. [CrossRef] [PubMed]
 
Andrews E. .Toubman S. . Patient-centered medical home: improving health care by shifting the focus to patients. Conn Med. 2009;73:479-480 [PubMed]journal. [PubMed]
 
Davis K. .Schoenbaum S.C. .Audet A.M. . A 2020 vision of patient-centered primary care. J Gen Intern Med. 2005;20:953-957 [PubMed]journal. [CrossRef] [PubMed]
 
Pinsky P.F. .Gierada D.S. .Black W. .et al Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med. 2015;162:485-491 [PubMed]journal. [CrossRef] [PubMed]
 
Hazlehurst B.L. .Kurtz S.E. .Masica A. .et al CER Hub: an informatics platform for conducting comparative effectiveness research using multi-institutional, heterogeneous, electronic clinical data. Int J Med Inf. 2015;84:763-773 [PubMed]journal. [CrossRef]
 
Paulus R.A. .Davis K. .Steele G.D. . Continuous innovation in health care: implications of the Geisinger experience. Health Aff (Millwood). 2008;27:1235-1245 [PubMed]journal. [CrossRef] [PubMed]
 
Adler-Milstein J. .Bates D.W. .Jha A.K. . Operational health information exchanges show substantial growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32:1486-1492 [PubMed]journal. [CrossRef] [PubMed]
 
Singh H. .Arora H.S. .Vij M.S. .Rao R. .Khan M.M. .Petersen L.A. . Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc. 2007;14:459-466 [PubMed]journal. [CrossRef] [PubMed]
 
Murphy D.R. .Wu L. .Thomas E.J. .Forjuoh S.N. .Meyer A.N. .Singh H. . Electronic trigger-based intervention to reduce delays in diagnostic evaluation for cancer: a cluster randomized controlled trial. J Clin Oncol. 2015;33:3560-3567 [PubMed]journal. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Find Similar Articles
CHEST Journal Articles
  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543