Essay: Reviewing medical records to improve patient safety in hospitals

Reviewing medical records to improve patient safety in hospitals, are the methods evidence based?
 
Abstract
Introduction: Infliction of unintended medical harm received increased attention. Increasing numbers of hospitals evaluate medical records of patients who died during their stay. This is labour-intensive and expensive. Information about adequacy of screening tools used for detecting adverse events (AEs) is therefore important.
Methods: We searched the literature for evidence concerning these screening methods based on the World Health Organisation (WHO) criteria for evaluating screening programs. Results are reported separately for the trigger tool of the Institute for Healthcare Improvement (IHI) and the one developed in the Harvard Medical Practice Study (HMPS).
Results: 4353 studies were identified. After selection 57 studies with relevant information about these tests remained. The reduction in prevalence of adverse events (AE) was between 0.5%-3.5% per year. For the IHI method the specificity ranged between 93%-100%, and the sensitivity between 34%-95%. Average positive predictive value (PPV) of the trigger systems was 38%. The Kappa for agreement between doctors on the presence of an AE ranged between fair and substantial in the HMPS studies and between fair and almost perfect with IHI triggers. Average costs of an AE were €8739.
Conclusion: Retrospective medical record review has not been extensively evaluated especially not according to the WHO criteria for screening programs. There is a lack of information on improvement in AE occurrence, cost-effectiveness and the predictive value of the triggers. We recommend policy makers to scientifically evaluate the chosen methods for their utility and cost effectiveness with improvement of care as the end goal.
 
Introduction
Unintentional medical harm inflicted on patients in hospitals received increased attention during the past years. Several studies have shown significant rates of adverse events (AEs) that harm patients during their hospital stay.1-3 Therefore, interest for implementing safety- and quality programs has grown. Avoiding medical injury during hospital admission and improving patient safety therefore has high priority in hospitals, for healthcare inspection and for the government.4 According to the report ‘to Err is Human’ of the Institute of Medicine, at least 44,000 people (and perhaps even 96,000) die each year in hospitals in the US as a result of medical errors that could have possibly been prevented. Fifteen years after the initial report, a recent update stresses the importance of keeping focus on improving patient safety.3 In the Netherlands, a report by NIVEL (Dutch institute for research in healthcare) in which hospital medical records were evaluated on care related harm, increased the need for transparency and responsibility about this subject. They estimated that in Dutch hospitals every year about 1700 patients (4.1% of the total number of deaths in hospitals) die because of unintentional, but avoidable, harm. Six years after this initial report, which was published in 2007, follow up showed improvement but, 2.6% of the total number of deaths in hospital still occurred because of unintentional but preventable harm. In other countries an incidence between 2.5% and 11.5% was found.5-9 Several reasons are thought to contribute to the risk of these AEs. First, the increasing number of elderly patients with severe comorbidity and growing medical technical possibilities to carry out more complex treatments and interventions, result in more risks. Second, due to this expansion in treatment options the estimation of the possible health benefits is more complex. This could lead to treating more patients in which the predicted outcome is unclear, especially in the elderly and newborns with substantial comorbidities. Furthermore, social changes as part-time working and stricter adherence to prescribed working hours have their influence on the expertise and experience of medical professionals and the continuity of care. Teamwork and integrated care are therefore necessary, but they depend on good communication within and between teams which is frequently suboptimal. Also economic limitations and cuts put health-care under pressure.10-14 All of these factors are thought to influence patient safety.
Reducing unintentional but preventable harm is important for hospitals and the methods used for detection are diverse. In 1991 the Harvard Medical Practice Study provided us with a powerful tool to identify cases with possible AE.15 Another method which is often used in hospitals worldwide to select cases with possible AEs is the global trigger tool (GTT), developed by the Institute for Healthcare Improvement (IHI).16 This method is found to be labour-intensive and expensive when used for reviewing all deaths in the hospital. Information about sensitivity, specificity, positive and negative predictive values of this screening tool for detecting AEs are therefore important. With this information we can potentially determine the costs for every detected AE. However, detecting AEs is of course not the ultimate goal. Rather, the procedure aims to avoid harm to patients and even to save lives. So, adequately detecting AEs is only a small part of the whole process, which also involves feedback to the medical departments, adjustments in the delivery of care and hence improved outcome for patients. From this viewpoint, searching for AEs is actually a screening method in which the AE is the disease for which early intervention should improve the outcome.
Therefore we wondered whether these screening methods are evidence based according to the WHO criteria (box 1) for evaluating screening programs.17 Although these criteria where initially developed for the evaluation of screening programs concerning for example the early detection of breast malignancies or colon cancer we think they are applicable to other screening programs as well. Therefore, we searched the literature for evidence concerning the use of these trigger tools and chart review methods to identify AEs and subsequently improving patient outcome, using these WHO criteria.
Box 1: 7 criteria for evaluating screening programs:
Methods
Data collection
For every criterion (Box 1) we performed a literature search in Pubmed, Embase and the Cochrane library. An overview of the search terms we used per WHO criterion is provided in table 1. We selected articles published between January 2000 and February 2016. Reviews were excluded as well as posters, comments, studies that did not concern a hospital setting and studies about adverse drug events.
Abstracts were reviewed by DK. Full articles were assessed by DK and RR. The following data were extracted if provided by the authors: Absolute number of cases reviewed, number of (preventable) AEs, sensitivity, specificity, positive and negative predictive value (PPV and NPV), reproducibility of the methods used, severity of AEs as well as costs. These variables were expressed as a mean with confidence intervals (CIs) when possible. We report all results for each of the WHO steps separately. Costs were not corrected for inflation. Different currencies were transformed to euros to make comparison easier. The exchange rate from June 2016 was used.
 
Results
The number of studies generated by every part of our search is shown in table 2. Our search provided a total of 4353 citations, of which 4334 were discarded after title and/or abstract screening, 19 studies remained for further evaluation. After reading the references in these studies we found 38 additional studies. Some studies were relevant for more than 1 step. Below, we report the results step by step.
Step 1: Our search revealed 3 suitable studies concerning an improvement in end results (table 3). In 2001, Wolff et al published about the AE rate over a certain time period concerning inpatients and patients at the emergency department.18 The absolute risk reduction for inpatients to suffer an AE was 0.61% (in a period of 8 years, on average 0.08% per year). For patients in the emergency department this was 2.78% (in a period of 2 years, on average 1.4% per year). In data from the Baylor Health Care System, Kennerly et al (2013) showed a 7% reduction in AEs in a period of 2 years (on average 3.5% per year).19 In a more recent study Suarez et al found during a 6-year study period a decrease in absolute risk for suffering an AE of 2.5% (on average 0,4% per year).20
Step 2: Our search revealed 32 suitable studies concerning the effectiveness of the components of the multiphasic screening process. Studies were available on the trigger tool and the AE assessment strategy. We found no studies specifically addressing the result of feedback to the medical departments and the change in the rate of AEs related to this.
2.1 Trigger tool
Briefly, we distinguished two different trigger tools that were often used: the trigger tool originating from the Harvard medical practice study15,21 (HMPS) with 18 triggers and the GTT22 (IHI) with 54 triggers. Only a few studies23-25 use a different set of triggers that were not used in any other study we found. When looking at those using the HMPS triggers (figure 1), the positive predictive value (PPV), for identifying cases with AEs varied between 17 and 68% with an average of 33% (95% CI 22-44).5,7,26-35
The agreement between the nurses on the presence of a trigger, showed a kappa value (K) between 0.53 and 0.76 (moderate to substantial agreement)36 and an average K of 0.65 (95%CI 0.50-0.80).29,34
The IHI method (figure 1) showed a PPV between 16 and 99%, with an average of 43% (95% CI 15-71).35,37-42 There was only one study that calculated the NPV and this was 99%.37 The K on the presence of a trigger varied between 0.2039 (slight agreement) and 0.7843 (substantial agreement), which corresponds with an average of 0.57 95% CI 0.15-0.99).2,39-41,43
Studies using other trigger tools have a PPV range between 39 and 96%, with an average of 54% (95%CI 25-84).24,25,44-46 Only two studies reported a K indicating the reproducibility of the trigger system. The K varied between 0,52 and 0,68.45,47
The observed variation in these numbers was not related to the number of cases that were investigated. There was also no association with the year a study was performed.
2.2 The AE assessment strategy (figure 2)
AE assessment in HMPS studies:
The K between medical doctors on the presence of an AE was on average 0.55 (moderate agreement, 95%CI 0.31-0.80).27,29-33,48
AE assessments in GTT studies:
Within the studies using the IHI trigger tool the K on the presence of an AE varied between 0.32 and 0.93, with an average of 0.64 (good agreement, 95%CI 0.42-0.86). The agreement on the severity of the AE was investigated in three studies, which showed an average K of 0.40 (fair agreement, 95%CI 0.06-0.73).1,2,19,38,39,41,43,49,50
AE assessments in other trigger systems:
Within studies using another trigger system, the K on the presence of an AE varied between 0.35 and 0.71, with an average of 0.38(fair agreement, 95%CI 0.15-0.62).47,51,52
 
Step 3:
Our search did not reveal any studies addressing this question.
Step 4:
We found 13 suitable studies concerning the costs-effectiveness or cost-benefit of the screening. The costs of an AE range between €800 (2013) and €51,804 (2007) in these studies with an average of €8739 (studies between 2001 and 2015). For preventable AE these numbers are between €3083 (2014) and €101,620 (2007) with an average of €36,0201.7,28,53-63 1(1 euro = 1.13 US dollar June 2016)
Step 5:
66 studies were identified that match our in and exclusion criteria. In figure 3 the results are shown according to the trigger tool used, with averages for every separate harm category.1,2,31,35,38,39,64-70. The definition for the different harm categories is as follows;
E: temporary harm to the patient and required intervention. F: temporary harm to the patient and required initial or prolonged hospitalization. G: permanent patient harm. H: intervention necessary to sustain life. I: death.71
Other studies use an alternative approach for detecting AEs like morbidity and mortality conferences, autopsy, malpractice claims analysis, error reporting systems, clinical surveillance etc.72 All these approaches have some advantages compared to our method but also considerable disadvantages.
Step 6:
For the IHI method the specificity ranged between 93% and 100%, and the sensitivity between 34% and 95%.38,67 We found no evidence for cost effectiveness and acceptability regarding this kind of screening in our search.
Step 7:
Preventable medical errors in hospitals have been estimated to result in costs between €16 billion and €37 billion a year nationwide in the United States.
Beyond the cost in human lives, preventable AEs also have effects on other costs. These costs include the expenses of necessary additional care, loss of income and household productivity of the patients, and disability.3
Goodman et al (2011) estimated the annual age-adjusted social costs based on the economic value of a life combined with the estimated annual number of deaths from adverse medical events. It was assumed that hospital injuries were comparable to workplace injuries to estimate the magnitude of the loss caused by these AEs. The social cost of all inpatient AEs (in US) was between 357 and 871 billion euros per year.73 In comparison: total social costs of traffic accidents in the US were estimated around 733 billion euros in 2010.74
 
Discussion & conclusion
With this extended literature search we found many studies addressing quality and safety strategies in health care. Remarkably we found little evidence for validity of the methods used. Furthermore, there was almost no research concerning the cost- and cost-effectiveness of retrospective medical record reviewing in these studies.
Concerning the first step we found only a few studies that report improvement in end-results. However, in our opinion the end goal of screening for AEs is first to detect them reliably and then to strive for a lower rate of preventable harm resulting in increased safety. Although much effort was put into detecting AEs there is little evidence in the literature for effective implementation of improvements to prevent them in the future. Moreover, Wolf et al (1996) showed that over a course of three years, the number of AEs decreased, but they found no significant change in the severity of the AEs. Also, in this study, only one doctor reviewed each medical record giving rise to concern about the reproducibility of this method in different hospitals.75
When evaluating the effectiveness of potential components of the screening instrument, we found it striking that so little studies make a distinction between preventable and non-preventable AEs. There is, in our opinion, little to gain from identifying non preventable AEs because apparently nothing can be done about them. The focus should therefore be on preventable AEs. Regrettably, there are to our knowledge no methods designed to solely extract cases with preventable AEs. Future research should therefore take this into account. Furthermore, the negative predictive value of the commonly used trigger systems is underexposed (only a few studies investigated non triggered cases for AEs) and the positive predictive value varies widely between the studies. This might be caused by the difference in performance of the trigger systems, however we cannot exclude that the prevalence of AEs in different hospitals and countries varies considerably. It appears to be uncommon to investigate the exact prevalence of AEs in a certain population. On the contrary, the prevalence of triggers is well known. However, to compare the PPV of trigger systems and triggers between studies information about the real prevalence of AEs in the investigated population is essential.
We found the inter observer agreement on the presence, preventability and the severity of the AEs disappointing. This is possibly caused by lack of an international standard on interpretation of the triggers or on the methods to identify AEs. Background and experience of those who trigger and those who investigate cases is therefore probably an important cause of this variation. This could affect the strength of the feedback to the medical departments and might give rise to discussions about the accuracy of the final judgment instead of leading to improvement of care.
Information about screening that should benefit the community at large rather than, or in addition to, the individual patient, is completely absent in these studies. The road to an AE is usually influenced by many different factors. It is a chain of events with numerous cross links to other processes that eventually leads to the adverse outcome. If one link is changed to prevent a rare but serious AE this might also influence other processes and hence the outcome of the community at large. Possibly this is mediated through a change in practice, which might not benefit everyone, imposing an extra burden on others or leads to unacceptable high costs. We think therefore that any changes should be closely monitored to reveal possible disadvantages for the community at large.
Rather than researching the cost-effectiveness of screening for AE, most studies focus on the total cost of (preventable) AEs. As a result of this, the annual direct medical costs in Dutch hospitals were estimated at a total of $400 million for all AEs and $181 million for preventable AEs in 2004. These total medical costs of AEs accounted for approximately 1% of the national health care budget.28 Bates et al described the costs of medical record review itself in 1995, specifically the cost for every (preventable) AE found and the cost for every admission that was screened.76 We found no studies thereafter addressing this subject. However, It is expected that in an already safe environment the costs for trying to achieve zero AEs rise exponentially.77
However, the studies included in step 4 only show the total costs of AEs in contrast to the cost-benefit or the cost-effectiveness of screening for AEs itself. That means we are not sure that these methods are cost effective or that the costs per QALY are acceptable. Only Adler et al (2015) studied the cost-benefit ratio of reducing AEs using a model approach.62 It is clear that the costs of AEs are high. Many cases have to be screened to find an AE (number needed to screen is 36 in the emergency department and 164 in other departments) and even more to find a preventable AE (53 in the emergency department).18,19 Also only three studies reported costs for preventable AEs and non-preventable AEs separately.28,56,61 We think information especially on the costs of preventable AEs is important because these are the ones that could be avoided by, for example, adapting protocols.
We interpreted step 5 as the total of AEs in a population sample in a health care environment. Again, because there is no international consensus on the definition of an AE and on the detection method, the results we found are diverse. Some of the studies we found used the same categories for harm (E/F/G/H/I). Where E means ’temporary harm to the patient and required intervention’ and I is ‘patient death’.16 However, different definitions of the harm categories, or even using no clear harm categories at all, made interpretation of the differences between these studies difficult. Retrospective medical record reviewing is used by several hospitals worldwide, but is a costly screening instrument with relies on, often incomplete, medical records. Furthermore, hindsight bias could also influence results. Although Resar et al (2003) state that the trigger tool methodology has the potential to track changes in protocols during time and judge the effect of this, it seems that this potential has not been investigated thoroughly.78 This is supported by Wilson et al (2005), who concluded that there was insufficient information to detect an increase in patient safety of a considerable effort to improve during a decade.79
Unfortunately, only two studies reported on the specificity and sensitivity of the IHI trigger tool. For the HMPS tool, the only study is Brennan et al (1991) calculated a specificity of 84% and a negative predictive value of 92%.21 Regarding the acceptability of the screening test, no studies were found.
Health care inspection and government obligate hospitals to report on a large set of parameters. Gathering this information is time consuming and costly.80 In a recent Dutch report of the Dutch care authority (NZa) and the institute for care in the Netherlands (ZiN), the need for illness- and specialism transcending numbers are stressed.81 This will increase the burden even further. The National Patient Safety Foundation (United States) recommends the creation of a common set of safety metrics.82 However, focusing on gathering many different outcomes or triggers does not change practice to become more safe. We should therefore rather focus on a smaller number of preventable outcomes and direct our efforts and financial resources to improve. Another possibility is to focus more on the causes of harm like adverse drug events, nosocomial infections, venous thromboembolism, decubitus, falls and surgical complications.83
It is clear from the literature that the impact of AEs on both hospital costs and on patients’ quality of life is enormous. It is also obvious that detecting preventable AEs in hospitals is important, however in this review we show that a well validated method for detecting AEs and improving the quality of care is not yet available. Therefore, we recommend policy makers to scientifically evaluate the chosen methods for their utility and cost effectiveness with improvement of care as the end goal.
References
1. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. The New England journal of medicine. 2010;363(22):2124-34.
2. Najjar S, Hamdan M, Euwema MC, Vleugels A, Sermeus W, Massoud R, et al. The Global Trigger Tool shows that one out of seven patients suffers harm in Palestinian hospitals: challenges for launching a strategic safety plan. International journal for quality in health care : journal of the International Society for Quality in Health Care / ISQua. 2013;25(6):640-7.
3. Kohn L, T., Corrigan J.M., Donaldson, M. To Err is human: building a safer health system. Washington, DC: 1999.
4. Wagner C. Onbedoelde schade in ziekenhuizen: resultaten dossieronderzoek naar patiëntveiligheid. Klachtenmanagement in de Zorg. 2007;4(3-4):28-31.
5. Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne. 2004;170(11):1678-86.
6. Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Medical care. 2000;38(3):261-71.
7. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ (Clinical research ed). 2001;322(7285):517-9.
8. Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals I: occurrence and impact. The New Zealand medical journal. 2002;115(1167):U271.
9. Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD. The Quality in Australian Health Care Study. The Medical journal of Australia. 1995;163(9):458-71.
10. Aranaz-Andres JM, Aibar-Remon C, Vitaller-Murillo J, Ruiz-Lopez P, Limon-Ramirez R, Terol- Garcia E, et al. Incidence of adverse events related to health care in Spain: results of the Spanish National Study of Adverse Events. Journal of epidemiology and community health. 2008;62(12):1022-9.
11. Olds DM, Clarke SP. The effect of work hours on adverse events and errors in health care. Journal of safety research. 2010;41(2):153-62.
12. Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Quality & safety in health care. 2004;13 Suppl 1:i85-90.
13. Lingard L, Espin S, Whyte S, Regehr G, Baker GR, Reznick R, et al. Communication failures in the operating room: an observational classification of recurrent types and effects. Quality & safety in health care. 2004;13(5):330-4.
14. Vincent C, Taylor-Adams S, Stanhope N. Framework for analysing risk and safety in clinical medicine. BMJ (Clinical research ed). 1998;316(7138):1154-7.
15. Brennan TA, Leape LL. Adverse events, negligence in hospitalized patients: results from the Harvard Medical Practice Study. Perspect Healthc Risk Manage. 1991;11(2):2-8.
16. Griffin FA RRC, MA: . IHI Global Trigger Tool for Measuring Adverse Events (Second Edition) IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement. 2009.
17. Sackett DL. Screening for early detection of disease: to what purpose? Bull N Y Acad Med. 1975;51(1):39-52.
18. Wolff AM, Bourke J, Campbell IA, Leembruggen DW. Detecting and reducing hospital adverse events: outcomes of the Wimmera clinical risk management program. The Medical journal of Australia. 2001;174(12):621-5.
19. Kennerly D, Richter KM, Good V, Compton J, Ballard DJ. Journey to no preventable risk: the Baylor Health Care System patient safety experience. American journal of medical quality : the official journal of the American College of Medical Quality. 2011;26(1):43-52.
20. Suarez C, Menendez MD, Alonso J, Castano N, Alonso M, Vazquez F. Detection of adverse events in an acute geriatric hospital over a 6-year period using the Global Trigger Tool. Journal of the American Geriatrics Society. 2014;62(5):896-900.
21. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. The New England journal of medicine. 1991;324(6):370-6.
22. Griffin FA, Classen DC. Detection of adverse events in surgical patients using the Trigger Tool approach. Quality & safety in health care. 2008;17(4):253-8.
23. Hayward RA, Hofer TP. Estimating hospital deaths due to medical errors: preventability is in the eye of the reviewer. Jama. 2001;286(4):415-20.
24. Verelst S, Jacques J, Van den Heede K, Gillet P, Kolh P, Vleugels A, et al. Retrospective medical record evaluation: reliability in assessing causation, preventability, and disability of adverse events. Int J Health Care Qual Assur. 2012;25(8):649-62.
25. Wolff AM, Bourke J. Detecting and reducing adverse events in an Australian rural base hospital emergency department using medical record screening and review. Emergency Medicine Journal. 2002;19(1):35-40.
26. Kable AK, Gibberd RW, Spigelman AD. Adverse events in surgical patients in Australia. International Journal for Quality in Health Care. 2002;14(4):269-76.
27. Sari AB, Sheldon TA, Cracknell A, Turnbull A. Sensitivity of routine system for reporting patient safety incidents in an NHS hospital: retrospective patient case note review. BMJ (Clinical research ed). 2007;334(7584):79.
28. Hoonhout LH, de Bruijne MC, Wagner C, Zegers M, Waaijman R, Spreeuwenberg P, et al. Direct medical costs of adverse events in Dutch hospitals. BMC Health Serv Res. 2009;9:27.
29. Soop M, Fryksmark U, Koster M, Haglund B. The incidence of adverse events in Swedish hospitals: A retrospective medical record review study. International Journal for Quality in Health Care. 2009;21(4):285-91.
30. Zegers M, de Bruijne MC, Wagner C, Hoonhout LH, Waaijman R, Smits M, et al. Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Quality & safety in health care. 2009;18(4):297-302.
31. Naessens JM, O’Byrne TJ, Johnson MG, Vansuch MB, McGlone CM, Huddleston JM. Measuring hospital adverse events: assessing inter-rater reliability and trigger performance of the Global Trigger Tool. International journal for quality in health care : journal of the International Society for Quality in Health Care / ISQua. 2010;22(4):266-74.
32. Zegers M, de Bruijne MC, Wagner C, Groenewegen PP, van der Wal G, de Vet HC. The inter- rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record. Journal of clinical epidemiology. 2010;63(1):94-102.
33. Zwaan L, De Bruijne M, Wagner C, Thijs A, Smits M, Van Der Wal G, et al. Patient record review of the incidence, consequences, and causes of diagnostic adverse events. Archives of internal medicine. 2010;170(12):1015-21.
34. Wilson RM, Michel P, Olsen S, Gibberd RW, Vincent C, El-Assady R, et al. Patient safety in developing countries: retrospective estimation of scale and nature of harm to patients in hospital. BMJ. 2012;344:e832.
35. Unbeck M, Schildmeijer K, Henriksson P, Jurgensen U, Muren O, Nilsson L, et al. Is detection of adverse events affected by record review methodology? an evaluation of the “Harvard Medical Practice Study” method and the “Global Trigger Tool”. Patient safety in surgery. 2013;7(1):10.
36. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
37. Naessens JM, Campbell CR, Huddleston JM, Berg BP, Lefante JJ, Williams AR, et al. A comparison of hospital adverse events identified by three widely used detection methods. International journal for quality in health care : journal of the International Society for Quality in Health Care / ISQua. 2009;21(4):301-7.
38. Sharek PJ, Parry G, Goldmann D, Bones K, Hackbarth A, Resar R, et al. Performance characteristics of a methodology to quantify adverse events over time in hospitalized patients. Health services research. 2011;46(2):654-78.
39. Schildmeijer K, Nilsson L, Arestedt K, Perk J. Assessment of adverse events in medical care: lack of consistency between experienced teams using the global trigger tool. BMJ quality & safety. 2012;21(4):307-14.
40. Kennerly DA, Saldana M, Kudyakov R, da Graca B, Nicewander D, Compton J. Description and evaluation of adaptations to the global trigger tool to enhance value to adverse event reduction efforts. Journal of patient safety. 2013;9(2):87-95.
41. Hwang JI, Chin HJ, Chang YS. Characteristics associated with the occurrence of adverse events: a retrospective medical record review using the Global Trigger Tool in a fully digitalized tertiary teaching hospital in Korea. Journal of evaluation in clinical practice. 2014;20(1):27-35.
42. Kurutkan MN, Usta E, Orhan F, Simsekler MC. Application of the IHI Global Trigger Tool in measuring the adverse event rate in a Turkish healthcare setting. The International journal of risk & safety in medicine. 2015;27(1):11-21.
43. O’Leary KJ, Devisetty VK, Patel AR, Malkenson D, Sama P, Thompson WK, et al. Comparison of traditional trigger tool to data warehouse based screening for identifying hospital adverse events. BMJ quality & safety. 2013;22(2):130-8.
44. Forster AJ, Andrade J, Van Walraven C. Validation of a discharge summary term search method to detect adverse events. Journal of the American Medical Informatics Association. 2005;12(2):200-6.
45. Lander L, Roberson DW, Plummer KM, Forbes PW, Healy GB, Shah RK. A trigger tool fails to identify serious errors and adverse events in pediatric otolaryngology. Otolaryngology – Head and Neck Surgery. 2010;143(4):480-6.
46. Murff HJ, Forster AJ, Peterson JF, Fiskio JM, Heiman HL, Bates DW. Electronically screening discharge summaries for adverse medical events. Journal of the American Medical Informatics Association : JAMIA. 2003;10(4):339-50.
47. Ock M, Lee SI, Jo MW, Lee JY, Kim SH. Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events. Journal of preventive medicine and public health = Yebang Uihakhoe chi. 2015;48(5):239-48.
48. Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Medical care. 2000;38(3):261-71.
49. Asavaroengchai S, Sriratanaban J, Hiransuthikul N, Supachutikul A. Identifying adverse events in hospitalized patients using Global Trigger Tool in Thailand. Asian Biomedicine. 2009;3(5):545-50.
50. Mattsson TO, Brixen K, Knudsen JL, Herrstedt J. Measuring adverse event rates in oncology inpatients using trigger tools: An assessment of measurement properties. Journal of Clinical Oncology Conference: ASCO’s Quality Care Symposium. 2012;30(34 SUPPL. 1).
51. Hofer TP, Bernstein SJ, DeMonner S, Hayward RA. Discussion between reviewers does not improve reliability of peer review of hospital quality. Medical care. 2000;38(2):152-61.
52. Kessomboon P, Panarunothai S, Wongkanaratanakul P. Detecting adverse events in Thai hospitals using medical record reviews: agreement among reviewers. Journal of the Medical Association of Thailand = Chotmaihet thangphaet. 2005;88(10):1412-8.
53. Brown P, McArthur C, Newby L, Lay-Yee R, Davis P, Briant R. Cost of medical injury in New Zealand: a retrospective cohort study. Journal of health services research & policy. 2002;7 Suppl 1:S29-34.
54. Ehsani JP, Jackson T, Duckett SJ. The incidence and cost of adverse events in Victorian hospitals 2003-04. Medical Journal of Australia. 2006;184(11):551-5.
55. Kaushal R, Bates DW, Franz C, Soukup JR, Rothschild JM. Costs of adverse events in intensive care units. Critical care medicine. 2007;35(11):2479-83.
56. Mello MM, Studdert, D.M., Thomas E.J., Yoon, C.S., Brennan, T.A. . Who pays for medical errors? An analysis of adverse event costs, the medical laiblity system, and incentives for patient safety improvement. Journal of empirical legal studies 2007;4(4):835-60.
57. Pappas SH. The cost of nurse-sensitive adverse events. J Nurs Adm. 2008;38(5):230-6.
58. Van Den Bos J, Rustagi K, Gray T, Halford M, Ziemkiewicz E, Shreve J. The $17.1 billion problem: the annual cost of measurable medical errors. Health affairs (Project Hope). 2011;30(4):596-603.
59. Wardle G, Wodchis WP, Laporte A, Anderson GM, Ross Baker G. The sensitivity of adverse event cost estimates to diagnostic coding error. Health services research. 2012;47(3 PART 1):984-1007.
60. David G, Gunnarsson CL, Waters HC, Horblyuk R, Kaplan HS. Economic measurement of medical errors using a hospital claims database. Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2013;16(2):305-10.
61. Magdelijns FJ, Stassen PM, Stehouwer CD, Pijpers E. Direct health care costs of hospital admissions due to adverse events in The Netherlands. European journal of public health. 2014;24(6):1028-33.
62. Adler L, Yi D, Li M, McBroom B, Hauck L, Sammer C, et al. Impact of Inpatient Harms on Hospital Finances and Patient Clinical Outcomes. Journal of patient safety. 2015.
63. Hoogervorst-Schilp J, Langelaan M, Spreeuwenberg P, de Bruijne MC, Wagner C. Excess length of stay and economic consequences of adverse events in Dutch hospital patients. BMC health services research. 2015;15(1):531.
64. Kennerly DA, Kudyakov R, da Graca B, Saldana M, Compton J, Nicewander D, et al. Characterization of adverse events detected in a large health care delivery system using an enhanced global trigger tool over a five-year interval. Health services research. 2014;49(5):1407-25.
65. Good VS, Saldana M, Gilder R, Nicewander D, Kennerly DA. Large-scale deployment of the Global Trigger Tool across a large hospital system: refinements for the characterisation of adverse events to support patient safety learning opportunities. BMJ quality & safety. 2011;20(1):25-30.
66. Garrett PR, Jr., Sammer C, Nelson A, Paisley KA, Jones C, Shapiro E, et al. Developing and implementing a standardized process for global trigger tool application across a large health system. Joint Commission journal on quality and patient safety / Joint Commission Resources. 2013;39(7):292-7.
67. Classen DC, Resar R, Griffin F, Federico F, Frankel T, Kimmel N, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30(4):581-9.
68. Mull HJ, Brennan CW, Folkes T, Hermos J, Chan J, Rosen AK, et al. Identifying Previously Undetected Harm: Piloting the Institute for Healthcare Improvement’s Global Trigger Tool in the Veterans Health Administration. Quality management in health care. 2015;24(3):140-6.
69. Rutberg H, Borgstedt Risberg M, Sjodahl R, Nordqvist P, Valter L, Nilsson L. Characterisations of adverse events detected in a university hospital: a 4-year study using the Global Trigger Tool method. BMJ open. 2014;4(5):e004879.
70. Farup PG. Are measurements of patient safety culture and adverse events valid and reliable? Results from a cross sectional study. BMC health services research. 2015;15:186.
71. Griffin FA. IHI Global Trigger Tool for Measuring Adverse Events (Second Edition) IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement 2009.
72. Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med. 2003;18(1):61-7.
73. Goodman JC, Villarreal P, Jones B. The social cost of adverse medical events, and what we can do about it. Health affairs (Project Hope). 2011;30(4):590-5.
74. Blincoe LJ M, TR, Zaloshnja, E. The Economic and Societal Impact Of Motor Vehicle Crashes in 2010. 2015.
75. Wolff AM. Limited adverse occurrence screening: using medical record review to reduce hospital adverse patient events. The Medical journal of Australia. 1996;164(8):458-61.
76. Bates DW, O’Neil AC, Petersen LA, Lee TH, Brennan TA. Evaluation of screening criteria for adverse events in medical patients. Medical care. 1995;33(5):452-62.
77. Juran JM. Classical model of optimum quality costs. From Juran Quality Control Handbook. : McGraw-Hill 1988.
78. Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Quality & safety in health care. 2003;12 Suppl 2:ii39-45.
79. Wilson RM, Van Der Weyden MB. The safety of Australian healthcare: 10 years after QAHCS. Med J Aust. 2005;182(6):260-1.
80. Plexus K. Onderzoek kosten kwaliteitsmetingen. 2015.
81. Nederland NZZ. Ontwikkeling algemene indicatoren 2016.
82. Foundation NPS. Free from Harm: Accelerating Patient Safety Improvement Fifteen Years after To Err Is Human 2015.
83. Jha A, Pronovost P. Toward a Safer Health Care System: The Critical Need to Improve Measurement. Jama. 2016;315(17):1831-2.
Appendices
Table 1: search terms used in the different WHO steps
Search terms
Step 1: ((mortality OR morbidity OR death OR dead OR lethality OR lethal) AND (chart OR dossier OR (medical record) OR (clinical record) OR (medical history)) AND (reviewing OR screening OR analysis)) AND ((AE) OR error OR mistake)
Step 2: (trigger or (inter-rater) or (intra-rater) or observer) AND ((dossier analysis) OR (medical record) or (clinical record) OR (medical history)) AND ((AE) OR error OR mistake)
Step 3: (disadvantage or con) AND ((chart review) OR (dossier analysis) OR (medical record) or (clinical record) OR (medical history)
Step 4: ((cost-benefit) OR (cost-effectiveness)) and ((global trigger tool) OR (chart review) OR (dossier analysis) OR (medical record) or (clinical record) OR (medical history)) AND ((AE) OR error OR mistake)
Step 5: ((incidence OR rate OR occurrence OR frequency OR prevalence) AND ((AE) OR (medical mistake) OR (medical error) OR incident OR harm)
Step 6: (cost or sensitivity or specificity) AND ((global trigger tool) OR (medical record review) OR (chart review)) AND (AE)
Step 7: ((social benefit) or (social cost)) AND ((AE) or (medical error)) AND ((trigger tool) OR (medical record review) OR (chart review))
Date of search was the 1st of February 2016. All search terms were applied in the databases of Pubmed, Embase and Cochrane.
Table 2: Results according to the different WHO steps
Step Results Result after title screening Result after abstract screening Additionally, from references
after abstract screening
1 Pubmed: 1473
Embase: 587
Cochrane: 54 125 1 2
2 Pubmed: 130
Embase: 25
Cochrane: 3 48 14 19
3 Pubmed: 369
Embase: 520
Cochrane: 5 1 0 0
4 Pubmed: 115
Embase: 171
Cochrane: 9 6 1 12
5 Pubmed: 360
Embase: 329
Cochrane: 14 174 12 2
6 Pubmed: 114
Embase: 49
Cochrane: 16 14 1 1
7 Pubmed: 8
Embase: 0
Cochrane: 2 3 0 2
Table 3: results of step 1
Time period Decline of AEs
Wolff et al (2001)
18
8 years inpatients 69/5111 -> 49/6615
2 years emergency 84/2577 -> 12/2496
Kennerly et al (2011) 19
2 years 31.1/100 -> 24.1/100
Suarez et al (2014) 20
6 years 73/240 -> 66/240
Figure legends
Figure 1. Boxplot of the PPV for every screening method separately. * marks an outlier.
Figure 2. Box-plot of the Kappa for agreement between doctors regarding the presence of an AE, for every screening method separately.
Figure 3. Bar chart of the harm categories of the AEs found in IHI and HMPS studies. E: temporary harm to the patient, required intervention. F: temporary harm to the patient, required initial or prolonged hospitalization. G: permanent patient harm. H: intervention necessary to sustain life. I: death.