Does quality improvement improve quality? Mary Dixon-Woods RAND Professor of Health Services Research Cambridge Centre for Health Services Research, University of Cambridge, Institute of Public Health, Forvie Site, Robinson Way, Cambridge CB2 0SR. email: md753@medschl.cam.ac.uk Graham Martin Professor of Health Organisation and Policy SAPPHIRE, Department of Health Sciences, Centre for Medicine, University of Leicester, Leicester LE1 7RH. email: gpm7@le.ac.uk Funding: Mary Dixon-Woods is funded by a Wellcome Trust Investigator award, WT097899 Abstract: Though quality improvement is frequently advocated as a way of addressing healthcare’s problems, evidence of its effectiveness has remained very mixed. The reasons for this are varied, but a growing literature highlights particular challenges. Fidelity in the application of QI methods is often mixed. QI work is often pursued through time-limited, small-scale projects, led by professionals who may lack the expertise, power or resources to instigate the changes required. There is insufficient attention to rigorous evaluation of improvement, and to sharing the lessons of successes and failures. Too many QI interventions are seen as “magic bullets” that will produce improvement in any situation, regardless of context. And too much improvement work is undertaken in isolation at a local level, failing to pool resources and develop collective solutions, and introducing new hazards in the process. This article considers these challenges and proposes four key ways in which quality improvement might itself be improved. Keywords: quality improvement; patient safety; healthcare organisation and delivery; evaluation; research design/methods; hospitals. The quality and safety of healthcare worldwide remains problematic. Many of the basic operational systems and routines of work required to care for patients are not fit for purpose. Few have been purposefully designed or documented; instead, they are handed down through genealogies, sometimes mutating along the way, so that a process intended to do the same thing may vary wildly across places, teams and shifts, and sub-optimal functioning of processes to serve clinical work are the norm. As a result, the reliability of NHS clinical systems is poor, varying from 81% to 87%. 1 Processes for apparently simple tasks, such as ensuring the right equipment is available in operating theatres, or that prescribed medication is administered on time, , fail to function as intended with worrying frequency. When trained clinical teams use methods adapted from high risk industries, they typically uncover multiple defects and hazards across their teams, units and organisations. 2 The associated risks are compounded when multiple systems and sectors interact, as is common in healthcare.3 These defects are highly consequential, impacting on efficiency, safety, and the well-being of staff and patients.4 US studies suggest that nurses deal with an average of 8.4 work system failures per 8 hour shift, and they are continually interrupted. 5, 6 The need for staff to learn and re-learn associated with the variability in fundamental processes is significant: much professional time is consumed unproductively in learning anew in each setting how to undertake tasks as basic as ordering tests, knowing whether equipment has been cleaned, or how things are arranged in the resuscitation trolley. Personnel may also make errors as they move from place to place, either because they have not yet learned the new procedures or they apply previous learning to new but different contexts, with sometimes tragic outcomes. 7 The problems with quality improvement Healthcare has increasingly been encouraged to use quality improvement (QI) techniques to tackle these operational defects (clearly, healthcare faces many other challenges – but they may require different approaches) Capacity to improve quality is clearly critical to healthcare organisations: every organisation needs to be able to detect its operational (and other) problems and solve them using structured methods. For many (though far from all) problems, that may mean using methods adapted from other industries, such as Lean and Six Sigma, or approaches developed within healthcare, such as the Institute for Healthcare Improvement’s Model for Improvement. This widely used modelcombines measurement – using statistical process control, for example – with small tests of change (Plan-Do-Study-Act or PDSA cycles).8 But despite the widespread advocacy for QI, the evidence that it produces positive impacts in healthcare has been very mixed, with many of the better-designed studies producing disappointing results.9-14 One recent review concluded that Lean interventions, for example, do not have a significant association with patient satisfaction or health outcomes, but do have a negative association with financial costs and worker satisfaction and inconsistent effects on process outcomes.15 What explains these discouraging findings is now the focus of growing interest. One explanation appears to lie in poor fidelity in use of QI methods: a recent review found, for example, poor adherence and reporting to the basic tenets of PDSA cycles in quality improvement reports.16 More generally, what may happen is that the superficial outer appearance of the intervention or quality improvement method is reproduced, but not the internal mechanisms (or set of mechanisms) that produced the outcomes in the first instance. 17, 18 These effects may arise because what is implemented in practice may be diluted, distorted, or diminished versions of the intervention, as has been found, for example, in relation to leadership walkrounds.19, 20 Second, much QI work continues to be undertaken in the form of time-limited small-scale projects, perhaps conducted as part of professional accreditation requirements. Some of the achievements of this work are striking, but caution is needed. One risk is that QI becomes an activity largely assigned to professionals in training, who rarely have the skills, resources or power to effect the kinds of changes that may be required. For instance, a problem with crowding in oncology outpatients may have its origins in a complex tangle of poorly designed or functioning processes (e.g. ensuring blood results are available on time), but diagnosing the cause and redesigning the workflow accordingly might need a dedicated team with specialist training in ergonomics and the clout to support the changes needed: these are not resources usually available to junior doctors or small QI teams. They may therefore come up with a small fix or workaround that fails to solve the true problems, and, in so doing, may introduce new risks. Another risk is that of encouraging “projectness” 21 – a sense that QI is a series of bounded, time-limited events rather than a continuous commitment, and overly focused on “innovation” rather than replication. Treating QI as a series of local projects may increase the tendency for wheel reinvention – or different “solutions” to the same problem. Undoubtedly, this expansion of overlapping efforts in part reflects the relative novelty of QI in healthcare. But it requires urgent attention, not least because ill-coordinated improvement may, ironically, intensify the problem of locally-specific work processes, routines and tasks that only apply in their context of origin. Multiple ill-coordinated small-scale QI projects may, accordingly, degrade rather than improve the ability to achieve improvements across healthcare as a whole.22 Moreover, as attention shifts from one project to another, the gains achieved in the first may attenuate, a phenomenon that has been termed the “improvement evaporation effect.”23 A third, and linked, problem is the ongoing failure to cumulate and share learning from QI efforts. The NHS continually loses learning, and this is an urgent problem. Though proper evaluation is essential to advancing the science of improvement, 24 those who introduce local QI interventions are sometimes so convinced that the change introduced is positive that they may eschew evaluation. 25 When people do come up with good ideas and test them rigorously, the learning may be difficult to share and challenging for others to discover – in part because they are never reported, or, if they are, are not in an accessible form. When people come up with ideas that don’t work, the learning is even more likely to remain obscured. These problems contribute significantly to wheel reinvention and to waste of energy and time. Yet traditional medical research funding mechanisms and publishing norms are poorly aligned with the imperative to evaluate, curate, and make available experiences (positive and negative) and outcomes of both QI methods and quality improvement interventions. Even when QI is reported, it tends to be poorly described. 26 It thus remains difficult even to find out about a success or a failure elsewhere, let alone to know what was really done, and with what outcomes. A further challenge lies in the ongoing emphasis on specific interventions as the keys to quality improvement, perhaps particularly when those interventions are valorised as magic bullets. 27 The dynamic interplay between intervention and context means that it is often difficult, and indeed not always helpful, to separate intervention from context, 28 to the extent that transplanting a programme in its entirety from one setting to another is rarely straightforward. 29 Excessive attention to QI interventions in the narrow sense – e.g. huddles, bundles, checklists and other popular tools – risks overlooking impact of context on intervention implementation, but perhaps more importantly the critical role of context itself as generative of safety and quality. Very often, the kind of place that has come up with the idea for doing huddles and has been able to implement and sustain them is also the kind of place has all the other characteristics that facilitate quality and safety. The notion that the huddle – or anything else – is then a plug-and-play “solution” is consequently misguided: the features of context (clarity of vision, infrastructure, organisational systems, values, skills and so on) that made it work in the first place need to be reproduced too. Healthcare organisations differ markedly from factory production lines, just as human bodies are not ‘widgets’: acknowledging and attending to the social and cultural context is vital if improvement interventions are to be made to work. The tendency to attribute effects to interventions (rather than interventions and contexts working together) is further exacerbated by the problem that the forces that create positive conditions for quality and safety may be invisible to those who create them or may not be possible (or straightforward) to articulate and hence for others to reproduce or recreate. The intervention as described in published reports may thus offer only a partial account of the reasons why the success was achieved: foregrounding a specific intervention, no matter how well characterised, as the explanation for the outcomes may risk backgrounding or rendering invisible important mechanisms that contribute to the achievement of those outcomes. The result is a theoretically deficient approach to improvement that may rely on “magical thinking.”30 Many of these challenges can be illustrated by looking at the example of sepsis management. Here, organisations are encouraged to do a “bundle” of six clinical activities within one hour: 1. Deliver high-flow oxygen. 2. Take blood cultures. 3. Administer empiric intravenous antibiotics. 4. Measure serum lactate and send full blood count. 5. Start intravenous fluid resuscitation. 6. Commence accurate urine output measurement. Delivering on each one of these goals requires a supporting infrastructure, ranging from role clarity through to sufficient well-maintained equipment. For example, obtaining a serum lactate within a rapid turnaround time requires optimised equipment and organisational systems, as well as staff with the right expertise available at the right time. Making all of these things happen requires a high level of skills in operations design, but may also require all kinds of other skills in implementation – including negotiating for clarity about roles and responsibilities, managing professional or managerial resistance to reconfigurations of tasks, delivering high quality training, and so on. It is probably not necessary for each individual organisation to invest the effort in figuring out all of the tasks and activities needed to achieve each of the goals. Nor is it likely that all organisations will have all the necessary expertise to come up with good solutions. But if a good solution is found, it may help others, since it can be shared and give them a head-start. Such a solution will need to go beyond a the narrow specifics of a well-bounded, easily describable intervention, and encompass the range of facilitating conditions – infrastructural, technological, social, maybe even cultural – that have often been relegated to the category of “context,” but which are themselves vital to the success of efforts to improve. And it is important that the solutions reached are broadly similar across organisations, so that once a practitioner has learned the system once she will know broadly what to do next time. It may be disastrous, for example, if the system for alerting professionals of the availability of a test result varies from one setting to another, since they may rely on being alerted in a particular way, with the potential for delay if it does not happen. Overcoming the challenges Where does this leave us, and how can healthcare improve? Several ways of addressing this can be proposed (Box 1): all will require much more coordination of quality improvement, and a far more professionalised approach, than has been evident so far. Healthcare should start by agreeing on the kinds of challenges for which full standardisation and interoperability are needed across the sector, which are the solutions that can be agreed at the level of principle and left up to local customisation at implementation, and which should be entirely locally developed. Healthcare leaders should identify the right kinds of structures for achieving these goals, ranging from international harmonisation mechanisms (similar, for example, to those used in the automobile industry) through to local innovation. Horizontal networks – including those enabled by the Royal Colleges, as well as initiatives such as the Health Foundation’s Q – are likely to be especially valuable, since such structures can accommodate professional groupings who can work together to agree on solutions that are satisfying, workable, informed by professional values and clinical expertise, capable of being customised for specific situations, and enforceable through peers rather than harsh, externally imposed sanctions 31, 32 Finally, it should address the problem of many hands22 by identifying who has responsibility for solving problems for which no single actor in the system has responsibility, but which affect healthcare as a collective. Box 1: How to improve the quality of quality improvement 1. Act like a sector. Allowing a thousand flowers of quality interventions to bloom is not a sensible or efficient way of going about fixing healthcare, and it introduces new risks. As we have argued elsewhere, many of the quality challenges that confront healthcare need to be solved at the level of entire systems, 22 not hospital by hospital, practice by practice, care home by care home. Healthcare needs to take itself seriously as a collective whole or sector- like entity, capable of agreeing standard operating procedures and systems that are designed with the right expertise, tested properly, implemented with professional leadership at the core, and remain open to innovation. Where technology or external standardisation is the issue – for example, the ongoing failure to address issues of alarm fatigue, incompatible devices, or drug-naming and packaging practices – political leadership will be needed, though professional advocacy and involvement will be essential. Much, however, can be achieved by coming together voluntarily: the key will be to find the right structures for enabling this. A key principle is that such structures should be properly inclusive, and include patients, carers and multiple professional disciplines as well as other sectors and other workers as appropriate. 2. Stop looking for magic bullets: focus on organisational strengthening and learn from positive deviance. When healthcare has sought to learn from other industries, it has not always done so in thoughtful or well-informed ways. It has instead tended to adopt specific interventions (e.g. checklists) and tried to treat them as magic bullets that are then implemented with little fidelity. Too little has been spent on the organisational strengthening needed to make improvement. Once the search for magic bullet interventions is abandoned, much can be learned from the characteristics, practices and behaviours that are implicated in the performance of demonstrably safe and high quality settings. This is the approach used, for example, in studies of high-reliability organisations. 33, 34 The increasingly popular positive deviance approach 35 similarly seeks to learn from exceptionally good performance. Sometimes, this approach can help to identify processes that promote high quality care; 3637 sometimes, it will identify characteristics of context (values, behaviours, structures and so on) that need to be propagated. What is clear already is that organisations need to develop clear goals, manage people and resources effectively, foster a sense of moral community, develop their information and intelligence systems, and ensure that they have the capacity to engage in problem-solving.4, 38 3. Build capacity for designing and testing solutions, and plan for replication and scaling from the start. Developing solutions to many quality and safety problems may require high-level skills and expertise from multiple disciplines, and highly sophisticated development processes. It is clear that we need to get better at developing or selecting interventions that have a high likelihood of success, testing them rigorously in different contexts, and offering organisations solutions (the technical and operational issues they need to tackle and the “hints and tips” on the things they will need to do to make the change happen). Much more attention is needed to developing high-quality prototypes of possible solutions in laboratory- like conditions (which may be a designated hospital or network of hospitals that agrees to act as the lab) and undertaking modelling and simulation before they are tested for real. The goal of such testing should be to identify, among other things, how the solution might work in different scenarios and conditions, and to work out what are the core, non-negotiable elements and what can be locally customised. Testing should also support intelligent replication and scaling: it is now clear that a simple description of the components of an intervention is not enough; what matters is likely to be the activation of mechanisms, even if precise activities undertaken to activate those mechanisms differ across contexts. Fidelity will lie in the mechanisms rather than fussy adherence to specific forms. 4. Think programmes and resources, not projects. QI projects are sometimes the right answer – for example, where there is a specific, bounded problem to be solved, and particularly if it is one where experience and evidence suggest a plausible solution – but where they are undertaken it should be with a commitment to sharing. More productive than individual, short-term projects in general may be to think and plan in terms of long-term programmes of work that are coordinated through some central hub, and that doctors-in-training and others work on for particular periods of time as part of a contribution to a bigger effort. For instance, they might be involved in some of the testing activities described above, or on data analysis. Many people who do improvement work are not trained academics and the reports of their work are not traditional academic outputs, but not being able to publish and share also diminishes the attractiveness of improvement work in terms of career rewards and satisfaction. Healthcare needs to do for QI what it has done for research: build an infrastructure that enables learning about successful and less successful efforts to be curated and searched by others. An open-access peer-reviewed curation model that would provide a searchable database of improvement resources people have developed or used in their organisations is one possibility worth exploring; authors should be offered guidance on the aspects of the intervention, context and implementation process they should cover, to make this resource as accessible, comprehensive and useful as possible. 1. Burnett S, Franklin BD, Moorthy K, Cooke MW, Vincent C. How reliable are clinical systems in the UK NHS? A study of seven NHS organisations. BMJ Qual Saf 2012; Jun;21(6):466-72. 2. Dixon-Woods M, Martin G, Tarrant C, Bion J, Goeschel C, Pronovost P, et al. Safer Clinical Systems: evaluation findings. Learning from the independent evaluation of the second phase of the Safer Clinical Systems programme. London: The Health Foundation; 2015. 3. Dixon-Woods M, Suokas A, Pitchforth E, Tarrant C. An ethnographic study of classifying and accounting for risk at the sharp end of medical wards. Soc Sci Med 2009; Aug;69(3):362-9. 4. Dixon-Woods M, Baker M, Charles J, Dawson J, Jerzembek G, Martin G, et al. Culture and behaviour in the English National Health Services: overview of lessons from a large multi-method study. BMJ Qual Saf 2013;23(2):106-15. 5. Tucker AL, Spear SJ. Operational failures and interruptions in hospital nursing. Health Serv Res 2006;41(3p1):643-62. 6. Tucker AL. The impact of operational failures on hospital nurses and their patients. J Oper Manage 2004;22(2):151-69. 7. Toft B. External inquiry into the adverse incident that occurred at Queen's Medical Centre, Notthingham, 4th January 2001. Department of Health: London; 2001. 8. Berwick DM. A primer on leading the improvement of systems. British Medical Journal 1996;312(7031):619. 9. Benning A, Ghaleb M, Suokas A, Dixon-Woods M, Dawson J, Barber N, et al. Large scale organisational intervention to improve patient safety in four UK hospitals: mixed method evaluation. BMJ 2011; Feb 3;342:d195. 10. Benning A, Dixon-Woods M, Nwulu U, Ghaleb M, Dawson J, Barber N, et al. Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ 2011; Feb 3;342:d199. 11. Coory M, White VM, Johnson KS, Hill DJ, Jefford M, Harrison S, et al. Systematic review of quality improvement interventions directed at cancer specialists. J Clin Oncol 2013; Apr 20;31(12):1583-91. 12. Mason S, Nicolay C, Darzi A. The use of Lean and Six Sigma methodologies in surgery: A systematic review. The Surgeon 2015;13(2):91-100. 13. Zhong W, Feinstein JA, Patel NS, Dai D, Feudtner C. Tall Man lettering and potential prescription errors: a time series analysis of 42 children's hospitals in the USA over 9 years. BMJ Qual Saf 2016; Apr;25(4):233-40. 14. Anthony T, Murray BW, Sum-Ping JT, Lenkovsky F, Vornik VD, Parker BJ, et al. Evaluating an evidence-based bundle for preventing surgical site infection: a randomized trial. Archives of Surgery 2011;146(3):263-9. 15. Moraros J, Lemstra M, Nwankwo C. Lean interventions in healthcare: do they actually work? A systematic literature review. Int J Qual Health Care 2016; Apr;28(2):150-65. 16. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf 2014; Apr;23(4):290-8. 17. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011; Jun;89(2):167-205. 18. Dixon-Woods M, Leslie M, Tarrant C, Bion J. Explaining Matching Michigan: an ethnographic study of a patient safety program. Implement Sci 2013; Jun 20;8(1):70. 19. Martin G, Ozieranski P, Willars J, Charles K, Minion J, McKee L, et al. Walkrounds in practice: corrupting or enhancing a quality improvement intervention? A qualitative study. The Joint Commission Journal on Quality and Patient Safety 2014;40(7):303-10. 20. Rotteau L, Shojania KG, Webster F. 'I think we should just listen and get out': a qualitative exploration of views and experiences of Patient Safety Walkrounds. BMJ Qual Saf 2014; Oct;23(10):823-9. 21. Dixon-Woods M, McNicol S, Martin G. Ten challenges in improving quality in healthcare: lessons from the Health Foundation's programme evaluations and relevant literature. BMJ Quality Safety 2012;21(10):876-84. 22. Dixon-Woods M, Pronovost PJ. Patient safety and the problem of many hands. BMJ Qual Saf 2016; Feb 24;. 23. Buchanan D, Fitzgerald L. Improvement evaporation: why do successful changes decay? In: Buchanan D, Fitzgerald L, Ketley D, editors. The sustainability and spread of organizational change London: Routledge; 2007. p. 22-44. 24. Marshall M, Pronovost P, Dixon-Woods M. Promotion of improvement as a science. Lancet 2013; Feb 2;381(9864):419-21. 25. Nicolay C, Purkayastha S, Greenhalgh A, Benn J, Chaturvedi S, Phillips N, et al. Systematic review of the application of quality improvement methodologies from the manufacturing industry to surgical healthcare. Br J Surg 2012;99(3):324-35. 26. Jones EL, Lees N, Martin G, Dixon-Woods M. How Well Is Quality Improvement Described in the Perioperative Care Literature? A Systematic Review. Jt Comm J Qual Patient Saf 2016; May;42(5):196-206. 27. Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. The art of medicine. Reality check for checklists. The Lancet 2009;374(9688):444-5. 28. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol 2009;43(3-4):267-76. 29. Bauman LJ, Stein REK, Ireys HT. Reinventing fidelity: The transfer of social technology among settings. Am J Commun Psychol 1991;19(4):619-39. 30. Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and its use in improvement. BMJ Qual Saf 2015; Mar;24(3):228-38. 31. Aveling EL, Martin G, Armstrong N, Banerjee J, Dixon-Woods M. Quality improvement through clinical communities: eight lessons for practice. J of Health Org and Mgt 2012;26(2):158-74. 32. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89(2):167-205. 33. Lekka C. High reliability organisations: A review of the literature. Research Report. Health and Safety Executive; 2011. Report No.: 899. 34. Weick KE, Sutcliffe KM, Obstfeld D. Organizing for high reliability: Processes of collective mindfulness. Crisis Management 1999;3:81-123. 35. Lawton R, Taylor N, Clay-Williams R, Braithwaite J. Positive deviance: a different approach to achieving patient safety. BMJ Quality & Safety 2014; July 21;. 36. Bradley EH, Curry LA, Spatz ES, Herrin J, Cherlin EJ, Curtis JP, et al. Hospital strategies for reducing risk-standardized mortality rates in acute myocardial infarction. Ann Intern Med 2012;156(9):618-26. 37. Bradley EH, Curry LA, Webster TR, Mattera JA, Roumanis SA, Radford MJ, et al. Achieving Rapid Door-To-Balloon Times: How Top Hospitals Improve Complex Clinical Systems. Circulation 2006; February 28;113(8):1079-85. 38. Aveling EL, Parker M, Dixon-Woods M. What is the role of individual accountability in patient safety? A multi-site ethnographic study. Sociol Health Illn 2015; Nov 4;38(2):216-32.