Massimo Micocci, Hannah Kettley-linsell, Shanshan Zhou, Paul Festor, Simone Borsci, Matthieu Komorowski, Myura Nagendran, Anthony Gordon, Peter Buckle, George Hanna, Aldo Faisal (2024). Exploring the Integration of AI in Sepsis Management: A Pilot Study on Clinician Acceptance and Decision-Making in ICU Settings. In: Artificial Intelligence and Social Computing. AHFE Open Access, vol. 122, pp. 198–207. https://doi.org/10.54941/ahfe1004657
1 NIHR London In-Vitro Diagnostics Cooperative, Department of Surgery and Cancer, Imperial College London, UK ↗
2 Department of Computing, Imperial College London, UK ↗
3 NIHR London In-Vitro Diagnostics Cooperative, Department of Surgery and Cancer Imperial College London, UK; Faculty of Behavioural, Management and Social Sciences (BMS), Cognition, Data and Education (CODE), University of Twente, NL ↗
4 Faculty of Medicine, Department of Surgery & Cancer, Imperial College London, UK ↗
5 NIHR London In-Vitro Diagnostics Cooperative, Department of Surgery and Cancer Imperial College London, UK; Faculty of Medicine, Department of Surgery & Cancer, Imperial College London, UK ↗↗
6 Faculty of Engineering, Department of Bioengineering, Imperial College London, UK ↗
This paper presents a human factors qualitative study on an AI application for managing sepsis in Intensive Care Units (ICUs). The study involved semi-structured interviews with nine ICU clinicians and nurses across three London hospitals. It consisted of two parts: the first applied methods to understand sepsis resuscitation processes and establish opportunities for the AI tool to mitigate gaps in the process. The second part examined adherence to AI recommendations based on factors like shift timing and user seniority, and whether shared risk in team decisions affects adherence. The findings revealed that while acknowledging the AI tool's potential benefits, participants would require a clear rationale explaining the AI results. They preferred AI suggestions that aligned with their views and did not risk patient safety, often seeking the confirmation of a colleague in uncertain situations. Overall, the study emphasised the cautious, context-dependent acceptance of AI recommendations in ICU settings. It also demonstrated the need for human factors studies to evaluate the user response to AI and its implications on decision-making.
Artificial Intelligence (AI) systems’ efficacy depends not only on their mathe- matical precision but also on the dynamic interplay of different components in the healthcare socio-technical system. Organisational regulations, work culture, specialised duties, additional computational tools, interactions with patients and other healthcare practitioners, and internal and external envi- ronmental elements are all important in determining how much trust people
Exploring the Integration of AI in Sepsis Management 199
have in AI systems (Asan, Bayrak and Choudhury, 2020). The adoption of future AI based medical systems depends on the definition of accurate, reli- able and shared methods for gathering (real-world) data. These include items such as stakeholders’ and users’ trust in the system, dataset reliability, regu- latory issues, cybersecurity, implications for decision-making and impact on operators and patients (Rajpurkar et al., 2022; Gerke, Minssen and Cohen, 2020; Bjerring and Busch, 2021; Lee and Yoon, 2021). Reliability is crit- ical, as the consistency of AI performance can vary with new data, and AI systems might produce biased or overfitted results from inadequate or subjective data, undermining user trust (Mcknight et al., 2011). However, maximal trust in AI is not ideal, as it may lead to uncritical acceptance of its recommendations, which can be dangerous in life-critical applications (Asan, Bayrak and Choudhury, 2020). As AI in healthcare is becoming more pervasive, there is a growing number of human factor’s challenges to be considered to ensure safe and effective patient care. The integration of AI in healthcare not only transforms clinicians’ roles and risks diminishing their hands-on skills and expertise, but also necessitates structured commu- nication for safe handovers between AI and clinicians, enhances situation awareness to prevent adverse outcomes, and potentially affects the emotional and personal dynamics of patient-clinician interactions (Sujan et al., 2019). Flawed machine learning recommendations could negatively influence clin- icians’ treatment choices, and simply providing explanations does not ade- quately mitigate the issue of overdependence on these imperfect algorithms (Jacobs et al., 2021).
This paper presents a pilot study on the integration of an AI tool – AI Clin- ician - for the clinical management of sepsis in Intensive Care Units (ICUs). The study adopts human factors methods and principles to understand the contextual factors affecting the use and integration of this new tool.
Integrating AI in Sepsis Management Sepsis is a leading cause of death and a major healthcare expense, consuming up to 40% of ICU budgets and costing the UK economy up to £10.2 billion annually (Fleischmann et al., 2015). Management challenges include early detection, severity assessment, and targeted therapy, which primarily involves intravenous fluids and vasopressors (Avni et al., 2015; Byrne and Van Haren, 2017; Cohen et al., 2015; Marik and Bellomo, 2016; Gotts and Matthay, 2016). On the other hand, opinions differ regarding the ideal quantity and schedule of these interventions. No tools currently offer individualized treat- ment; approximately 50% of fluid treatments fail to improve cardiac output, indicating potential harm (Mackenzie and Noble, 2014). The utilisation of resources, hospital stays, and worse outcomes can all result from incorrect vasopressor dosage (Mackenzie and Noble, 2014; Levy, Evans and Rhodes, 2018; Gotts and Matthay, 2016). Personalised medicine is still a goal, and current recommendations support a broad approach.
The “AI Clinician” model was created to enhance the resuscitation of sepsis patients by adopting reinforcement learning to suggest sequential deci- sions about fluids and vasopressors over time as the patient’s condition changes (Komorowski et al., 2018). Past studies have demonstrated that it
outperforms human clinicians on average by using large ICU databases to learn effective treatment procedures (Komorowski et al., 2018).
To transition research into practical clinical application, a human factors evaluation of contextual elements is crucial. The healthcare system, being complex, demands an understanding of its broad work system for quality care and patient outcomes (Carayon, 2006). The Work System Model emphasizes the interplay of five elements: individuals, tasks, tools/technologies, physi- cal environment, and organizational conditions (Carayon, 2009); alterations in any element affect the others, necessitating a comprehensive approach to integrating new technologies in healthcare. Environmental influences, such as patient situation, resource availability and interpersonal relationships, are known to affect the decision-making (Bucknall, 2003); the overall organi- zational context is key for assessing the impact of new interventions. The process of shared clinical decision-making in the ICU involves four stepped levels, from the lowest to the highest levels of collaboration: individual decision, information exchange, deliberation, and shared decision and this process is influenced by individual and system factors. System factors, such as interdisciplinary rounds and unit culture, seem to have a strong impact on this process (Ganz et al., 2016). The perspective is consistent with systems engineering research to date and recommends applying approaches such as the SEIPS model to comprehend the socio-technical work system as a whole (Carayon et al., 2020). This all-encompassing strategy, which captures cur- rent and evolving concerns pertinent to this sector, is essential for creating trust models in healthcare AI that work.
Aim and Objectives This study aims to identify organisational and contextual factors that would affect the use and integration of the AI Clinician tool; we also seek to estab- lish the qualitative evidence base for the implementation of the AI Clinician technology to support sepsis management.
METHOD
We undertook a qualitative, scenario-based study, composed of two sets of semi-structured interviews with key healthcare stakeholders. The first set aimed to understand sepsis resuscitation processes through Process Mapping (PM) methods (Micocci et al., 2021; Antonacci et al., 2021; Kalman, 2002) and discuss opportunities for the integration of the AI Clinician tool. The second part of the study explored whether adherence to AI suggestions is influenced by contextual factors like shift timing or user seniority, and if shared risk in team decisions increases adherence. The patient scenarios pre- sented consisted of a set of real-world patient data and sepsis management against the AI recommendation.
Data Collection We conducted semi-structured interviews with questions about the partici- pants’ experience in sepsis management (semi-structured interviews I) and their rationale in managing a real-life patient scenario (semi-structured inter- views II). These activities were conducted remotely (MS Teams) and, with the
Exploring the Integration of AI in Sepsis Management 201
consent of participants, audio recorded. A total of n = 8 participants (n = 5 ICU clinicians and n = 3 staff nurses) at three London hospitals, all part of the Imperial College Healthcare NHS Trust, took part in the two activities. Par- ticipants were coded with a progressive personal ID (P1-P9); one participant (P2) took part in both semi-structured interviews I and II.
During semi-structured interviews I, participants were requested to outline the clinical process for relevant patient groups in their hospitals, particu- larly how patients are identified, diagnosed, treated, and managed, including the initiation and titration of fluids and vasopressors. We also explored the current challenges and gaps in this pathway, their impact on patient outcomes, and the typical duration of resuscitation. Following this, we pre- sented the AI Clinician technology, detailing its functions, potential uses, benefits, and drawbacks. Participants were then asked how this technology might alter their treatment decisions, its possible applications, advantages, disadvantages, and strategies to overcome any barriers to its adoption.
The semi-structured interview II protocol comprised a real-life routine sce- nario of a patient with sepsis, including key parameters and volume of fluids and vasopressors given within 12 hours, as opposed to recommendations provided by the AI Clinician tool (Table 1).
Table 1. Patient case parameters as shown in the semi-structured interview II.
Name: Sally Red Age: 45 Gender: Female Weigh: 146 Kg No. of comorbidities:1 Admission SOFA score: 15 Any mechanical ventilation so far: Yes Any renal replacement so far: Yes
In a comparative exercise, participants compared the treatments they recommended with those recommended by the AI Clinician. They were grad- ually given patient data and AI recommendations while they examined a patient case at three different time points: four, seven, and ten hours after the start of treatment. The roles and responsibilities of bedside nurses and junior and senior doctors were described in introductory comments to create the scene:
Junior doctors: You’re performing a ward round, evaluating a 45-year-old woman in ICU requiring vasopressors and fluids.
Senior doctors: During a night shift, you’re consulted by nurses for a newly admitted 45-year-old woman in ICU needing vasopressors and fluids.
Bedside nurses: You’re monitoring a newly admitted 45-year-old woman in ICU requiring vasopressors and fluids.
Participants were asked to justify the continuation of their treatment after the patient data at each time point presented. Following the display of the AI Clinician’s recommendation, participants were asked if they would change their final decision as well as their initial dosages of fluid and vasopressor. A think-aloud protocol was used to gain insight into their reasoning process.
Data Analysis The Framework Analysis Method (Ritchie et al., 2013) was employed for organizing and conducting a thematic analysis of the interview data. All inter- views were audio-recorded, transcribed verbatim, and coded using NVivo software. The research analysis underwent multiple stages, starting with familiarization with the data, followed by open coding of initial transcripts and creating a code list. This list expanded as more transcripts were analysed. Codes were grouped into subthemes to identify patterns, and final themes were determined through consensus. Finally, the findings were reviewed by the research team for robustness.
FINDINGS Clinical Pathway, Gaps and Opportunities for AI Clinician
The management of sepsis involves several steps:
Sepsis is identified as organ dysfunction due to a dysregulated response to infection. Patients, often presenting with symptoms like pneumonia or high fever, are initially managed in A&E or medical wards. Blood tests are conducted to identify infection markers. The National Early Warning Score (NEWS2) assesses a patient’s physiological parameters (respiration, oxygen saturation, blood pressure, pulse, consciousness, temperature) to identify critically ill patients, including those with sepsis. The Sepsis Six Care Bundle, including six interventions (antibiotics, oxygen therapy, intravenous fluids, blood cultures, measuring urine output and lactate) for high-risk patients is implemented within an hour of identification.
Patients are constantly monitored for vital signs and hemodynamic stabil- ity. Regular ward rounds are conducted for assessment and modification of treatment plans. Fluids are administered based on patient response and condition, with careful monitoring to avoid complications like pulmonary oedema.
Critical patients are admitted to ICU for intensive monitoring and treat- ment, including fluid management and vasopressor administration based on individual patient needs. Vasopressors are used to maintain blood pressure, with their administration and dosage varying based on consul- tant experience and patient condition. Patients in ICU receive continuous monitoring, with treatment adjustments made based on a variety of fac- tors including vital signs, response to treatment, and comorbidities. Once
Exploring the Integration of AI in Sepsis Management 203
patients stabilize, discussions around reducing support like vasopres- sors take place, considering the overall clinical picture and response to treatment.
Overall, the sepsis pathway does not follow a linear trajectory and each treatment is adjusted upon patient presentation and response to medication. The examination of the patients along with a history of symptoms is a funda- mental approach to understanding the infection trajectory and management responsibilities are shared among ICU team members (i.e. consultants, regis- trars, junior doctors and ICU nurses). Nurses oversee the monitoring of the patients, titrating vasopressors and fluids within the range recommended, and flag-up critical cases that will be discussed by the medical team. A stan- dardised pathway and sepsis management are difficult to achieve, given the high variability of patient presentations. We have identified five bottlenecks that affect optimal care; participants discussed a range of opportunities for AI Clinician in the treatment of sepsis and how it could help mitigating these bottlenecks. Table 2 summarises our findings.
Table 2. Gaps in the pathway and mitigation strategies as provided by the AI clinician tool.
Table available in the original PDF paper.
Gaps in the pathway Mitigation strategies through AI Clinician Assessing the volume of fluids/vasopressors is one of the biggest dilemmas.
Sepsis pathway does not follow a linear trajectory and each patient has an individual response to treatment.
Pros: The new tool could be helpful for physicians in starting treatment by giving them precise parameters and objectives for patient care. This function is very helpful when choosing therapies such as fluids and vasopressors but requires further testing.
Cons: Vasopressors are not administered in medical wards; the use of the new tool in medical wards has not been explored.
Some patients are not suitable for ICU (e.g. patients with dementia or not suitable for vasopressors in response of heart conditions) and require assistance in medical wards.
Only experienced nurses are allowed to adjust vasopressors, under the control of consultants and within the boundaries of prescriptions.
Pros: Regardless of their level of experience, bedside nurses could also gain advantages in using this tool. It can help them manage vasopressors within the recommended range and ensure that consultant guidelines are followed. Resuscitation targets that treatment aims to normalize are not clear (e.g. optimal arterial pressure to target, blood lactate, heart rate, urine output, etc.). Continuous patient observation is required to titrate fluids and vasopressors.
Implications of AI Clinician for Decision-Making We have categorised feedback from participants that takes into account their varied roles and responsibilities within the sepsis resuscitation pathway.
The use of the AI tool by bedside nurses involves a critical evaluation of its recommendations, particularly in cases where these suggestions do not align with prescribed targets or lack clear rationale. For instance, P7, a junior staff nurse, conducted a detailed analysis of patient parameters and expressed par- tial agreement with the AI’s vasopressor recommendation. However, they would have ceased vasopressor administration before reaching the target Mean Arterial Pressure (MAP) of >65. They also disagreed with the recom- mendation for fluid boluses, citing concerns about potential fluid overload and suboptimal urine output, suggesting that any fluid administration should be accompanied by other measures, like diuretics. Similarly, a senior charge nurse, P2, indicated a willingness to initiate treatment based on the AI’s recommendation but emphasized the importance of continuous monitoring, especially regarding urine output when administering fluid boluses. After 7 hours, P2 observed that fluids had contributed to achieving the desired MAP target, leading to a recommendation of reducing fluid volume or discontinu- ing vasopressors while monitoring MAP. P2 struggled to understand why the AI recommended maintaining a minimum vasopressor dose even after achiev- ing the target MAP, while agreeing on the rationale behind fluid boluses, albeit with suggestions for continuous urine output monitoring. Similarly, P9, a staff nurse, disagreed with the AI’s fluid bolus recommendation after reaching the desired MAP, opting instead to maintain vasopressor volume while reducing fluids. Overall, while the AI tool provides recommendations, bedside nurses like P7, P2, and P9 demonstrate the importance of contex- tualizing these suggestions within the specific patient scenario, often seeking advice from senior team members and relying on their professional judgment and continuous monitoring to make final treatment decisions.
The use of AI Clinician for doctors finds its best application in concor- dant situations; otherwise, confirmation from colleagues would be sought instead. During night shifts, when the whole medical staff is not available, the AI would be followed only if concordant and sensible, as explained by P8, a foundation year 1 doctor. However, this does not provide reassurance; the patient’s reaction to treatment must be continuously assessed and, as P8 explained, a second opinion from colleagues would be preferred, given their experience and ability to assess the whole clinical picture.
DISCUSSION AND LIMITATIONS This study shows that although the new AI tool offers potential for manag- ing sepsis, it has drawbacks as well. Bedside nurses’ worries, for instance, regarding following AI advice that might not line up with clinical observa- tions or goals that have already been met or that are not fully understood. Furthermore, it was shown that the tool’s recommendations for small vol- ume adjustments had no effect on patient outcomes, indicating the need for more substantial, useful insights. In spite of these challenges, the research indicates that the tool’s most promising application is as an extra viewpoint
Exploring the Integration of AI in Sepsis Management 205
throughout the decision-making process, as opposed to taking the place of clinical judgement. The AI Clinician can be used as a guide by healthcare pro- fessionals, even junior staff members, especially when it comes to identifying abnormal patient parameters. But in order to comprehend the logic of the algorithm and how reinforcement learning improves care, they must possess education and training in AI technology. It is important to incorporate the AI tool into standard operating procedures and make clear its purpose so that it enhances rather than complicates clinical decision-making. It is essential to use AI tools responsibly because overusing them can damage one’s ability to make decisions; the tool can help with de-escalation decisions and treat- ment initiation, but it must be used cautiously, particularly if the clinician’s assessment and the instrument’s suggestions conflict.
Due to its small sample size and singular patient case emphasis, the study has limitations. It does not take into consideration the vast range of patient presentations that are typical of sepsis. This makes it difficult to assess the tool’s potential effectiveness in a standardised clinical context. Further research should consider a more extensive and heterogeneous cohort of sub- jects as well as further patient scenarios. This approach will offer a fuller understanding of the tool’s applicability and effectiveness in routine sepsis management.
CONCLUSION
This study highlights the critical importance of adopting human factors approach to understand the real-world implications of implementing AI tools like the AI Clinician in healthcare settings. The findings advocate for the development of an interface that not only provides actionable insights but also permits healthcare professionals to remain at the forefront of patient care decisions. The integration of AI tools in healthcare can be optimised by taking these factors into account and emphasising user empowerment. This will ensure that the tools are useful technologies that enhance, rather than lessen, the knowledge and independence of healthcare workers.
Ethics Ethics approval was sought, and obtained, from the Imperial College Research Ethics Committee (ICREC reference: 21IC6873).
ACKNOWLEDGMENT
This research has been funded by Artificial Intelligence (AI Award 2020 Phase 2) Round 1 stage 2 and supported by the NIHR, London In-Vitro Diagnostics Cooperative (LIVD), Imperial College London.
REFERENCES
Antonacci, G., Lennox, L., Barlow, J., Evans, L. and Reed, J. (2021) ‘Process mapping in healthcare: a systematic review’, BMC health services research, 21(1), pp. 1–15. Asan, O., Bayrak, A. E. and Choudhury, A. (2020) ‘Artificial intelligence and human
trust in healthcare: focus on clinicians’, Journal of medical Internet research, 22(6), p. e15154.
Avni, T., Lador, A., Lev, S., Leibovici, L., Paul, M. and Grossman, A. (2015) ‘Vasopressors for the treatment of septic shock: systematic review and meta-analysis’, PloS one, 10(8), p. e0129305. Bjerring, J. C. and Busch, J. (2021) ‘Artificial intelligence and patient-centered
decision-making’, Philosophy & Technology, 34, pp. 349–371. Bucknall, T. (2003) ‘The clinical landscape of critical care: nurses’ decision-making’, Journal of advanced nursing, 43(3), pp. 310-319. Byrne, L. and Van Haren, F. (2017) ‘Fluid resuscitation in human sepsis: time to
rewrite history?’, Annals of intensive care, 7, pp. 1–8. Carayon, P. (2006) ‘Human factors of complex sociotechnical systems’, Applied ergonomics, 37(4), pp. 525–535. Carayon, P. (2009) ‘The balance theory and the work system model… Twenty years
later’, Intl. Journal of Human–Computer Interaction, 25(5), pp. 313–327. Carayon, P., Wooldridge, A., Hoonakker, P., Hundt, A. S. and Kelly, M. M. (2020) ‘SEIPS 3.0: Human-centered design of the patient journey for patient safety’, Applied ergonomics, 84, p. 103033. Cohen, J., Vincent, J.-L., Adhikari, N. K., Machado, F. R., Angus, D. C., Calandra,
T., Jaton, K., Giulieri, S., Delaloye, J. and Opal, S. (2015) ‘Sepsis: A roadmap for future research’, The Lancet infectious diseases, 15(5), pp. 581–614. Fleischmann, C., Scherag, A., Adhikari, N., Hartog, C., Tsaganos, T., Schlattmann, P., Angus, D. and Reinhart, K. (2015) ‘Global burden of sepsis: A systematic review’, Critical Care, 19, pp. 1–201. Ganz, F. D., Engelberg, R., Torres, N. and Curtis, J. R. (2016) ‘Development of a
model of interprofessional shared clinical decision making in the ICU: A mixed- methods study’, Critical care medicine, 44(4), pp. 680–689. Gerke, S., Minssen, T. and Cohen, G. (2020) ‘Ethical and legal challenges of artificial intelligence-driven healthcare’, Artificial intelligence in healthcare: Elsevier, pp. 295–336. Gotts, J. E. and Matthay, M. A. (2016) ‘Sepsis: pathophysiology and clinical
management’, Bmj, 353. Jacobs, M., Pradier, M. F., McCoy Jr, T. H., Perlis, R. H., Doshi-Velez, F. and Gajos, K. Z. (2021) ‘How machine-learning recommendations influence clini- cian treatment selections: The example of antidepressant selection’, Translational psychiatry, 11(1), pp. 108. Kalman, H. K. (2002) ‘Process mapping: Tools, techniques, & critical success
factors’, Performance Improvement Quarterly, 15(4), pp. 57–73. Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. and Faisal, A. A. (2018) ‘The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care’, Nature medicine, 24(11), pp. 1716–1720. Lee, D. and Yoon, S. N. (2021) ‘Application of artificial intelligence-based tech-
nologies in the healthcare industry: Opportunities and challenges’, International Journal of Environmental Research and Public Health, 18(1), p. 271. Levy, M. M., Evans, L. E. and Rhodes, A. (2018) ‘The surviving sepsis campaign bundle: 2018 update’, Intensive care medicine, 44, pp. 925–928. Mackenzie, D. C. and Noble, V. E. (2014) ‘Assessing volume status and fluid respon-
siveness in the emergency department’, Clinical and experimental emergency medicine, 1(2), p. 67. Marik, P. and Bellomo, R. (2016) ‘A rational approach to fluid therapy in sepsis’, BJA: British Journal of Anaesthesia, 116(3), pp. 339–349.
Exploring the Integration of AI in Sepsis Management 207
Massimo Micocci, Adam L Gordon, A. Joy Allen, Timothy Hicks, Patrick Kierkegaard, Anna McLister, Simon Walne, Gail Hayward, Peter Buckle, on behalf of the CONDOR study team (2021) ‘COVID-19 testing in English care homes and implications for staff and residents’, Age and Ageing, pp. 1–5. Mcknight, D. H., Carter, M., Thatcher, J. B. and Clay, P. F. (2011) ‘Trust in a specific
technology: An investigation of its components and measures’, ACM Transactions on management information systems (TMIS), 2(2), pp. 1–25. Rajpurkar, P., Chen, E., Banerjee, O. and Topol, E. J. (2022) ‘AI in health and medicine’, Nature medicine, 28(1), pp. 31–38. Ritchie, J., Lewis, J., Nicholls, C. M. and Ormston, R. (2013) Qualitative research
practice: A guide for social science students and researchers. sage. Sujan, M., Furniss, D., Grundy, K., Grundy, H., Nelson, D., Elliott, M., White, S., Habli, I. and Reynolds, N. (2019) ‘Human factors challenges for the safe use of artificial intelligence in patient care’, BMJ health & care informatics, 26(1).
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Artificial Intelligence and Social Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-98-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional