The anatomy of clinical decision-making: aligning AI design with ICU routines

In the face of numerous explanations for why AI-driven decision support systems (DSS) have failed to deliver on their promise of improving organizational decision-making, this paper problematizes the under-theorized mismatch between the design of DSS and the actual decision-making processes that the technology is supposed to support. We examine this mismatch by studying the implementation of a DSS in the intensive care unit (ICU) of a large academic hospital. Based on 27 months of ethnographic fieldwork, we contend that the studied DSS was designed on the assumption that individual intensivists are responsible for making life-critical discharge decisions at one particular moment in time. However, our study of actual decision-making practices reveals that discharge decision-making is instead a protracted process, involving multiple actors fragmented across time and space. To account for these complexities, we advocate for a ‘dynamic routines’ perspective, which highlights the actual patterns of action pursued throughout a clinical decision-making process. Our application of this perspective contributes to a more granular understanding of discharge decision-making, which can help future DSS designers better grasp the peculiarities and complexities—or ‘anatomy’—of the decision-making process. We also suggest integrating an ‘anticipatory ethnographic approach’ into the design and pre-implementation phases of future DSS to help bridge the current gap between design assumptions and actual decision-making practices.