Using critiquing for improving medical protocols: Harder than it seems

Medical protocols are widely recognised to provide clinicians with high-quality and up-to-date recommendations. A critical condition for this is of course that the protocols themselves are of high quality. In this paper we investigate the use of critiquing for improving the quality of medical protocols. We constructed a detailed formal model of the jaundice protocol of the American Association of Pediatrics in the Asbru representation language. We recorded the actions performed by a pediatrician while solving a set of test cases. We then compared these expert actions with the steps recommended by the formalised protocol, and analysed the differences that we observed. Even our relatively small test set of 7 cases revealed many mismatches between the actions performed by the expert and the protocol recommendations, which suggest improvements of the protocol. A major problem in our case study was to establish a mapping between the actions performed by the expert and the steps suggested by the protocol. We discuss the reasons for this difficulty, and assess its consequences for the automation of the critiquing process.