Rule-based and reinforcement-learning policies for resource allocation in a pediatric echo lab: a simulation study

arXiv:2506.06297v2 Announce Type: replace Abstract: Scheduling echocardiographic examinations in a hospital setting presents significant challenges due to non-deterministic factors such as patient no-shows, variable arrival times, and heterogeneous exam durations, as well as asymmetric resource constraints between fetal and non-fetal patient streams. Fetal cases involve specialized ultrasound examinations performed during pregnancy, whereas non-fetal cases correspond to general diagnostic echocardiography services. Using one week (184 patients) of operational data from the Echo Laboratory at Stanford University's Lucile Packard Children's Hospital, we estimate patient no-show probabilities and derive empirical distributions of arrival times and exam durations, and use them to build a discrete-event stochastic simulation model using SimPy, exposed through an open-source Gymnasium interface (https://github.com/desResLab/echo_rl). As a baseline for policy optimization, we systematically compare two scheduling paradigms: on-the-fly allocation, in which patients are served according to their arrival time without prior reservations, and reservation-based allocation, in which varying proportions of resources are preassigned. We consider a hospital with seven ultrasound rooms (one equipped for both fetal and non-fetal exams, six for non-fetal exams only) and six sonographers (four qualified for both exam types, two for non-fetal exams only). In this setting, on-the-fly allocation markedly reduces the average patient waiting time compared with every reservation-based configuration tested. Building on this foundation, we train a Dueling Double Deep Q-Network (DQN) with prioritized experience replay to approximate an optimal dynamic on-the-fly allocation policy. The learned RL policy is comparable to, but does not outperform, the best on-the-fly rule-based policy.