Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework

Niloufar Eghbali, Tuka Alhanai, Mohammad M. GhassemiAAAI 2025 · DOI

TL;DR

This paper introduces ConformalDQN, a novel AI framework that uses conformal prediction to provide reliable uncertainty estimates for mechanical ventilation decisions in intensive care units. By integrating this uncertainty quantification with deep reinforcement learning, ConformalDQN aims to improve patient safety and outcomes by avoiding potentially harmful actions and handling situations where the AI encounters data it hasn't seen before.

What Problem Does This Paper Address?

Mechanical ventilation (MV) is a life-saving intervention in intensive care units (ICUs), but setting the right parameters is incredibly complex. Incorrect settings can lead to serious complications like ventilator-induced lung injury (VILI), increasing mortality risk. While reinforcement learning (RL) offers a promising avenue for optimizing these settings, a major challenge is the "distribution shift" problem. This occurs when the data the AI encounters during real-world deployment differs from the data it was trained on. This can lead to the AI making unreliable or even dangerous decisions, especially in the dynamic and unpredictable environment of an ICU. Existing RL approaches often struggle to quantify their own uncertainty, making it difficult to trust their recommendations in critical situations.

Methods

The core of this work is ConformalDQN, which builds upon the Double Deep Q-Network (DDQN) architecture. Here's a breakdown of the key components:

  • DDQN for Q-value Estimation: This is the standard RL component that learns to estimate the expected future rewards (Q-values) for taking specific actions (ventilator settings) in given patient states.
  • Action Probability Estimator: A network is trained to estimate the probability of actions taken by the "behavioral policy" (in this case, likely physician actions from the dataset). This is crucial for the conformal prediction step.
  • Conformal Predictor: This is the novel part. It uses a calibration dataset to determine a "conformity threshold." This threshold acts as a measure of how "unusual" or uncertain a prediction is. It provides distribution-free uncertainty estimates, meaning it doesn't rely on specific assumptions about the data distribution.
  • Uncertainty-Aware Action Selection: The final action selection process integrates both the Q-values from DDQN and the uncertainty estimates from the conformal predictor. Actions are only considered if they are both high-value (according to Q-values) and fall within the "confident" set defined by the conformity threshold. If no actions meet the confidence threshold, the standard Q-learning selection is used. This mechanism helps the model avoid actions it's uncertain about, even if they have high Q-values.

The training involves a composite loss function that combines the standard DDQN loss, a negative log-likelihood loss for the behavioral policy actions, and L2 regularization to prevent overconfident predictions.

Datasets

  • MIMIC-IV Database: This is the primary dataset used. It's a large, de-identified clinical database from an academic medical center. The researchers extracted data from 29,270 patients who underwent mechanical ventilation during their ICU stay.
  • Out-of-Distribution (OOD) Dataset: To specifically test robustness, an OOD dataset was created by identifying patients whose initial features fell within the top or bottom 1% of the distribution at ventilation onset. This dataset represents approximately 20% of the total patient data.

Key Insights and Ideas

  • Conformal Prediction for Safety: The paper highlights how conformal prediction can be a powerful tool for ensuring safety in AI-driven healthcare. By providing a quantifiable measure of uncertainty, it allows the AI to be more cautious when it's unsure, which is critical in life-support scenarios.
  • Mitigating Distribution Shift: The core contribution is demonstrating that ConformalDQN can effectively mitigate the risks associated with distribution shift. Unlike standard DDQN, which tends to overestimate Q-values in out-of-distribution scenarios, ConformalDQN maintains stable Q-values, suggesting more reliable decision-making.
  • Mimicking Clinician Behavior: The action distribution analysis shows that ConformalDQN's recommendations closely align with physician practices, particularly in adopting lung-protective strategies (lower PEEP, conservative tidal volumes). This alignment is crucial for clinical adoption.
  • Tunable Conservatism: A key advantage is that the confidence threshold can be adjusted post-training without retraining the model. This allows for flexible tuning of the exploration-conservatism trade-off, enabling the agent to explore more or be more conservative as needed.

Key Results

  • Improved Patient Outcomes: ConformalDQN demonstrated a stronger negative correlation between predicted Q-values and patient mortality compared to other baselines (CQL, DQN, physician policy), suggesting it can better predict lower mortality rates.
  • Higher Mean Q-Values: Evaluated using Fitted Q-Evaluation (FQE), ConformalDQN achieved the highest mean Q-value (0.639 ± 0.026), indicating better expected long-term patient outcomes (specifically 90-day survival rate).
  • Robustness in OOD Scenarios: ConformalDQN showed stable Q-value estimates below the overestimation threshold in both in-distribution (ID) and out-of-distribution (OOD) data, outperforming standard DDQN which exhibited significant overestimation in OOD cases.
  • Clinically Relevant Actions: The action distributions closely mirrored physician policies, favoring lung-protective settings.

Limitations and Future Work

  • Discrete Action Space: The current framework uses a discretized action space for ventilator parameters. The authors suggest extending this to continuous action spaces to capture finer-grained control and potentially improve performance.
  • State-Conditioned Conformal Prediction: Future work could involve developing a state-conditioned conformal predictor. This would allow the uncertainty estimates to adapt more dynamically based on the specific patient's current state, leading to more nuanced and context-specific decision-making.
  • Clinical Applicability: While promising, further efforts are needed to make the model clinically applicable, likely involving more extensive validation and integration into clinical workflows.

Summary updated

BibTeX
@inproceedings{eghbali2025distribution,
  author = {Niloufar Eghbali and Tuka Alhanai and Mohammad M. Ghassemi},
  title = {{Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35013},
}
Figure