Abstract
Deep learning models in healthcare often achieve high accuracy but produce "physiologically impossible" errors, such as instantaneous transitions between non-adjacent sleep stages. This paper introduces the Rule-based Reinforcement Learning Layer (RRLL), a lightweight, "plug-and-play" module that leverages reinforcement learning and domain-specific rules to correct nonsensical predictions. By penalizing transitions that violate physiological constraints, RRLL improves both predictive accuracy and clinical reliability across tasks like sleep staging and seizure detection.
Introduction
In clinical settings, many physiological processes follow strict sequential rules. For instance, human sleep progresses through specific stages; a direct transition from a "Wake" state to "REM" sleep is biologically impossible. However, standard deep learning models, typically trained on independent and identically distributed (IID) data, often ignore these temporal constraints. While a model might reach 90% accuracy, the remaining 10% of errors frequently manifest as "spiky," biologically impossible jumps that erode clinician trust and complicate real-world deployment.
Methodology
The Rule-based Reinforcement Learning Layer (RRLL) is designed as a modular "sanity check" that can be added to any pre-trained, frozen predictor. The framework operates in three main stages:
- Output Tentative Labels: A base predictor (e.g., a CNN or Transformer) generates a tentative label and a feature vector for each input.
- RL Reassigned Labels: An RL agent, trained via the REINFORCE algorithm, observes the current state—comprising the tentative label, features, and the previous assigned label—and takes an action to either maintain or reassign the label. The reward function is the core innovation: it heavily penalizes "impossible transitions" defined by domain experts (e.g., Ictal $\not\to$ Preictal in seizure detection).
- Rule-based Inference: During deployment, the trained RRLL acts as a learned temporal filter, smoothing predictions to ensure they align with physiological reality.
Experimental Evaluation
The researchers evaluated RRLL on two major healthcare challenges:
- SHHS (Sleep Heart Health Study): A dataset containing over 42,000 hours of EEG recordings for five-stage sleep classification.
- CHB-MIT Seizure Database: 844 hours of scalp EEG data for detecting the onset of epileptic seizures.
Key Results and Insights
The RRLL showed significant improvements across both domains:
- Sleep Staging: On the SHHS dataset, RRLL outperformed state-of-the-art models like SleepFormer. Notably, it improved the F1-score for the difficult "N1" sleep stage from approximately 0.47 to 0.64.
- Seizure Detection: RRLL achieved an accuracy of 98.1%, significantly higher than the base DCRNN model (93.9%).
- Robustness: The authors found that RRLL remains robust even when the base model overfits, consistently pruning away the nonsensical predictions that emerge during training decay.
- Qualitative Success: Visualizations (hypnograms) confirmed that RRLL successfully removed the "spiky" transitions that characterize model errors, producing sequences that mirror real human physiological data.
Conclusion
RRLL demonstrates that defining what is "impossible" is often simpler and more effective than modeling the entire complexity of disease progression. By focusing on a small set of impossibility rules, this lightweight layer provides a robust, plug-and-play solution for enhancing the clinical utility and reliability of deep learning models in healthcare.
Summary updated