TL;DR
This paper tackles the challenge of automatically predicting patient triage levels in real-world emergency rooms using machine learning. By leveraging anonymized, multi-modal ambulance data, the researchers explore how transformer-based models can handle noisy, unbalanced, and semi-structured data, ultimately aiming to improve the efficiency and accuracy of emergency care prioritization.
What Problem Does This Paper Address?
In emergency departments, quickly and accurately prioritizing patients based on the urgency of their condition is critical. This process, known as triage, typically involves assigning patients to one of several urgency levels within the first 10 minutes of arrival. While established systems like the Manchester Triage System (MTS) provide a framework, the real-world application is complex. The MTS uses a decision tree with "diagrams" (diagnoses) and "indications" (discriminators), but these are not always sharply defined, leading to potential variations in how medical professionals assess urgency. Furthermore, training machine learning models for this task is difficult due to the inherent noise, missing data, unbalanced class distributions (long-tail problem), and the multi-modal nature of the data (combining text with structured vital signs and measurements). This paper aims to bridge the gap between these complex real-world scenarios and the development of effective ML-based triage support systems.
Methods
The core of this work revolves around adapting and evaluating transformer-based models, specifically medBERT.de, for triage prediction. The researchers experiment with several approaches:
- Training Strategies: They compare standard cross-entropy loss with weighted cross-entropy loss to address class imbalance. They also investigate single-task models (training for each class independently) versus multi-task models (training for all classes simultaneously), where the focus gradually shifts to the target class.
- Input Data Integration: Recognizing that text alone is insufficient, they explore methods to incorporate structured data (vital signs, patient demographics, etc.) into the BERT models.
- "Extra as Text": Structured data is manually translated into natural language sentences and appended to the original text using special tokens (
[SEP]). - "Extra as Feature": Structured data is processed through MLPs and then combined with BERT embeddings in a custom classification head. This approach avoids manual translation bias.
- "Expert": A rule-based component, leveraging medical expert knowledge, is integrated to process structured information.
- Hybrid Approach: They also test combining machine learning with a rule-based component.
Datasets
The study utilizes anonymized, real-world ambulance reports from a German emergency ward, spanning two years and encompassing over 18,000 cases. This dataset is characterized by:
- Multi-modal Data: A mix of structured information (around 600 features, including vital signs, demographics, medical history) and unstructured text (emergency situation descriptions, initial diagnoses, symptoms).
- Real-world Noise: Data collected by various paramedics, potential inconsistencies, and symptoms that may resolve between data collection and hospital arrival.
- Unbalanced Labels: A long-tail distribution of triage labels, diagrams, and indications, with many labels occurring infrequently.
- MTS-based Labels: The data is labeled according to the Manchester Triage System (MTS), with 54 diagrams, 125 indications, and 5 triage levels. However, the real-world dataset exhibits a subset of these labels.
Key Insights and Ideas
- Multi-modal Data is Crucial: Text features alone are often insufficient for accurate triage. Structured data like temperature or oxygen saturation are essential for specific indications (e.g., "Very Hot" requires a temperature above 41°C).
- Multi-task Learning Outperforms Single-task: Training models to predict all triage-related labels (diagram, indication, triage) simultaneously leads to better performance, especially for the macro F1 scores, indicating improved generalization across less frequent classes.
- "Extra as Text" vs. "Extra as Feature": While "extra as text" is intuitive, the "extra as feature" approach, which directly processes structured data, avoids manual translation bias and is more scalable due to BERT's input limitations.
- Rule-based "Expert" Systems Offer Value: Integrating expert knowledge, even in a simple form, can significantly boost performance, particularly for the macro F1 scores, highlighting the importance of domain expertise in handling data scarcity.
- Top-3 Predictions Improve Performance: Considering the top-3 predicted labels, rather than just the top-1, substantially improves performance metrics, suggesting that the model's confidence distribution is valuable.
Key Results
- Multi-task models generally perform better than single-task models.
- The "expert" model (rule-based integration of structured data) showed notable improvements, especially in macro F1.
- Hybrid models combining "expert" and "extra" features (text or feature-based) generally provided the best overall approach. The model combining expert and extra-text features showed strong performance.
- Direct prediction of triage labels (Triage (P)) often yielded slightly better weighted and macro F1 scores than inferring triage from predicted indications (Triage (D)). However, the authors note that Triage (D) might be more representative of real-world decision-making and achieved a better MRSE, suggesting it provides labels closer to the gold standard.
- Considering top-3 predictions significantly boosted performance. For example, weighted scores for indications improved, though macro scores remained lower, likely due to the long-tail problem.
Limitations and Future Work
- Noise and Missing Labels: The inherent noise and missing data in real-world ambulance reports remain significant challenges.
- Long-Tail Problem: The highly unbalanced distribution of labels, especially for indications, makes it difficult to achieve high performance across all classes.
- Abstract Labels: Some labels, like "Recent Problem," are very general and difficult to predict solely from text.
- Scalability of "Extra as Text": The manual translation approach for structured data is not scalable due to model input limitations.
- Focus on Urgent Cases: The authors emphasize the need to investigate how to improve performance specifically for "really urgent cases," as misclassification here can have severe consequences.
- Future Work: The authors suggest further research into integrating additional structured information as "extra features" to boost performance. They also propose exploring hybrid machine learning systems that combine data-driven approaches with rule-based components to better handle the long-tail problem and improve predictions for critical cases. Investigating how to better represent the correlations between labels and improving the handling of abstract labels are also important future directions.
Summary updated