TL;DR
This paper explores using pre-trained language models (PLMs) in Spanish to predict health insurance coverage for labor accidents. They found that fine-tuning existing clinical PLMs on provider-specific data yields strong results, comparable to custom-trained models but with significantly less computational cost and time. This work also introduces what they believe to be the largest clinical text corpus in Spanish to date.
What Problem Does This Paper Address?
Clinical Natural Language Processing (NLP) holds immense potential for extracting valuable information from electronic health records, aiding in decision-making. However, the complexity of clinical language, filled with jargon and abbreviations, makes automated analysis challenging. While the field has seen a surge in pre-trained language models (PLMs) since 2017, most research has focused on English. This paper addresses the gap in Spanish clinical NLP, specifically for the task of predicting health insurance coverage for labor accidents. The current system relies on limited diagnostic information, missing the richer context available in admission and medical reports. The authors aim to leverage PLMs to improve the accuracy and scope of these coverage predictions, easing the burden on employers and potentially streamlining the claims process.
Methods
The researchers experimented with three main approaches using Spanish clinical PLMs:
- Fine-tuning an existing clinical PLM: They used
bsc-bio-ehr-es, a Spanish PLM pre-trained on biomedical and clinical text, and fine-tuned it on admission reports, medical reports, and a combined dataset. - Continual pre-training: They took the
bsc-bio-ehr-esPLM and further pre-trained it on the provider's admission and medical report data before fine-tuning. - Pre-training from scratch: They trained a new PLM from scratch using a large corpus of admission and medical reports from the provider.
All models were evaluated on their ability to predict insurance coverage.
Datasets
The study utilized several datasets derived from a single health insurance provider specializing in labor accidents:
-
Fine-tuning Datasets:
- Admission reports (300k documents, 22.5M tokens)
- Medical reports (300k documents, 26.3M tokens)
- Combined admission and medical reports (300k documents, 57.2M tokens) These datasets included coverage decision labels.
-
Continual Pre-training Datasets:
- Admission reports (1.5M documents, 112.6M tokens)
- Medical reports (1.2M documents, 154.0M tokens)
- Combined admission and medical reports (855k documents, 164.6M tokens) These datasets were used for further pre-training and did not require labels.
-
Pre-training from Scratch Dataset:
- Combined admission and medical reports (7.1M documents, 1.03B tokens) This large dataset was used to train a PLM from the ground up and is noted as the largest clinical text corpus in Spanish to date.
Key Insights and Ideas
- Fine-tuning is often sufficient: For this specific task, fine-tuning a publicly available clinical PLM (
bsc-bio-ehr-es) on the provider's data achieved performance comparable to more computationally intensive methods like continual pre-training or training from scratch. This suggests that for certain downstream tasks, leveraging existing, well-trained models can be highly efficient. - Provider-specific data matters: While general clinical PLMs are useful, fine-tuning them with data specific to the insurance provider's domain (admission and medical reports) significantly improved performance.
- Medical reports offer richer information: Models trained on medical reports generally outperformed those trained on admission reports, indicating that the physician's notes contain more critical clinical details for coverage prediction.
- Combined data is best: Models that incorporated both admission and medical reports consistently performed the best, highlighting the value of integrating information from different stages of patient care.
- Early prediction benefits: The ability to predict coverage before a physician's final diagnosis could help physicians focus on details that might influence their decision, and also allow patients to better understand their situation.
Key Results
- All fine-tuned models achieved high AUC scores, with the best performing models (continual PLM and custom PLM) reaching around 96.3% AUC.
- The
Admission+medicalmodels generally outperformedAdmissionorMedicalonly models across all approaches. - The custom PLM trained from scratch on 7.1 million documents achieved the highest AUC (96.3%), but was the most time-consuming and computationally expensive to train.
- Fine-tuning
bsc-bio-ehr-eson the combined admission and medical reports yielded an AUC of 96.1%, very close to the custom PLM, but with significantly less resource investment. - A shadow deployment comparing the custom PLM to the current model showed a potential increase in coverage classification by 20-24%, with estimated annual savings of 1.5 to 2.5MM US.
Limitations and Future Work
- Architecture: The custom PLM used the same architecture as
bsc-bio-ehr-es. Exploring different architectures could potentially yield better results. - Data Generalization: The study used data from only one provider. Expanding to data from multiple providers could improve the generalizability of the models.
- Task Specificity: The custom PLM was not compared to other PLMs on broader language tasks like Named Entity Recognition (NER) or question answering. Such comparisons could reveal if the custom PLM offers advantages in those areas.
- Bias: The authors acknowledge that the models were not tested for bias related to protected fields, meaning they could inadvertently favor certain patients or accidents. This is a critical area for future work to ensure equitable decision-making.
- Privacy: The data was not anonymized for internal use. The authors suggest keeping the model private due to the risk of privacy attacks extracting sensitive information.
Summary updated