Mining Unstructured Medical Texts With Conformal Active Learning

Juliano Genari, Guilherme Tegoni Goedert

Abstract

The extraction of relevant data from Electronic Health Records (EHRs) is crucial to identifying symptoms and automating epidemiological surveillance processes. By harnessing the vast amount of unstructured text in EHRs, we can detect patterns that indicate the onset of disease outbreaks, enabling faster, more targeted public health responses. Our proposed framework provides a flexible and efficient solution for mining data from unstructured texts, significantly reducing the need for extensive manual labeling by specialists. Experiments show that our framework achieving strong performance with as few as 200 manually labeled texts, even for complex classification problems. Additionally, our approach can function with simple lightweight models, achieving competitive and occasionally even better results compared to more resource-intensive deep learning models. This capability not only accelerates processing times but also preserves patient privacy, as the data can be processed on weaker on-site hardware rather than being transferred to external systems. Our methodology, therefore, offers a practical, scalable, and privacy-conscious approach to real-time epidemiological monitoring, equipping health institutions to respond rapidly and effectively to emerging health threats.

Introduction

Epidemiological surveillance—tracking health indicators to detect disease outbreaks—is a cornerstone of public health. While Electronic Health Records (EHRs) contain a wealth of information, much of it is trapped in unstructured clinical notes. Manually extracting this data is prohibitively expensive and slow for real-time applications. Furthermore, privacy regulations often prevent the use of cloud-based AI services, requiring solutions that can run on-premise on limited hardware. This paper introduces a "Conformal Active Learning" framework designed to address these challenges by minimizing manual labeling effort while providing rigorous prediction guarantees.

Methodology: Conformal Active Learning

The core of our approach is an iterative "human-in-the-loop" pipeline that maximizes the informational value of every manual label provided by a specialist.

The Conformal Active Learning iterative framework.

The framework consists of several key stages:

  1. Conformal Prediction Calibration: Instead of standard point estimates, the model uses label-conditional conformal prediction to output sets of labels. This provides a mathematical guarantee (e.g., 90% confidence) that the true label is included, with the set size serving as a rigorous measure of uncertainty.
  2. Uncertainty Quantification and Clustering: Unlabeled texts are ranked by uncertainty. To ensure diversity in the labeling batch, the top $k$ most uncertain texts are clustered (using k-means on embeddings), and only the cluster centroids are presented to the human expert.
  3. Model-Agnostic Design: The framework is compatible with various architectures. We evaluated both a lightweight TF-IDF + XGBoost pipeline and a transformer-based model (DeBERTaV3).
  4. OLIM Interface: To facilitate deployment, we developed the Open Labeller for Iterative Machine learning (OLIM), a Dockerized web interface for managing the active learning loop locally.

Key Insights and Results

Our experiments, conducted using Amazon Product Reviews as a linguistically complex proxy for EHR data, yielded several critical insights for low-resource clinical settings.

The Advantage of Lightweight Models

Surprisingly, the lightweight TF-IDF + XGBoost model frequently outperformed the more complex DeBERTaV3 transformer in low-data regimes. With only 100-200 labels, the simpler model achieved an AUC-ROC of 0.94 for common categories. This suggests that for many clinical classification tasks, massive deep learning models may be unnecessary, favoring simpler models that are easier to deploy on hospital hardware.

The "Mixed Uncertainty" Strategy

A pivotal finding was that labeling only the most uncertain examples is sub-optimal. By mixing highly uncertain texts with a fraction of low-uncertainty "easy" cases (a 70/30 split), we observed a dramatic improvement in performance and stability.

Comparison of model performance using high uncertainty vs. mixed uncertainty sampling.

As shown in the figure above, the mixed uncertainty approach allowed the model to reach high AUC-ROC scores much faster than focusing solely on high-uncertainty samples. This strategy helps the model validate straightforward cases and prevents it from over-fitting to outliers.

Overcoming the Cold-Start Problem

For rare or highly imbalanced classes, active learning can struggle to find initial positive examples. We found that "seeding" the process with a small number of pre-labeled examples—identified through simple keyword searches—effectively jumpstarts the learning process. For rare labels where random sampling failed (AUC-ROC ~0.50), our seeded active learning framework achieved an AUC-ROC of 0.88 with just 100 manual labels.

Conclusion

The Conformal Active Learning framework offers a scalable, privacy-preserving solution for mining unstructured medical texts. By combining rigorous uncertainty quantification with strategic sample selection, the system achieves high performance with minimal human intervention. This approach empowers health institutions to perform real-time epidemiological monitoring on-site, ensuring both rapid response capabilities and patient data security.

Summary updated

Abstract

The extraction of relevant data from Electronic Health Records (EHRs) is crucial to identifying symptoms and automating epidemiological surveillance processes. By harnessing the vast amount of unstructured text in EHRs, we can detect patterns that indicate the onset of disease outbreaks, enabling faster, more targeted public health responses. Our proposed framework provides a flexible and efficient solution for mining data from unstructured texts, significantly reducing the need for extensive manual labeling by specialists. Experiments show that our framework achieving strong performance with as few as 200 manually labeled texts, even for complex classification problems. Additionally, our approach can function with simple lightweight models, achieving competitive and occasionally even better results compared to more resource-intensive deep learning models. This capability not only accelerates processing times but also preserves patient privacy, as the data can be processed on weaker on-site hardware rather than being transferred to external systems. Our methodology, therefore, offers a practical, scalable, and privacy-conscious approach to real-time epidemiological monitoring, equipping health institutions to respond rapidly and effectively to emerging health threats.
BibTeX
@misc{genari2025mining,
  author = {Juliano Genari and Guilherme Tegoni Goedert},
  title = {{Mining Unstructured Medical Texts With Conformal Active Learning}},
  year = {2025},
  eprint = {2502.04372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.04372},
}
Figure