Extracting information from free text through unsupervised graph-based clustering: an application to patient incident records

M. Tarik Altuncu, Eloise Sorin, Joshua D. Symons, Erik Mayer, Sophia N. Yaliraki, Francesca Toni, Mauricio BarahonaarXiv 2019 · OpenReview

Abstract

The large volume of text in electronic healthcare records often remains underused due to a lack of methodologies to extract interpretable content. Here we present an unsupervised framework for the analysis of free text that combines text-embedding with paragraph vectors and graph-theoretical multiscale community detection. We analyse text from a corpus of patient incident reports from the National Health Service in England to find content-based clusters of reports in an unsupervised manner and at different levels of resolution. Our unsupervised method extracts groups with high intrinsic textual consistency and compares well against categories hand-coded by healthcare personnel. We also show how to use our content-driven clusters to improve the supervised prediction of the degree of harm of the incident based on the text of the report. Finally, we discuss future directions to monitor reports over time, and to detect emerging trends outside pre-existing categories.

Introduction

The healthcare industry possesses a vast repository of unstructured data, primarily in the form of free-text incident reports. In the UK’s National Health Service (NHS), the National Reporting and Learning System (NRLS) has accumulated over 13 million records describing accidents, near-misses, and equipment failures. Despite their potential, these records are rarely utilized for large-scale analysis. Manual categorization is labor-intensive, prone to human error, and often constrained by rigid taxonomies that fail to capture clinical nuances. Furthermore, "Degree of Harm" (DoH) labels—essential for prioritizing safety interventions—are frequently inconsistent. This research introduces an automated framework to group reports based on semantic content and improve the prediction of incident severity.

Methodology

The proposed pipeline transitions from raw unstructured text to structured clinical insights through four primary stages:

  1. Text Embedding (Doc2Vec): Utilizing the Doc2Vec framework, each report is represented as a high-dimensional vector. Unlike traditional word-count methods, this captures semantic context, recognizing relationships between terms like "scalpel" and "surgical knife."
  2. Graph Construction (MST-kNN): Reports are treated as nodes in a graph. To manage noise and ensure global connectivity while highlighting local clusters, the authors employed a combination of a Minimum Spanning Tree (MST) and k-Nearest Neighbors (kNN).
  3. Multiscale Clustering (Markov Stability): This core innovation uses a simulated diffusion process to identify natural groupings. By observing how information spreads across the graph over different time scales, the algorithm identifies clusters at multiple levels of granularity, from broad themes to specific sub-topics.
  4. Supervised Augmentation: The unsupervised cluster assignments are integrated as features into standard classifiers (e.g., SVM, Random Forest) to enhance the prediction of the Degree of Harm.
The multi-stage pipeline for unsupervised text clustering and supervised harm classification.

Datasets

The study utilized several key data sources:

  • NRLS Dataset: 13 million anonymized incident reports from the NHS (2004–2016).
  • Case Study Subset: 3,229 reports from St Mary’s Hospital, London, for detailed clustering analysis.
  • Prediction Subset: 4,064 records from trusts with varying reporting cultures to test the DoH prediction model.
  • Wikipedia Benchmark: 5 million articles used to compare generic language models against domain-specific ones.

Key Insights and Results

Domain Specificity and Robustness

A critical finding was that domain-specific training is essential. Models trained on the 13 million medical records significantly outperformed those trained on general Wikipedia text. The specific jargon and "voice" of medical incident reporting require specialized embeddings to achieve high textual consistency. Furthermore, the method proved robust across different dataset sizes; performance remained stable whether the model was trained on 1 million or 13 million records.

Performance comparison of hyper-parameters and training corpora (NRLS vs. Wikipedia).

Clinical Relevance vs. Procedural Labels

While manual hand-coded categories are often procedural (e.g., "Documentation"), the unsupervised method identified clusters based on clinical content (e.g., "radiotherapy" or "labour ward"). These content-driven clusters often cut across multiple manual categories, providing a more practical lens for clinical safety leads. The Markov Stability approach also revealed a "quasi-hierarchy," where specific clusters naturally merged into broader themes as the analysis scale increased.

Performance Metrics

  • Prediction Accuracy: Integrating Doc2Vec embeddings with Markov Stability cluster labels improved the F1 score for predicting Degree of Harm by up to 100% compared to using categorical features alone.
  • Human-Level Consistency: In validation tests, the automated model achieved agreement levels with original labels comparable to those of human clinicians (who agreed with original labels only ~59% of the time), suggesting the model performs at a near-human level for subjective tasks.

Conclusion

This framework demonstrates that "dark data" in healthcare can be unlocked without massive manual labeling efforts. By combining deep-learning embeddings with graph-theoretical clustering, the system provides a scalable way to monitor safety trends. Future work will focus on improving the interpretability of these "black box" embeddings and developing real-time systems to assist staff with classification at the point of data entry.

Summary updated

Abstract

The large volume of text in electronic healthcare records often remains underused due to a lack of methodologies to extract interpretable content. Here we present an unsupervised framework for the analysis of free text that combines text-embedding with paragraph vectors and graph-theoretical multiscale community detection. We analyse text from a corpus of patient incident reports from the National Health Service in England to find content-based clusters of reports in an unsupervised manner and at different levels of resolution. Our unsupervised method extracts groups with high intrinsic textual consistency and compares well against categories hand-coded by healthcare personnel. We also show how to use our content-driven clusters to improve the supervised prediction of the degree of harm of the incident based on the text of the report. Finally, we discuss future directions to monitor reports over time, and to detect emerging trends outside pre-existing categories.
BibTeX
@misc{altuncu2019extracting,
  author = {M. Tarik Altuncu and Eloise Sorin and Joshua D. Symons and Erik Mayer and Sophia N. Yaliraki and Francesca Toni and Mauricio Barahona},
  title = {{Extracting information from free text through unsupervised graph-based clustering: an application to patient incident records}},
  year = {2019},
  eprint = {1909.00183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.00183},
}
Figure