Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs

Teodor-Calin Ionescu, Lifeng Han, Jan Heijdra Suasnabar, Anne Stiggelbout, Suzan Verberne

TL;DR

This paper explores how advanced AI techniques, specifically neural topic modeling (BERTopic) and Large Language Models (LLMs), can automatically extract meaningful themes from cancer patient interviews. The goal is to provide clinicians with concise, patient-centered insights to improve healthcare practices and patient experiences.

What Problem Does This Paper Address?

Healthcare often relies on structured medical data, potentially overlooking the rich, nuanced experiences and emotional journeys of patients. Extracting these insights from lengthy, unstructured patient narratives (like interview transcripts) is a significant challenge. This paper aims to bridge this gap by developing an automated system that can efficiently identify key themes and concerns from patient interviews, making this valuable information accessible to clinicians.

Methods

The study employs a two-stage approach:

  1. Topic Modeling: Two neural topic modeling algorithms, BERTopic and Top2Vec, were compared. These models leverage pre-trained language models (embeddings) to understand semantic relationships within text, going beyond simple keyword matching. BERTopic was ultimately chosen for its flexibility and performance.
  2. Topic Labeling: After topics were extracted as keyword lists, a Large Language Model (LLM), specifically GPT-4o mini, was used to automatically generate descriptive labels for each topic. This process was refined by providing the LLM with both keywords and sample document snippets to ensure accurate and contextually relevant labels.

The research also investigated the impact of domain-specific clinical/medical embedding models (like BioClinicalBERT) and sentence-based chunking strategies to optimize topic extraction for medical narratives.

Datasets

The primary dataset used in this study consists of 13 anonymized, manually transcribed interviews with pancreatic cancer patients. These interviews, totaling 132,722 words, were originally in Dutch and machine-translated into English. No new benchmarks were introduced; the focus was on applying existing techniques to this specific, valuable dataset.

Key Insights and Ideas

  • BERTopic Outperforms Top2Vec: In this specific context of patient narratives, BERTopic demonstrated superior performance in generating coherent and less overlapping topics compared to Top2Vec.
  • Clinical Embeddings Matter: Using embedding models pre-trained on clinical data (like BioClinicalBERT) significantly improved the precision and interpretability of extracted topics, suggesting that domain-specific models are crucial for medical text.
  • Chunking is Key: The way interview transcripts are segmented (chunked) into smaller pieces for analysis critically impacts the granularity and quality of the extracted topics.
  • LLMs for Summarization: LLMs can effectively automate the labeling of extracted topics, making them more accessible to clinicians, though the quality of labels depends heavily on the context provided.
  • Overarching Themes Emerge: A global analysis of all interviews revealed dominant themes like "Coordination and Communication in Cancer Care Management" and "Patient Decision-Making in Cancer Treatment Journey," highlighting common patient concerns across diverse experiences.

Key Results

  • BERTopic's Superiority: In a pilot study evaluating a single interview, BERTopic generated 17 topics, with 12 receiving a coherence score of 4.0 or higher from human evaluators. Top2Vec generated 19 topics but had lower average coherence scores.
  • Keyword Quality: While BERTopic excelled in topic coherence, Top2Vec's keyword lists were rated as more helpful for understanding the topic's meaning by human evaluators.
  • Clinical Embeddings Impact: BioClinicalBERT consistently produced the most interpretable and clinically meaningful topics when applied to the full dataset, outperforming other clinical embedding models.
  • Dominant Global Themes: The global analysis identified "Coordination and Communication in Cancer Care Management" and "Patient Decision-Making in Cancer Treatment Journey" as the most prevalent themes across all 13 interviews.

Limitations and Future Work

  • Translation Artifacts: The interviews were machine-translated from Dutch to English, which may have introduced distortions or altered nuances, potentially impacting topic extraction.
  • Small-Scale Human Evaluation: The human evaluation involved only three volunteers who were not clinical experts, limiting the depth of professional applicability assessment.
  • Chunking Sensitivity: Topic modeling proved sensitive to chunking strategies, requiring a balance between granularity and narrative coherence.
  • Future Work: The authors suggest exploring multilingual capabilities, developing more robust evaluation frameworks, integrating the system into clinical decision-support tools, and testing with clinical experts. They also emphasize the need to move beyond translated text by using original language data and exploring more adaptive chunking strategies. Expanding the analysis to larger and more diverse datasets is also a key future direction.

Summary updated

Abstract

This study investigates the use of neural topic modeling and LLMs to uncover meaningful themes from patient storytelling data, to offer insights that could contribute to more patient-oriented healthcare practices. We analyze a collection of transcribed interviews with cancer patients (132,722 words in 13 interviews). We first evaluate BERTopic and Top2Vec for individual interview summarization by using similar preprocessing, chunking, and clustering configurations to ensure a fair comparison on Keyword Extraction. LLMs (GPT4) are then used for the next step topic labeling. Their outputs for a single interview (I0) are rated through a small-scale human evaluation, focusing on {coherence}, {clarity}, and {relevance}. Based on the preliminary results and evaluation, BERTopic shows stronger performance and is selected for further experimentation using three {clinically oriented embedding} models. We then analyzed the full interview collection with the best model setting. Results show that domain-specific embeddings improved topic \textit{precision} and \textit{interpretability}, with BioClinicalBERT producing the most consistent results across transcripts. The global analysis of the full dataset of 13 interviews, using the BioClinicalBERT embedding model, reveals the most dominant topics throughout all 13 interviews, namely ``Coordination and Communication in Cancer Care Management" and ``Patient Decision-Making in Cancer Treatment Journey''. Although the interviews are machine translations from Dutch to English, and clinical professionals are not involved in this evaluation, the findings suggest that neural topic modeling, particularly BERTopic, can help provide useful feedback to clinicians from patient interviews. This pipeline could support more efficient document navigation and strengthen the role of patients' voices in healthcare workflows.
BibTeX
@misc{ionescu2026analyzing,
  author = {Teodor-Calin Ionescu and Lifeng Han and Jan Heijdra Suasnabar and Anne Stiggelbout and Suzan Verberne},
  title = {{Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs}},
  year = {2026},
  eprint = {2601.12154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.12154},
}
Figure