TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain

Bohao Chu, Meijie Li, Sameh Frihat, Chengyu Gu, Georg Lodde, Elisabeth Livingstone, Norbert FuhrEMNLP 2025

TL;DR

This paper introduces TRACSUM, a novel benchmark and task for aspect-based summarization in the medical domain, focusing on sentence-level traceability. By pairing generated summaries with specific source sentences, TRACSUM aims to improve the factual accuracy and verifiability of medical summaries, a critical need for evidence-based medicine.

What Problem Does This Paper Address?

Large Language Models (LLMs) have revolutionized document summarization, making vast amounts of textual information more accessible. However, a significant concern, especially in critical fields like medicine, is the factual accuracy of these summaries. LLMs can sometimes "hallucinate" information or present it inaccurately, which poses serious risks when medical professionals rely on these summaries for decision-making. Existing summarization methods often produce general summaries, but users in specialized fields like medicine often need information tailored to specific aspects (e.g., patient population, intervention, outcomes). Furthermore, without a clear way to trace the origin of information within a summary, users cannot easily verify its accuracy or locate the original context. This paper addresses the need for aspect-based summaries that are not only accurate but also traceable to their source sentences, enabling users to critically assess the information.

Methods

The core contribution is the TRACSUM task, which requires generating aspect-based summaries of clinical articles and providing sentence-level citations for each summary. To achieve this, the authors propose a TRACK-THEN-SUM (TTS) pipeline. This pipeline consists of two main components:

  1. Tracker (T): This module identifies sentences within the source article that are most relevant to a specific medical aspect.
  2. Summarizer (S): This module then generates a concise summary based on the sentences identified by the tracker.

The authors also explore a variant, TTS ⊕ f., which incorporates the full document context in addition to the tracked sentences, aiming to improve summarization quality. For evaluation, they introduce a fine-grained automatic framework that assesses completeness (claim recall and citation recall) and conciseness (claim precision and citation precision), going beyond simple accuracy metrics.

Datasets

The paper introduces TRACSUM, a new benchmark dataset. This dataset was constructed by:

  • Annotating 500 medical abstracts.
  • Identifying seven key medical aspects (Aims, Intervention, Outcomes, Participants, Medicine, Duration, Side Effects).
  • Generating approximately 3.5K summary-citation pairs.

The initial generation of summary-citation pairs was performed using Mistral Large, followed by human evaluation and revision by medical students and NLP researchers to ensure quality.

Key Insights and Ideas

  • Traceability is Crucial for Trust: The ability to trace summary content back to specific source sentences is paramount for building trust in LLM-generated medical summaries.
  • Aspect-Based Summarization is More Useful: Users often need summaries focused on specific aspects rather than general overviews, especially in specialized domains.
  • Explicit Tracking Improves Accuracy: Performing sentence-level tracking before summarization (as in TTS) enhances the accuracy of the generated summaries.
  • Full Context Can Help, But Needs Careful Handling: Incorporating the full document context (TTS ⊕ f.) can improve claim recall by helping models understand abbreviations and domain-specific terms, but it requires careful management to avoid introducing irrelevant information.
  • LLMs are Promising, But Not Perfect: While LLMs show strong performance, especially larger ones, they still require robust evaluation frameworks to ensure factual accuracy and traceability.

Key Results

  • TRACSUM as an Effective Benchmark: Experiments show that TRACSUM can effectively benchmark LLMs for traceable, aspect-based summarization.
  • TTS Performance: The proposed TRACK-THEN-SUM pipeline, particularly the TTS ⊕ f. variant, demonstrated strong performance, outperforming baseline LLMs on citation-based metrics.
  • LLM Performance: Larger open-source LLMs (e.g., Llama-3.1-70B, Mistral-8x7B) generally outperformed smaller models. Proprietary models like GPT-40 also performed well.
  • Human Evaluation Alignment: The automatic evaluation framework showed moderate positive correlation with human judgments (Spearman's p = 0.612, Pearson's r = 0.577), indicating its potential as a reliable assessment tool.
  • Impact of Full Context: The TTS ⊕ f. variant improved claim recall from 67.1% to 79.8% without significantly compromising other metrics.
  • Aspect Performance Variation: Performance varied across aspects, with "Outcomes" and "Intervention" being more challenging, likely due to their complex information. "Duration" was easier, partly because many instances were negative (no relevant information).

Limitations and Future Work

  • Dataset Generation Bias: The initial dataset was generated by Mistral Large, which could introduce model-specific biases. The authors mitigated this through human evaluation and excluding Mistral Large from some evaluations.
  • Human Evaluation Cost: Conducting extensive human evaluations is costly and time-consuming.
  • Future Work: The authors suggest further research into improving the accuracy and efficiency of traceable summarization, exploring different LLMs and pipeline architectures, and expanding the benchmark to include more diverse medical document types. They also highlight the need for more robust entailment evaluators.

Summary updated

BibTeX
@inproceedings{chu2025tracsum,
  author = {Bohao Chu and Meijie Li and Sameh Frihat and Chengyu Gu and Georg Lodde and Elisabeth Livingstone and Norbert Fuhr},
  title = {{TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure