Nutri-bullets: Summarizing Health Studies by Composing Segments

Darsh J. Shah, Lili Yu, Tao Lei, Regina Barzilay

Abstract

We introduce Nutri-bullets, a multi-document summarization task for health and nutrition. First, we present two datasets of food and health summaries from multiple scientific studies. Furthermore, we propose a novel extract-compose model to solve the problem in the regime of limited parallel data. We explicitly select key spans from several abstracts using a policy network, followed by composing the selected spans to present a summary via a task specific language model. Compared to state-of-the-art methods, our approach leads to more faithful, relevant and diverse summarization -- properties imperative to this application. For instance, on the BreastCancer dataset our approach gets a more than 50% improvement on relevance and faithfulness.

Introduction

In the health and nutrition domain, factual accuracy is paramount. A single "hallucinated" fact in a summary can lead to dangerous medical misinformation. While large language models (LLMs) excel at generating fluent text, they often struggle with factual consistency in technical domains where training data is scarce. Multi-document summarization (MDS) for health is particularly challenging because summaries must synthesize a collective view from multiple studies while remaining strictly faithful to the source material.

Standard sequence-to-sequence (seq2seq) models often fail due to hallucinations and data scarcity. To address this "faithfulness gap," we propose an "extract-compose" framework that builds summaries from verified snippets of scientific evidence.

Methods: The Extract-Compose Pipeline

Our approach moves away from "black box" abstractive summarization toward a modular three-stage pipeline designed to prioritize accuracy over mere fluency.

The Nutri-bullets pipeline: from raw abstracts to fused summaries via span extraction and selection.
  1. Span Extraction: The model identifies "knowledge spans"—the shortest phrases in an abstract explaining a specific relationship (e.g., how a nutrient affects a condition). This grounds the summary's building blocks in the source text.
  2. Content Selection: Using a Reinforcement Learning (RL) policy network, the model selects an ordered subset of relevant and diverse spans. The RL agent optimizes for rewards based on the structure of human-written "gold" summaries.
  3. Text Fusion: A specialized Blank Language Model (BLM) takes the selected spans and "fills in the blanks" to create cohesive sentences. By using extracted segments as anchors, the BLM is prevented from deviating from the source facts.

Datasets

We introduce two new datasets for health and nutrition MDS:

  • HealthLine: 1,894 instances of abstracts paired with bulleted summaries.
  • BreastCancer: A high-complexity dataset with 141 instances, where summaries reference an average of 18 articles each.

Additionally, we collected a large-scale dataset of crowd-sourced annotations covering 91,000 entities and 34,000 relation pairs to support the extraction phase.

Results and Discussion

The Nutri-bullets approach, particularly the BLM variant, demonstrates significant improvements over state-of-the-art baselines like Copy-gen and GraphWriter.

Human evaluation results comparing the proposed BLM model against baselines across three key metrics.

Human evaluations reveal that on the BreastCancer dataset, our model was rated over 50% higher for relevance and faithfulness than the next best baseline. In head-to-head comparisons, human judges preferred Nutri-bullets summaries over standard Transformer outputs 94% of the time for the BreastCancer dataset.

A key insight is the "fluency trap": standard models often receive high fluency scores even when they are unfaithful. Our framework effectively "hard-codes" faithfulness by forcing the model to compose from extracted spans.

Conclusion

The "extract-compose" logic provides a robust safety rail for technical summarization. While the model currently relies on the quality of initial entity extraction and is optimized for bullet-style summaries, it represents a significant step toward trustworthy automated health communication. Future work will explore handling conflicting findings across studies and adapting the framework for narrative-driven scientific reviews.

Summary updated

Abstract

We introduce \emph{Nutri-bullets}, a multi-document summarization task for health and nutrition. First, we present two datasets of food and health summaries from multiple scientific studies. Furthermore, we propose a novel \emph{extract-compose} model to solve the problem in the regime of limited parallel data. We explicitly select key spans from several abstracts using a policy network, followed by composing the selected spans to present a summary via a task specific language model. Compared to state-of-the-art methods, our approach leads to more faithful, relevant and diverse summarization -- properties imperative to this application. For instance, on the BreastCancer dataset our approach gets a more than 50\% improvement on relevance and faithfulness.\footnote{Our code and data is available at \url{https://github.com/darsh10/Nutribullets.}}
BibTeX
@inproceedings{shah2021nutri,
  author = {Darsh J. Shah and Lili Yu and Tao Lei and Regina Barzilay},
  title = {{Nutri-bullets: Summarizing Health Studies by Composing Segments}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17624},
}
Figure