Nutri-bullets: Summarizing Health Studies by Composing Segments

Darsh J. Shah, Lili Yu, Tao Lei, Regina Barzilay

Abstract

Summarizing health and nutrition studies requires high faithfulness and relevance, which standard sequence-to-sequence models often fail to provide due to hallucinations and data scarcity. We present Nutri-bullets, an "extract-compose" framework that explicitly extracts factual spans and uses reinforcement learning to select and fuse them. Our approach significantly outperforms baselines in faithfulness and relevance on two new health summarization benchmarks, particularly in low-resource settings.

Introduction

In the health and nutrition domain, the rapid publication of scientific studies necessitates efficient summarization for public consumption. However, manual summarization is resource-intensive and requires significant domain expertise. While modern AI models excel at generating fluent text, they frequently suffer from two major issues in high-stakes fields: unfaithfulness (hallucinating facts) and data scarcity (lack of high-quality training pairs). Standard sequence-to-sequence (seq2seq) models often fail to accurately capture essential facts from long, multi-document inputs, leading to summaries that may be fluent but factually incorrect.

Methods: The Extract-Compose Pipeline

Nutri-bullets addresses these challenges through a structured three-stage pipeline that prioritizes factual grounding over unconstrained generation.

The Nutri-bullets pipeline: (i) input sentences, (ii) knowledge span extraction, (iii) selection for summarization, and (iv) text fusion.
  1. Span Extraction ($P(Z|X)$): Instead of processing the entire text at once, the model identifies specific "knowledge spans"—short phrases containing critical entity-relation information (e.g., "Flaxseed extract may improve heart health").
  2. Content Selection Policy ($P(C|Z)$): A Reinforcement Learning (RL) policy network treats selection as a sequential decision process, picking an ordered subset of spans from a pool. The RL agent is rewarded based on how well the selection mirrors the structure of human-written summaries and ensures information diversity.
  3. Text Fusion ($P(y|C)$): A Blank Language Model (BLM) fills the gaps between selected spans to create a fluent sentence. This model is specifically designed to ensure the final output is readable without altering the underlying facts.

Datasets and Evaluation

The framework was evaluated on two new benchmarks introduced by the authors:

  • HealthLine: 1,894 instances of scientific abstracts paired with short, bulleted summaries.
  • BreastCancer: A smaller, more challenging dataset of 141 instances where summaries reference an average of 18 articles each.

To train the extraction modules, the authors also collected a massive dataset of 91K entities and 34K relation tuples (e.g., "Food X" -> "Increases" -> "Condition Y").

Results

Nutri-bullets, particularly the variant using the Blank Language Model (BLM), demonstrated superior performance across multiple metrics compared to standard Transformer and GraphWriter baselines.

Human evaluation results on BreastCancer and HealthLine datasets across Faithfulness, Relevance, and Fluency.

Human evaluators rated Nutri-bullets significantly higher for faithfulness and relevance. On the challenging BreastCancer dataset, Nutri-bullets was preferred 94% of the time over a strong Transformer baseline. The model also achieved higher Knowledge Graph (KG) metrics, proving it captures actual scientific relationships better than standard seq2seq models. While standard models often achieve high fluency, they frequently fail on faithfulness; Nutri-bullets effectively "anchors" the AI to the truth.

Conclusion

By shifting from "black box" generation to a structured "extract-compose" framework, Nutri-bullets provides a more reliable method for summarizing technical health data. While the model's performance depends on the quality of initial span extraction, it represents a significant step forward in generating faithful summaries for high-stakes domains. Future work will focus on improving sentence cohesion and scaling the approach to broader scientific fields.

Summary updated

Abstract

We introduce \emph{Nutri-bullets}, a multi-document summarization task for health and nutrition. First, we present two datasets of food and health summaries from multiple scientific studies. Furthermore, we propose a novel \emph{extract-compose} model to solve the problem in the regime of limited parallel data. We explicitly select key spans from several abstracts using a policy network, followed by composing the selected spans to present a summary via a task specific language model. Compared to state-of-the-art methods, our approach leads to more faithful, relevant and diverse summarization -- properties imperative to this application. For instance, on the BreastCancer dataset our approach gets a more than 50\% improvement on relevance and faithfulness.\footnote{Our code and data is available at \url{https://github.com/darsh10/Nutribullets.}}
BibTeX
@inproceedings{shah2021nutri,
  author = {Darsh J. Shah and Lili Yu and Tao Lei and Regina Barzilay},
  title = {{Nutri-bullets: Summarizing Health Studies by Composing Segments}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17624},
}
Figure