Nutri-bullets Hybrid: Consensual Multi-document Summarization

Darsh J. Shah, Lili Yu, Tao Lei, Regina BarzilayNAACL 2021 · DOI

Introduction

In the domain of nutrition and health, scientific literature often presents a landscape of conflicting evidence. Synthesizing multiple studies into a coherent, faithful summary is a complex task for human experts and a significant challenge for automated systems. Standard multi-document summarization (MDS) models, typically based on large sequence-to-sequence Transformers, often struggle with niche domains due to a lack of parallel training data. More critically, these "black-box" models are prone to hallucinations and frequently fail to explicitly identify contradictions between sources, which can lead to the dissemination of medical misinformation.

To address these issues, researchers from MIT and ASAPP developed "Nutri-bullets," a hybrid summarization system designed to identify consensus and contradictions across multiple scientific documents. By combining symbolic information extraction with neural text-infilling, the model produces summaries that are significantly more faithful to source material than end-to-end baselines.

The Nutri-bullets Pipeline

The Nutri-bullets model moves away from direct text generation toward a controlled, "concept-to-text" approach. The pipeline consists of four distinct stages:

  1. Information Extraction (IE): Raw text from PubMed abstracts is converted into symbolic tuples. Each tuple represents a relationship between a food/nutrient and a health condition (e.g., [Pears, Ovarian Cancer, Controls]).
  2. Content Selection: Using a Markov Decision Process (MDP) trained via Reinforcement Learning, the system selects a subset of extracted tuples that are relevant and non-redundant.
  3. Consensus Aggregation: The system applies deterministic rules to compare tuples across different studies. If studies report differing effects for the same entities, the system flags a Contradiction; otherwise, it flags Agreement.
  4. Surface Realization: The symbolic data is converted back into natural language using a neural text-infilling model. This model takes a "prototype" sentence template and populates it with the selected entities and relations.
Overview of the Nutri-bullets pipeline, showing input documents, the intermediate symbolic representation, and the final aggregated output.

Aggregation and Text Infilling

A key innovation of this work is "aggregation cognizance." Unlike standard summarizers that treat multiple documents as a single bag of words, Nutri-bullets is designed to recognize the logical relationship between documents. By explicitly labeling contradictions rather than averaging out conflicting claims, the model provides a more nuanced and accurate synthesis of the available evidence.

The surface realization process, where the aggregation operator (e.g., Contradiction) and selected tuples are combined with a prototype sentence to generate the final summary.

Experimental Results

The researchers evaluated the model using the HealthLine dataset, comprising 6,640 scientific abstracts and human-written summaries. The Nutri-bullets model was compared against state-of-the-art Transformer and "Copy-gen" baselines across several metrics:

  • Faithfulness: The hybrid model achieved a KG(I) score of 0.76, measuring factual alignment with the input, compared to 0.64 for the standard Transformer baseline.
  • Aggregation Accuracy: The model significantly outperformed baselines in identifying the correct relationship between studies, scoring 0.89 on Aggregation Cognisance versus the baseline's 0.67.
  • Human Evaluation: Judges rated Nutri-bullets summaries as more relevant and factually grounded. While neural baselines sometimes offered higher fluency, they were prone to critical errors, such as reversing the reported effects of dietary interventions.

Conclusion and Future Work

The Nutri-bullets framework demonstrates that symbolic intermediate steps provide a level of factual control that pure neural models currently lack. While the system's performance is currently bounded by the accuracy of the initial Information Extraction step, it offers a robust path forward for summarization in high-stakes domains like healthcare. Future research will focus on learning more complex aggregation patterns and expanding the system's utility to broader scientific fields.

Summary updated

BibTeX
@inproceedings{shah2021nutri,
  author = {Darsh J. Shah and Lili Yu and Tao Lei and Regina Barzilay},
  title = {{Nutri-bullets Hybrid: Consensual Multi-document Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.411},
}
Figure