Nutribullets Hybrid: Multi-document Health Summarization

Darsh J. Shah, Lili Yu, Tao Lei, Regina BarzilayarXiv 2021 · OpenReview

Abstract

We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. The key challenge in creating such summaries is the lack of large parallel training data required for training typical summarization systems. To this end, we introduce a hybrid generation approach inspired by traditional concept-to-text systems. To enable accurate comparison between different sources, the model first learns to extract pertinent relations from input documents. The content planning component uses deterministic operators to aggregate these relations after identifying a subset for inclusion into a summary. The surface realization component lexicalizes this information using a text-infilling language model. By separately modeling content selection and realization, we can effectively train them with limited annotations. We implemented and tested the model in the domain of nutrition and health -- rife with inconsistencies. Compared to conventional methods, our framework leads to more faithful, relevant and aggregation-sensitive summarization -- while being equally fluent.

Introduction: The Challenge of Conflicting Health Evidence

In the domain of nutrition and health, scientific consensus is often elusive. Individual studies frequently yield inconsistent or even contradictory results regarding the effects of specific foods on health conditions. For instance, while one study might suggest a protective effect of certain fruits against cancer, another might find no significant association.

Overview of the Nutribullets Hybrid pipeline, synthesizing conflicting PubMed abstracts into a coherent summary.

Standard neural summarization models, such as those based on the Transformer architecture, often struggle with these inconsistencies. They tend to prioritize linguistic fluency over factual faithfulness, leading to "hallucinations" where contradictions are ignored or misrepresented. This phenomenon is often referred to as the "ROUGE Trap," where models achieve high scores based on word overlap but fail to capture the underlying logic of the research. Furthermore, the lack of large-scale parallel datasets—comprising multiple conflicting papers paired with expert summaries—makes training end-to-end models difficult.

Methodology: A Hybrid Generation Approach

To address these challenges, we introduce Nutribullets Hybrid, a system that decomposes the summarization task into three distinct, manageable stages. This approach combines symbolic logic for content planning with modern neural language models for surface realization.

The three-stage Nutribullets Hybrid architecture: (a) Information Extraction, (b) Aggregation, and (c) Surface Realization.

1. Information Extraction (IE)

The first stage involves converting unstructured text from PubMed abstracts into a structured format. The system extracts symbolic tuples representing key relations, such as [Food/Nutrition, Condition, Relation]. For example, a study finding that pears decrease breast cancer risk would be represented as [Pears, Breast Cancer, Decreases].

2. Content Selection and Aggregation

Once the relations are extracted, the model must decide which information to include and how to relate different findings:

  • Selection: A Reinforcement Learning agent (specifically a Markov Decision Process) is trained to select the most relevant and diverse set of tuples from the extracted database.
  • Aggregation: Deterministic operators are applied to identify the relationship between selected tuples. If multiple studies report differing effects for the same food-condition pair, the system explicitly flags this as a "Contradiction."

3. Surface Realization

The final stage lexicalizes the aggregated information. Instead of generating text from scratch, the model uses a text-infilling language model. It starts with a "prototype" summary—a template based on the identified relationship (e.g., consensus or contradiction)—and fills in the blanks with the selected entities. This ensures the output remains faithful to the underlying logic while maintaining natural language fluency.

Data and Results

The model was trained and evaluated using the HealthLine dataset, which contains over 6,600 scientific abstracts paired with human-written summaries from Healthline.com. These summaries represent expert syntheses of multiple studies for a general audience.

The Nutribullets Hybrid model significantly outperformed standard end-to-end baselines across several critical metrics:

  • Aggregation Cognizance: The model achieved an absolute 20% improvement in correctly expressing consensus or contradictions.
  • Faithfulness: Content faithfulness scores were 7% higher than the best baseline, indicating a marked reduction in factual errors.
  • Human Evaluation: Human annotators rated the hybrid model significantly higher for relevance (3.03 vs. 2.66) while finding its fluency competitive with purely neural models.

Conclusion

By separating content selection from realization, Nutribullets Hybrid offers a robust framework for summarizing complex, conflicting scientific literature. While the system's performance is currently limited by the accuracy of the initial Information Extraction phase, it provides a clear path forward for "update-friendly" summarization. As new research emerges, the system can dynamically update its knowledge base and generate revised summaries that reflect the evolving scientific consensus.

Summary updated

Abstract

We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. The key challenge in creating such summaries is the lack of large parallel training data required for training typical summarization systems. To this end, we introduce a hybrid generation approach inspired by traditional concept-to-text systems. To enable accurate comparison between different sources, the model first learns to extract pertinent relations from input documents. The content planning component uses deterministic operators to aggregate these relations after identifying a subset for inclusion into a summary. The surface realization component lexicalizes this information using a text-infilling language model. By separately modeling content selection and realization, we can effectively train them with limited annotations. We implemented and tested the model in the domain of nutrition and health -- rife with inconsistencies. Compared to conventional methods, our framework leads to more faithful, relevant and aggregation-sensitive summarization -- while being equally fluent.
BibTeX
@misc{shah2021nutribullets,
  author = {Darsh J. Shah and Lili Yu and Tao Lei and Regina Barzilay},
  title = {{Nutribullets Hybrid: Multi-document Health Summarization}},
  year = {2021},
  eprint = {2104.03465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.03465},
}
Figure