Towards Zero Shot Conditional Summarization with Adaptive Multi-task Fine-Tuning

Travis R. Goodwin, Max E. Savery, Dina Demner-FushmanEMNLP 2020 · DOI

Abstract

Conditional summarization—generating summaries tailored to specific queries or topics—is often hindered by a lack of domain-specific training data. This paper presents a novel approach using Multi-Task Fine-Tuning (MTFT) across 21 diverse language tasks combined with a "self-adaptive" task-mixing strategy. Our results demonstrate significant improvements in zero-shot performance for complex domains, such as consumer health, achieving a 35.5% relative improvement in ROUGE-L on the MEDIQA benchmark.

Introduction

Traditional automatic summarization systems typically produce generic, "one-size-fits-all" summaries. However, many real-world applications require conditional summarization, where the output is focused on a specific question (e.g., "What are the side effects of this drug?") or a particular theme. The primary obstacle to developing such systems is the data bottleneck: creating high-quality, human-authored conditional summaries is expensive and time-consuming, particularly in specialized fields like medicine.

This research explores whether a model can learn the underlying concept of conditional summarization by being exposed to a wide variety of auxiliary tasks, subsequently applying this knowledge to unseen domains in a zero-shot fashion.

Methodology

The researchers employed two prominent transformer-based architectures: T5-Base and BART-Large. The core of the methodology is Multi-Task Fine-Tuning (MTFT), where models are trained simultaneously on 21 distinct natural language processing tasks, including question answering (e.g., SQuAD), causal reasoning (e.g., CoPA), and general summarization (e.g., CNN/DailyMail).

A critical component of this approach is the task-mixing strategy used during training. To prevent the model from overfitting on data-rich tasks or ignoring difficult ones, the authors introduced two "online" strategies:

  1. Adaptive Mixing: Adjusts task selection probability based on validation loss.
  2. Self-Adaptive Mixing: Incorporates training loss and an "exploration ratio" to balance task difficulty and data volume.
Comparison of different task-mixing strategies, showing the superiority of the Self-Adaptive approach.

As shown in the data, the Self-Adaptive strategy with a focus parameter of $\gamma=4$ yielded the best overall performance, indicating that prioritizing tasks where the model struggles enhances generalization.

Experimental Setup and Datasets

The models were trained on a diverse set of 21 tasks. To evaluate the effectiveness of MTFT in specialized domains, the authors introduced or extended several medical datasets, including Cochrane, CQaD-S, and MedlinePlus Summaries. Zero-shot evaluation was conducted on the MEDIQA-AnS (consumer health) dataset and the DUC/TAC news tracks.

Results and Discussion

The application of MTFT resulted in substantial performance gains across multiple benchmarks. In the consumer health domain (MEDIQA), the BART model saw a relative ROUGE-L improvement of 35.5% when augmented with MTFT.

Zero-shot performance on the MEDIQA dataset across various metrics.

Key findings include:

  • Knowledge Injection: Training on seemingly unrelated tasks like causal reasoning significantly improves the model's ability to structure conditional summaries in new domains.
  • Model Sensitivity: BART showed a more pronounced positive response to MTFT compared to T5, likely due to differences in their initial pre-training objectives.
  • Metric Limitations: The study highlights that traditional overlap-based metrics like ROUGE and BLEU may not fully capture the quality of highly abstractive summaries. Embedding-based measures (EACS, VECS) are suggested as more robust alternatives.

Conclusion

This work demonstrates that Multi-Task Fine-Tuning, coupled with an intelligent self-adaptive mixing strategy, is a powerful technique for zero-shot conditional summarization. By "teaching" models the general structure of conditioning, we can bypass the need for massive, domain-specific datasets. Future research will focus on refining evaluation metrics for conditional context and exploring the transition from zero-shot to few-shot learning.

Summary updated

BibTeX
@inproceedings{goodwin2020towards,
  author = {Travis R. Goodwin and Max E. Savery and Dina Demner-Fushman},
  title = {{Towards Zero Shot Conditional Summarization with Adaptive Multi-task Fine-Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.289},
}
Figure