Zero-Shot Rationalization by Multi-Task Transfer Learning from Question Answering

Po-Nien Kung, Tse-Hsuan Yang, Yi-Cheng Chen, Sheng-Siang Yin, Yun-Nung ChenEMNLP 2020 · OpenReview

Abstract

Po-Nien Kung, Tse-Hsuan Yang, Yi-Cheng Chen, Sheng-Siang Yin, Yun-Nung Chen. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

Deep learning models are often criticized as "black boxes." To establish trust, particularly in high-stakes domains like medicine or finance, models must provide rationales: specific snippets of input text that justify a prediction. However, obtaining rationale annotations is labor-intensive and expensive compared to simple labels. While supervised methods exist, they rely on these rare annotations. This paper addresses the challenge by proposing a zero-shot rationalization framework that leverages multi-task transfer learning from Question Answering (QA).

Methodology: Rationalization as Question Answering

The core innovation is treating the identification of a rationale as a QA task. The authors hypothesize that a rationale is essentially the "answer" to the question "Why was this label chosen?"

Shared Architecture and Multi-Task Training

The framework utilizes a shared ALBERT encoder with multiple task-specific heads: one for extractive QA (trained on SQuAD 2.0) and others for domain-specific classification (e.g., Beer or Movie ratings). To align these tasks, classification is reformulated into a QA format. For instance, a beer review is paired with a question like "What is this beer appearance score?"

Overview of the Multi-Task Training and Zero-Shot Rationalization framework.

The model undergoes Multi-Task Learning (MTL), cycling through QA and classification tasks. This forces the encoder to develop a universal understanding of how specific text spans relate to queries. During inference, the model uses its QA head to extract rationales from the target domain, despite never having seen a human-annotated rationale for that domain during training.

Experimental Results

The model was evaluated on the BeerReview and MovieReview datasets, using SQuAD 2.0 as the source for teaching reading comprehension mechanics.

Performance on BeerReview

The proposed multi-task approach (S+B+M, combining SQuAD, Beer, and Movie data) achieved significant improvements in precision. As shown in the results table, it reached 93.41% precision at a 10% rationale length, outperforming previous state-of-the-art methods.

Performance comparison on the BeerReview dataset.

Generalization and Stability

In the MovieReview task, the zero-shot model achieved a 9.3 IOU F1 score, notably surpassing supervised baselines like "Bert-To-Bert" (7.5) which had access to rationale labels. Furthermore, the MTL approach demonstrated remarkable stability across different review aspects (e.g., aroma vs. palate), suggesting a robust, generalized understanding of evidence extraction.

Conclusion

This work demonstrates that rationalization can be effectively framed as a transfer learning problem from QA. By training on general reading comprehension tasks alongside classification, models can learn to extract high-quality, salient rationales in a zero-shot manner. While currently limited to extractive rationalization, this approach offers a scalable path toward more interpretable AI without the need for exhaustive rationale annotations.

Summary updated

BibTeX
@inproceedings{kung2020zero,
  author = {Po-Nien Kung and Tse-Hsuan Yang and Yi-Cheng Chen and Sheng-Siang Yin and Yun-Nung Chen},
  title = {{Zero-Shot Rationalization by Multi-Task Transfer Learning from Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=tYIoCy8WaGB},
}
Figure