Transfer Learning and Masked Generation for Answer Verbalization

Sebastien Montella, Lina Rojas-Barahona, Frederic Bechet, Johannes Heinecke, Alexis NasrNAACL 2022

Sure, here's a blog-style summary of the paper "Transfer Learning and Masked Generation for Answer Verbalization":

TL;DR

This paper introduces a novel approach to answer verbalization for Question Answering (QA) systems by combining transfer learning with a masked generation strategy. By using placeholders for answers during training, the model learns to generate more abstract and generalizable verbalizations, leading to significant improvements in evaluation metrics.

What Problem Does This Paper Address?

Question Answering (QA) systems have made remarkable progress, often achieving human-level performance in extracting answers from knowledge bases (KBs). However, these systems typically provide raw, uncontextualized answers. The problem this paper addresses is how to transform these raw answers into natural, informative, and contextualized verbalizations that are easier for users to understand. This task, known as Answer Verbalization (AV), is crucial for improving the user experience of QA systems. Existing methods often struggle with generalization and can be computationally expensive.

Methods

The authors propose a two-stage approach:

  1. Masked Answer Verbalization: Instead of directly generating the raw answer, the model is trained to generate a verbalized template where the answer is replaced by a placeholder (e.g., [ANSWER]). This "masked" approach encourages the model to learn more generalizable patterns for constructing sentences, rather than memorizing specific answer strings.
  2. Transfer Learning with Pretrained Models: The core of their method leverages powerful pretrained language models (PLMs) like BART and T5. These models are fine-tuned on the AV task. The authors hypothesize that BART, with its denoising pretraining objective, is particularly well-suited for AV as it can learn to reconstruct a "noisy" input (the question) into a more complete output (the verbalized answer).

For evaluation, the placeholder is replaced with the actual raw answer to allow for direct comparison with other methods.

Datasets

The paper evaluates its approach on three datasets:

  • VQuAnDa: A dataset of 5000 complex questions with SPARQL queries and semi-automatically generated answer verbalizations.
  • ParaQA: An extension of VQuAnDa that provides multiple verbalizations (paraphrases) for each question, offering more training data.
  • VANiLLa: A large dataset (over 100,000 pairs) built using a semi-automatic framework, focusing on direct questions with single-entity answers. The authors also introduce a minor revision of VANiLLa by correcting entity hallucinations in 5% of the data.

Key Insights and Ideas

  • Placeholders Enhance Generalization: Using answer placeholders during training significantly improves the model's ability to generalize, leading to better performance on unseen data. This is because the model learns sentence structure and context rather than just memorizing specific answer-entity mappings.
  • BART's Denoising Strength: BART's pretraining objective, which involves reconstructing corrupted input, makes it particularly effective for answer verbalization. The authors suggest that questions can be viewed as "noisy" versions of their verbalized answers, a task BART excels at.
  • Data Quality Matters: The paper highlights the critical impact of reference quality on evaluation metrics. Correcting even a small percentage of entity hallucinations in the VANiLLa dataset led to noticeable improvements in SacreBLEU scores, demonstrating the sensitivity of automatic metrics to noisy ground truth.

Key Results

The proposed masked generation approach with transfer learning achieved state-of-the-art results on the evaluated datasets:

  • Relative Gains: The method achieved relative gains of 44.25% in BLEU on VQuAnDa, 3.26% on ParaQA, and 29.10% on VANiLLa compared to the VOGUE baseline.
  • SacreBLEU Improvements: The masked approach consistently outperformed the non-masked approach across all metrics, including SacreBLEU, Chrf++, and TER. For instance, on VQuAnDa, masked BART achieved a SacreBLEU score of 60.85 compared to 39.19 for unmasked BART.
  • Data Quality Impact: Correcting entity hallucinations in VANiLLa led to an absolute median gain of 0.81 SacreBLEU when evaluating on the corrected test set, compared to a much smaller gain of 0.13 SacreBLEU when evaluating on the original, noisy test set.

Limitations and Future Work

  • Language Specificity: The authors acknowledge that their placeholder approach might require adjustments for languages with complex gender agreement, such as French or German.
  • Computational Cost: While transfer learning with PLMs is powerful, it can still be computationally expensive. The paper notes that previous unsupervised methods were computationally intensive due to permutation likelihood estimation.
  • Dataset Quality: The paper strongly emphasizes the limitations imposed by noisy datasets and the need for more reliable automatic evaluation metrics. Future work could focus on developing more robust evaluation methods or further improving dataset construction pipelines.
  • Hallucination Correction: The paper demonstrates the benefit of correcting entity hallucinations and suggests this as a promising avenue for improving AV systems and evaluation.

Summary updated

BibTeX
@inproceedings{montella2022transfer,
  author = {Sebastien Montella and Lina Rojas-Barahona and Frederic Bechet and Johannes Heinecke and Alexis Nasr},
  title = {{Transfer Learning and Masked Generation for Answer Verbalization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure