TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

Siddhant Garg, Thuy Vu, Alessandro Moschitti

Introduction

In the ecosystem of modern virtual assistants like Alexa or Siri, Answer Sentence Selection (AS2) serves as a foundational component. The task is straightforward yet demanding: given a user's question and a set of candidate sentences, the system must accurately identify the specific sentence that provides the answer. While Transformer-based models like BERT have set new benchmarks across NLP, applying them effectively to AS2 has remained a significant challenge due to the unique constraints of the task.

The Challenges of AS2 Fine-Tuning

Fine-tuning large-scale pre-trained models for AS2 is often hindered by three primary obstacles:

  1. Data Scarcity: Academic AS2 datasets are typically small, which leads to high variance and makes it difficult for models to generalize.
  2. Training Instability: Small datasets often cause "on-off" behavior during fine-tuning, where models fail to converge or collapse into predicting a single label.
  3. Noise Sensitivity: Real-world data is inherently noisy. Standard fine-tuning techniques are often fragile when faced with mislabeled examples or low-quality training pairs.

The TANDA Framework

To overcome these hurdles, the authors introduce TANDA (Transfer and Adapt), a sequential fine-tuning framework designed to stabilize and enhance Transformer performance for AS2.

The TANDA sequential fine-tuning pipeline: Pre-training, Transfer on the large-scale ASNQ dataset, and Adaptation to the target domain.

The TANDA pipeline consists of three distinct stages:

  1. Pre-training: Utilizing a standard pre-trained language model (e.g., BERT or RoBERTa) as the base.
  2. Transfer Step: The model is fine-tuned on a massive, general-purpose AS2 dataset. This stage is crucial as it teaches the model the underlying logic of matching questions to answers, effectively bridging the gap between general language understanding and the specific AS2 task.
  3. Adapt Step: A final, lightweight fine-tuning stage is performed on the specific target domain (e.g., technical support or movie trivia).

Scaling with the ASNQ Dataset

The success of the "Transfer" step relies on the availability of large-scale data. The authors addressed this by creating the ASNQ (Answer Sentence Natural Questions) dataset. By transforming Google’s "Natural Questions" dataset—originally designed for Machine Reading—into an AS2 format, they produced a corpus of over 20 million question-sentence pairs.

Example of answer sentence selection context, illustrating how candidate sentences are extracted and labeled within a document.

ASNQ is two orders of magnitude larger than previous benchmarks. Its scale allows the model to learn robust features that are less susceptible to the noise found in smaller, manually annotated datasets.

Key Results and Impact

The TANDA approach yields significant improvements in both performance and reliability:

  • State-of-the-Art Performance: TANDA achieved a 92.0% MAP on WikiQA and 94.3% MAP on TREC-QA, representing a massive leap over previous bests (83.4% and 87.5%, respectively).
  • Remarkable Robustness: In experiments where 20% noise was intentionally added to the training data, standard fine-tuning performance plummeted by 22%. In contrast, TANDA’s performance dropped by a mere 2.5%, demonstrating a ten-fold increase in resilience.
  • Industrial Viability: Beyond academic benchmarks, the framework was validated on internal Alexa datasets, confirming its effectiveness in high-traffic production environments.

Conclusion

TANDA demonstrates that an intermediate "Transfer" step on a large-scale task-specific dataset like ASNQ is the key to unlocking the full potential of Transformers for AS2. By separating the learning of task logic from domain adaptation, TANDA provides a stable, high-performance, and noise-robust solution for modern question-answering systems. Future work will explore the applicability of this framework to other NLP tasks and different Transformer architectures.

Summary updated

Abstract

We propose TANDA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving MAP scores of 92% and 94.3%, respectively, which largely outperform the previous highest scores of 83.4% and 87.5%, obtained in very recent work. We empirically show that TANDA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TANDA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TANDA in an industrial setting, using domain specific datasets subject to different types of noise.
BibTeX
@inproceedings{garg2020tanda,
  author = {Siddhant Garg and Thuy Vu and Alessandro Moschitti},
  title = {{TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6282},
}
Figure