Abstract
Semih Yavuz, Kazuma Hashimoto, Wenhao Liu, Nitish Shirish Keskar, Richard Socher, Caiming Xiong. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
In task-oriented dialogue systems, Dialog Act (DA) tagging is the process of identifying a speaker's intent (e.g., "requesting" a price or "informing" a location). While the intent of a request is often universal, the vocabulary changes drastically between domains—asking for a "departure time" in a flight app looks very different from asking for "seating" in a restaurant app.
This "domain shift" usually means that a model trained on one domain performs poorly on another. Traditionally, solving this requires expensive manual labeling for every new domain. The authors tackle the challenge of unsupervised domain adaptation: how can we take a model trained on a "Source" domain (with labels) and make it work on a "Target" domain using only unlabeled text?
Methods: MASKAUGMENT
The core innovation is MASKAUGMENT, which leverages the pre-trained knowledge already inside BERT. Instead of complex word-swapping or back-translation, the authors simply replace random tokens in a dialogue turn with the BERT [MASK] token.
The framework uses a Teacher-Student consistency regularization approach. For a single unlabeled sentence from the target domain, they create two versions:
- The Teacher: A version with light masking (retaining most of the original content).
- The Student: A version with heavy masking (retaining less content).
The student is trained to match the teacher’s predicted probability distribution for dialog acts. This forces the model to learn robust, high-level features that represent the "intent" regardless of specific domain-heavy keywords that might be masked out. The final model is trained using a combination of losses: standard supervised learning on the source domain, Masked Language Modeling (MLM) on the target domain, and the new Teacher-Student "Disagreement Loss" (DAL).
Datasets and Evaluation
The researchers evaluated their approach on two major benchmarks:
- GSIM (Simulated Dialogue): Focused on buying movie tickets (GMov) and reserving restaurant tables (GRes).
- SGD (Schema-Guided Dialogue): A massive dataset covering 20 domains. The authors used SMusic (Music), SMedia (Media), and SRide (Ride-sharing) as source domains to test generalization on SFlights (Flights).
Key Results
- Significant F1 Gains: MASKAUGMENT improved the Target domain F1 score by up to 3% when full source data was available.
- Low-Resource Hero: The impact was even more dramatic in low-resource settings. When only 10 labeled dialogues were available in the source domain, the proposed method improved target domain performance by a staggering 10%.
- Closing the Recall Gap: One of the biggest issues in domain adaptation is low recall. MASKAUGMENT significantly narrows this gap, becoming much better at identifying frequent but domain-sensitive acts like
sys-offerandrequest.
Insights and Limitations
The authors found that the [MASK] token is a feature, not a bug; because BERT is already "comfortable" with the token, using it as an augmentation tool provides a very stable signal for consistency training. These benefits are orthogonal to domain-adaptive pre-training (MLM).
However, certain acts, like sys-notify-failure, remained difficult for the model to distinguish in the "scratch-BERT" setting. Additionally, performance relies on tuning the masking probabilities for the teacher and student, which may vary depending on the complexity of the target domain.
Summary updated