TL;DR
This paper tackles the data sparsity problem in Natural Language Understanding (NLU) by introducing a dual task: Semantic-to-Sentence Generation (SSG). By leveraging both unlabeled sentences and "pure semantic forms" through dual pseudo-labeling and reinforcement learning-based dual learning, the framework significantly boosts NLU performance and achieves state-of-the-art results on standard benchmarks, even with highly limited labeled data.
What Problem Does This Paper Address?
NLU systems in task-oriented dialogue (like Alexa or Siri) convert user utterances into structured semantic forms—specifically, user intents (e.g., "book flight") and slots (e.g., "destination: New York"). Training these deep learning models requires massive amounts of annotated data, which is expensive to collect and hard to scale to new domains.
While semi-supervised learning (SSL) has been used to mitigate this by exploiting unlabeled sentences, previous methods completely ignore pure semantic forms (intents and slots without accompanying text). Because semantic forms are well-structured, they are actually very cheap to synthesize using basic domain knowledge. The authors aim to build a framework that can exploit both unlabeled text and these unexpressed semantic forms to train better NLU models.
Methods
The authors introduce a novel dual framework consisting of a primal task (NLU: sentence $\rightarrow$ semantics) and a dual task (SSG: semantics $\rightarrow$ sentence). The SSG model uses a hierarchical BLSTM encoder and a Semantically Controlled LSTM (SC-LSTM) decoder with a copy mechanism to generate text from intents and slots.
To train this system using both labeled and unlabeled data, they propose two main techniques:
- Dual Pseudo-Labeling: The NLU model predicts pseudo-labels for unlabeled sentences, while the SSG model generates pseudo-sentences for pure semantic forms. Both models are then fine-tuned iteratively on these newly created pseudo-pairs.
- Dual Learning: They set up a two-agent reinforcement learning game using policy gradients. It features two closed loops: NLU $\rightarrow$ SSG and SSG $\rightarrow$ NLU. To make this work for structured data, they design novel validity rewards (checking if generated semantics make logical sense, like "Boston" being a valid city) and reconstruction rewards (measuring how well the cycle reconstructs the original input).
Datasets
- ATIS (Airline Travel Information Systems): A classic dataset of audio recordings and transcripts for flight reservations.
- SNIPS: A crowdsourced natural language corpus used to benchmark the performance of voice assistants.
- OVERNIGHT: Used briefly in the paper to demonstrate the framework's applicability to broader semantic parsing tasks.
Key Insights and Ideas
- Semantic forms are an untapped goldmine: Synthesizing pure semantic forms (which is cheap) and using a dual model to generate text for them provides a massive boost to NLU performance, outperforming traditional pseudo-labeling that only looks at unlabeled text.
- SSG pseudo-samples matter more: Ablation studies revealed that creating pseudo-samples from the SSG model (semantics $\rightarrow$ text) is actually more essential to the performance boost than creating them from the NLU model (text $\rightarrow$ semantics).
- Validity rewards are crucial: In the dual learning loop, simply rewarding reconstruction isn't enough. Adding a "validity reward" that checks if slots and intents naturally co-occur (e.g., the slot
FromCityco-occurring with the intentfind_flight) significantly improves Slot F1 scores.
Key Results
- Semi-Supervised Efficiency: In semi-supervised settings (using only 5% to 50% of labeled data), the dual framework consistently outperformed baselines like standard pseudo-labeling, sentence auto-encoders, and template synthesis.
- Low-Data Performance: Using just 10% of the ATIS training data, the method achieved a 93.53% Slot F1 score and 93.51% Intent Accuracy.
- State-of-the-Art in Supervised Settings: When given 100% of the training data, the method achieved SOTA results: 96.37% Slot F1 and 98.54% Intent Accuracy on ATIS, and 96.11% Slot F1 / 98.86% Intent Accuracy on SNIPS.
- BERT Integration: Adding pre-trained BERT embeddings pushed performance even higher, hitting 99.14% Intent Accuracy on SNIPS.
Limitations and Future Work
- Limitations: The framework currently models intent detection and slot filling as independent sub-tasks within the NLU module, whereas recent research shows benefits in modeling their interdependencies jointly. Additionally, the SSG model struggles slightly to generate highly varied natural language for completely unseen slot-value pairs.
- Future Work: The authors plan to explore joint modeling of intent detection and slot filling within this dual framework. They also suggest investigating domain adaptation and transfer learning for the dual task to help scale dialogue systems to wider, more complex application domains.
Summary updated