Dual Learning for Semi-Supervised Natural Language Understanding

Su Zhu, Ruisheng Cao, Kai Yu

TL;DR

This paper tackles the data sparsity problem in Natural Language Understanding (NLU) by introducing a dual task: Semantic-to-Sentence Generation (SSG). By leveraging both unlabeled sentences and "pure semantic forms" through dual pseudo-labeling and reinforcement learning-based dual learning, the framework significantly boosts NLU performance and achieves state-of-the-art results on standard benchmarks, even with highly limited labeled data.

What Problem Does This Paper Address?

NLU systems in task-oriented dialogue (like Alexa or Siri) convert user utterances into structured semantic forms—specifically, user intents (e.g., "book flight") and slots (e.g., "destination: New York"). Training these deep learning models requires massive amounts of annotated data, which is expensive to collect and hard to scale to new domains.

While semi-supervised learning (SSL) has been used to mitigate this by exploiting unlabeled sentences, previous methods completely ignore pure semantic forms (intents and slots without accompanying text). Because semantic forms are well-structured, they are actually very cheap to synthesize using basic domain knowledge. The authors aim to build a framework that can exploit both unlabeled text and these unexpressed semantic forms to train better NLU models.

Methods

The authors introduce a novel dual framework consisting of a primal task (NLU: sentence $\rightarrow$ semantics) and a dual task (SSG: semantics $\rightarrow$ sentence). The SSG model uses a hierarchical BLSTM encoder and a Semantically Controlled LSTM (SC-LSTM) decoder with a copy mechanism to generate text from intents and slots.

To train this system using both labeled and unlabeled data, they propose two main techniques:

  1. Dual Pseudo-Labeling: The NLU model predicts pseudo-labels for unlabeled sentences, while the SSG model generates pseudo-sentences for pure semantic forms. Both models are then fine-tuned iteratively on these newly created pseudo-pairs.
  2. Dual Learning: They set up a two-agent reinforcement learning game using policy gradients. It features two closed loops: NLU $\rightarrow$ SSG and SSG $\rightarrow$ NLU. To make this work for structured data, they design novel validity rewards (checking if generated semantics make logical sense, like "Boston" being a valid city) and reconstruction rewards (measuring how well the cycle reconstructs the original input).

Datasets

  • ATIS (Airline Travel Information Systems): A classic dataset of audio recordings and transcripts for flight reservations.
  • SNIPS: A crowdsourced natural language corpus used to benchmark the performance of voice assistants.
  • OVERNIGHT: Used briefly in the paper to demonstrate the framework's applicability to broader semantic parsing tasks.

Key Insights and Ideas

  • Semantic forms are an untapped goldmine: Synthesizing pure semantic forms (which is cheap) and using a dual model to generate text for them provides a massive boost to NLU performance, outperforming traditional pseudo-labeling that only looks at unlabeled text.
  • SSG pseudo-samples matter more: Ablation studies revealed that creating pseudo-samples from the SSG model (semantics $\rightarrow$ text) is actually more essential to the performance boost than creating them from the NLU model (text $\rightarrow$ semantics).
  • Validity rewards are crucial: In the dual learning loop, simply rewarding reconstruction isn't enough. Adding a "validity reward" that checks if slots and intents naturally co-occur (e.g., the slot FromCity co-occurring with the intent find_flight) significantly improves Slot F1 scores.

Key Results

  • Semi-Supervised Efficiency: In semi-supervised settings (using only 5% to 50% of labeled data), the dual framework consistently outperformed baselines like standard pseudo-labeling, sentence auto-encoders, and template synthesis.
  • Low-Data Performance: Using just 10% of the ATIS training data, the method achieved a 93.53% Slot F1 score and 93.51% Intent Accuracy.
  • State-of-the-Art in Supervised Settings: When given 100% of the training data, the method achieved SOTA results: 96.37% Slot F1 and 98.54% Intent Accuracy on ATIS, and 96.11% Slot F1 / 98.86% Intent Accuracy on SNIPS.
  • BERT Integration: Adding pre-trained BERT embeddings pushed performance even higher, hitting 99.14% Intent Accuracy on SNIPS.

Limitations and Future Work

  • Limitations: The framework currently models intent detection and slot filling as independent sub-tasks within the NLU module, whereas recent research shows benefits in modeling their interdependencies jointly. Additionally, the SSG model struggles slightly to generate highly varied natural language for completely unseen slot-value pairs.
  • Future Work: The authors plan to explore joint modeling of intent detection and slot filling within this dual framework. They also suggest investigating domain adaptation and transfer learning for the dual task to help scale dialogue systems to wider, more complex application domains.

Summary updated

Abstract

Natural language understanding (NLU) converts sentences into structured semantic forms. The paucity of annotated training samples is still a fundamental challenge of NLU. To solve this data sparsity problem, previous work based on semi-supervised learning mainly focuses on exploiting unlabeled sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence generation (SSG), and propose a new framework for semi-supervised NLU with the corresponding dual model. The framework is composed of dual pseudo-labeling and dual learning method, which enables an NLU model to make full use of data (labeled and unlabeled) through a closed-loop of the primal and dual tasks. By incorporating the dual task, the framework can exploit pure semantic forms as well as unlabeled sentences, and further improve the NLU and SSG models iteratively in the closed-loop. The proposed approaches are evaluated on two public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting show that our methods can outperform various baselines significantly, and extensive ablation studies are conducted to verify the effectiveness of our framework. Finally, our method can also achieve the state-of-the-art performance on the two datasets in the supervised setting.
BibTeX
@article{zhu2020dual,
  author = {Su Zhu and Ruisheng Cao and Kai Yu},
  title = {{Dual Learning for Semi-Supervised Natural Language Understanding}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=XewnawGlOA3},
}

Loading related content...

Figure