Dual Learning for Semi-Supervised Natural Language Understanding

Su Zhu, Ruisheng Cao, Kai Yu

Abstract

Natural language understanding (NLU) converts sentences into structured semantic forms. The paucity of annotated training samples is still a fundamental challenge of NLU. To solve this data sparsity problem, previous work based on semi-supervised learning mainly focuses on exploiting unlabeled sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence generation (SSG), and propose a new framework for semi-supervised NLU with the corresponding dual model. The framework is composed of dual pseudo-labeling and dual learning method, which enables an NLU model to make full use of data (labeled and unlabeled) through a closed-loop of the primal and dual tasks. By incorporating the dual task, the framework can exploit pure semantic forms as well as unlabeled sentences, and further improve the NLU and SSG models iteratively in the closed-loop. The proposed approaches are evaluated on two public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting show that our methods can outperform various baselines significantly, and extensive ablation studies are conducted to verify the effectiveness of our framework. Finally, our method can also achieve the state-of-the-art performance on the two datasets in the supervised setting.

1. Introduction

Natural Language Understanding (NLU) is a critical component of task-oriented dialogue systems, responsible for mapping user utterances into structured semantic representations, such as intents and slots. A persistent challenge in NLU is the high cost of data annotation, which limits the scalability of deep learning models to new domains. While semi-supervised learning (SSL) typically leverages unlabeled text, it often ignores the potential of "pure semantic forms"—structured intents and slots without corresponding text—which are relatively inexpensive to synthesize using domain knowledge.

This paper introduces a dual task to NLU: Semantic-to-Sentence Generation (SSG). By establishing a closed-loop relationship between NLU (sentence $\rightarrow$ semantics) and SSG (semantics $\rightarrow$ sentence), the authors propose a framework that utilizes both unlabeled sentences and synthesized semantic forms to improve model performance.

The primal NLU task and its dual SSG task form a closed loop for data exploitation.

2. Methodology

The proposed framework bridges the gap between natural language and structured semantics through a dual model approach.

2.1 Semantic-to-Sentence Generation (SSG)

The dual SSG model generates natural language sentences from structured semantic inputs. It employs a hierarchical BLSTM encoder to process slot-value pairs and a Semantically Controlled LSTM (SC-LSTM) decoder. A copy mechanism is integrated to ensure that specific slot values, such as city names, are accurately preserved in the generated output.

2.2 Dual Learning and Pseudo-Labeling

To leverage unlabeled data effectively, the framework employs two main strategies:

  • Dual Pseudo-Labeling: The NLU model predicts semantic labels for unlabeled text, while the SSG model generates sentences for synthesized semantic forms. Both models are iteratively fine-tuned on these pseudo-labeled pairs.
  • Dual Learning: This approach uses a reinforcement learning setup where NLU and SSG act as two agents in a closed loop. The models are updated via policy gradients based on two types of rewards: reconstruction rewards (measuring how well the original input is recovered after a full cycle) and validity rewards (ensuring generated semantics are logically consistent, such as checking if a specific slot is valid for a given intent).
The dual learning framework illustrating the interaction between NLU and SSG through reconstruction and validity rewards.

3. Experiments and Results

The framework was evaluated on the ATIS (Airline Travel Information Systems) and SNIPS datasets.

3.1 Semi-Supervised Efficiency

In settings with limited labeled data (ranging from 5% to 50%), the dual framework consistently outperformed standard SSL baselines like sentence auto-encoders and template synthesis. Notably, with only 10% of ATIS training data, the method achieved a 93.53% Slot F1 score. Ablation studies revealed that pseudo-samples generated by the SSG model were more critical to performance gains than those from the NLU model.

3.2 State-of-the-Art Performance

When trained on full datasets, the framework achieved state-of-the-art results. On ATIS, it reached 96.37% Slot F1 and 98.54% Intent Accuracy. On SNIPS, it achieved 96.11% Slot F1 and 98.86% Intent Accuracy. Further integration with pre-trained BERT embeddings pushed the SNIPS Intent Accuracy to 99.14%.

4. Conclusion

By introducing the dual task of SSG, this work provides a robust framework for semi-supervised NLU. The ability to exploit both unlabeled text and synthesized semantic forms effectively addresses the data sparsity problem. Future work will explore joint modeling of intents and slots and investigate domain adaptation to scale dialogue systems to more complex applications.

Summary updated

Abstract

Natural language understanding (NLU) converts sentences into structured semantic forms. The paucity of annotated training samples is still a fundamental challenge of NLU. To solve this data sparsity problem, previous work based on semi-supervised learning mainly focuses on exploiting unlabeled sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence generation (SSG), and propose a new framework for semi-supervised NLU with the corresponding dual model. The framework is composed of dual pseudo-labeling and dual learning method, which enables an NLU model to make full use of data (labeled and unlabeled) through a closed-loop of the primal and dual tasks. By incorporating the dual task, the framework can exploit pure semantic forms as well as unlabeled sentences, and further improve the NLU and SSG models iteratively in the closed-loop. The proposed approaches are evaluated on two public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting show that our methods can outperform various baselines significantly, and extensive ablation studies are conducted to verify the effectiveness of our framework. Finally, our method can also achieve the state-of-the-art performance on the two datasets in the supervised setting. Our code is available at \url{https://github.com/rhythmcao/slu-dual-learning.git}.
BibTeX
@article{zhu2020dual,
  author = {Su Zhu and Ruisheng Cao and Kai Yu},
  title = {{Dual Learning for Semi-Supervised Natural Language Understanding}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=XewnawGlOA3},
}
Figure