Dual Supervised Learning for Natural Language Understanding and Generation

Shang-Yu Su, Chao-Wei Huang, Yun-Nung Chen

TL;DR

This paper introduces a novel framework that leverages the inherent duality between Natural Language Understanding (NLU) and Natural Language Generation (NLG) to improve both tasks simultaneously. By treating NLU and NLG as a pair of inverse problems and incorporating this duality into a dual supervised learning objective, the authors demonstrate significant performance gains on benchmark datasets.

What Problem Does This Paper Address?

Natural Language Processing (NLP) often deals with two seemingly opposite tasks: understanding language (NLU) and generating language (NLG). NLU aims to extract meaning from text, typically by converting it into a structured semantic representation. Conversely, NLG takes a semantic representation and converts it into human-readable text. While both are crucial for applications like dialogue systems, they have traditionally been trained and optimized independently. This paper identifies that this inherent "dual" relationship between NLU and NLG has been largely underexplored. The authors propose to bridge this gap by explicitly exploiting this duality to create a more effective learning framework that benefits both tasks concurrently.

Methods

The core of the proposed framework is Dual Supervised Learning. The authors frame NLU and NLG as a probabilistic duality: if the models for both tasks are optimal, their conditional probabilities should satisfy a specific relationship. This duality is incorporated into the learning objective as a regularization term, alongside the standard supervised loss for each task. This creates a multi-objective optimization problem.

To estimate the necessary marginal distributions (which are often intractable), the paper proposes specialized methods based on the nature of the data:

  • Language Modeling: For sequential natural language data, a standard autoregressive language model (using GRUs) is employed to estimate the probability distribution of word sequences.
  • Masked Autoencoder (MADE): For discrete semantic frames (which are not inherently sequential), a Masked Autoencoder is used. This approach allows for modeling complex, non-directional dependencies between slot-value pairs within a semantic frame by using masked connections and an ensemble of different decomposition strategies.

Datasets

The experiments are conducted on the E2E NLG challenge dataset. This dataset consists of approximately 50,000 instances in the restaurant domain, where each instance is a pair of a semantic frame and its corresponding natural language utterance. The dataset is used for both training and evaluation.

Key Insights and Ideas

  • Exploiting Duality is Powerful: The paper's central idea is that treating NLU and NLG as a dual problem pair and explicitly incorporating this relationship into the training objective significantly boosts performance for both tasks.
  • Beyond Independent Training: Traditional approaches train NLU and NLG separately. This work shows that a joint learning framework that leverages their inverse relationship is more effective.
  • Data-Specific Distribution Estimation: Recognizing that different data types (text vs. semantic frames) have different structural properties is key. Using language models for text and masked autoencoders for structured semantic frames is a principled approach to estimating their distributions.
  • Masked Autoencoders for Structured Data: MADE proves to be a valuable tool for modeling the complex, inter-dependent nature of semantic frames, outperforming simpler independence assumptions.

Key Results

The proposed dual supervised learning framework, with its specialized distribution estimation methods, consistently outperforms a baseline model that trains NLU and NLG independently.

  • Performance Boost: The results show improvements in NLU (micro-F1) and NLG (BLEU, ROUGE-1, ROUGE-2, ROUGE-L) scores across the board.
  • Stronger Regularization is Better: The performance gains increase with stronger regularization (higher Lagrange parameters), indicating the importance of the duality constraint.
  • Ablation Study: An ablation test where slot-value pairs in semantic frames were treated as independent showed worse performance, highlighting the importance of modeling their dependencies using MADE.

Limitations and Future Work

  • Limitations: The paper acknowledges that the true marginal distributions of data are often intractable, and approximations are used. The effectiveness of the framework relies on the quality of these approximations.
  • Future Work: The paper suggests that this dual supervised learning framework can be extended to other dual task pairs in NLP and beyond. The authors also highlight the potential for further exploration of different methods for estimating data distributions.

Summary updated

Abstract

Natural language understanding (NLU) and natural language generation (NLG) are both critical research topics in the NLP field. Natural language understanding is to extract the core semantic meaning from the given utterances, while natural language generation is opposite, of which the goal is to construct corresponding sentences based on the given semantics. However, such dual relationship has not been investigated in the literature. This paper proposes a new learning framework for language understanding and generation on top of dual supervised learning, providing a way to exploit the duality. The preliminary experiments show that the proposed approach boosts the performance for both tasks.
BibTeX
@inproceedings{su2019dual,
  author = {Shang-Yu Su and Chao-Wei Huang and Yun-Nung Chen},
  title = {{Dual Supervised Learning for Natural Language Understanding and Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=hjGAhWDHwQW},
}
Figure