Dual Supervised Learning for Natural Language Understanding and Generation

Shang-Yu Su, Chao-Wei Huang, Yun-Nung Chen

Abstract: Shang-Yu Su, Chao-Wei Huang, Yun-Nung Chen. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

In the architecture of modern spoken dialogue systems, Natural Language Understanding (NLU) and Natural Language Generation (NLG) are the two fundamental pillars. NLU serves as the system's "ear," mapping natural language utterances to structured semantic frames, while NLG acts as the "mouth," converting those structured frames back into human-readable text. Traditionally, these components have been developed and optimized as independent modules. However, this paper argues that NLU and NLG are intrinsically linked as probabilistic "duals" of one another, and that leveraging this relationship can significantly enhance the performance of both.

The Dual Supervised Learning Framework

The core innovation of this research is the Dual Supervised Learning (DSL) framework. This approach treats the relationship between language ($y$) and semantics ($x$) as a structural regularizer. Mathematically, the tasks are connected via Bayes' Rule: the joint probability can be expressed as $P(x, y) = P(x)P(y|x) = P(y)P(x|y)$. Here, $P(y|x)$ represents the NLG model, and $P(x|y)$ represents the NLU model.

The duality between Natural Language Understanding (NLU) and Natural Language Generation (NLG).

By adding a duality constraint to the standard Maximum Likelihood Estimation (MLE) loss function, the authors force the models to remain mathematically consistent. If the NLU model's interpretation of a sentence does not align with the NLG model's generation from that same meaning, the system receives a penalty ($l_{duality}$). This ensures that the two models "agree" on the mapping between the surface form and the underlying intent.

Modeling Semantic Distributions with MADE

A significant challenge in implementing the duality constraint is estimating the marginal distributions $P(x)$ and $P(y)$. While natural language ($P(y)$) is easily modeled using standard recurrent neural networks (RNNs), semantic frames ($P(x)$) present a unique problem. Semantic frames, such as Price: Cheap, Food: Italian, are sets of slot-value pairs rather than sequential data. Treating them as sequences forces an arbitrary order that may not exist.

To address this, the authors utilize a Masked Autoencoder for Distribution Estimation (MADE). MADE allows the system to capture complex dependencies between different semantic slots—for instance, recognizing that certain restaurant types are more likely to be associated with specific price points—without imposing a fixed sequence. This allows for a more accurate estimation of the semantic prior, which is crucial for the duality constraint to function effectively.

Experimental Evaluation

The framework was evaluated using the E2E NLG Challenge dataset, which contains over 50,000 instances in the restaurant domain. While the dataset was originally intended for generation, the authors adapted it for NLU by treating the prediction of slot-value pairs as a multi-label classification task. Both the NLU and NLG models were built using Gated Recurrent Units (GRUs) to maintain a symmetric and comparable architecture.

Key Results and Discussion

The experimental results demonstrated that the DSL framework consistently outperformed the baseline approach of training NLU and NLG models in isolation.

Performance comparison between the baseline and various Dual Supervised Learning (DSL) configurations.

As shown in the results table, the DSL scheme (specifically with $\lambda = 0.1$) achieved the highest scores across nearly all metrics. The NLU F1 score improved from 71.14% to 72.32%, while the NLG BLEU score saw a substantial increase from 55.05 to 57.16.

An ablation study (row 'e' in the table) highlighted the importance of the MADE component. When the semantic slots were modeled as independent variables rather than through MADE, performance dropped. This confirms that capturing the internal correlations within semantic frames is vital for maintaining the probabilistic duality between the two tasks.

Conclusion

This paper demonstrates that NLU and NLG should not be viewed as isolated tasks but as two sides of the same coin. By implementing a Dual Supervised Learning framework, researchers can use the mathematical consistency between understanding and generation as a powerful regularization signal. This "rising tide" effect improves both the system's ability to interpret human speech and its ability to respond naturally, providing a more robust foundation for future dialogue systems.

Summary updated

BibTeX
@inproceedings{su2019dual,
  author = {Shang-Yu Su and Chao-Wei Huang and Yun-Nung Chen},
  title = {{Dual Supervised Learning for Natural Language Understanding and Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=hjGAhWDHwQW},
}
Figure