Reinforcement Learning for Improved Low Resource Dialogue Generation

Ana Valeria González-GarduñoAAAI 2019 · DOI

Introduction

Dialogue systems have traditionally been bifurcated into two distinct paradigms: open-domain "chitchat" systems and goal-oriented agents. While chitchat models excel in fluency, they often lack substance and generate repetitive, generic responses. Conversely, goal-oriented systems are precise but brittle, requiring extensive manual labeling and rigid Dialogue State Tracking (DST). This research, presented at the AAAI-19 Doctoral Consortium, proposes a unified framework to bridge this gap, specifically targeting the challenges of data sparsity in low-resource languages and the high cost of domain adaptation.

The Hybrid Hierarchical Model

To address the tendency of neural generative models to produce "meaningless" responses (e.g., "I don't know"), the author introduces a hybrid architecture. This model combines a Hierarchical Encoder-Decoder (HRED) with an exemplar-based retrieval mechanism.

When generating a response, the system first retrieves the most relevant dialogue from a training set. This retrieved "example" is then fed into the neural decoder as additional context. By using retrieval as a "hint," the model is provided with a template for what a correct, precise answer looks like. This grounding ensures that the output remains grammatically fluent while being anchored in factual, relevant information.

Domain Adaptation via Reinforcement Learning

A significant bottleneck in modern dialogue systems is the requirement for turn-level supervision, which is expensive and difficult to scale across new domains. Moving from a "hotel booking" domain to a "doctor's appointment" domain typically requires manually labeling every turn of the conversation for the new context.

The proposed solution utilizes Reinforcement Learning (RL), specifically Policy Gradients, to facilitate domain adaptation without turn-by-turn labels. Instead of relying on granular supervision, the model is trained using a dialogue-level reward based on joint goal accuracy—essentially, whether the final outcome of the conversation was successful. This allows the system to learn the optimal conversational path in a new domain with significantly less human intervention.

Targeting Low-Resource Languages

The research places a heavy emphasis on cross-lingual transfer, particularly for Scandinavian languages such as Danish, Norwegian, Swedish, and Icelandic. These languages lack the massive corpora available for English or Spanish, which usually stalls neural development. The framework aims to leverage models trained on high-resource languages and adapt them to these low-resource targets using RL and hybrid retrieval-generation techniques.

Preliminary Results and Future Directions

Early findings suggest that conditioning decoders on retrieved examples improves response accuracy over standard HRED models. Furthermore, the author notes that using a multi-layer perceptron (MLP) for relevancy ranking offers a computationally efficient alternative to complex recurrent models, making the system more practical for real-time applications.

Future research directions include:

  • Ellipsis Resolution: Training models to understand omitted words that are contextually implied by previous turns.
  • Adversarial Learning: Implementing a discriminator to distinguish between human and machine-generated dialogue logs, pushing the generator toward more naturalistic speech.

Conclusion

By operating in the "grey area" between chitchat and task-completion, this research agenda seeks to create more robust, natural, and data-efficient conversational agents. The combination of retrieval-augmented generation and RL-based domain adaptation provides a promising path forward for developing sophisticated dialogue systems in linguistic environments where data is scarce.

Summary updated

BibTeX
@inproceedings{gonzalezgarduno2019reinforcement,
  author = {Ana Valeria González-Garduño},
  title = {{Reinforcement Learning for Improved Low Resource Dialogue Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019884},
}
Figure