ADVISER: A Dialog System Framework for Education & Research

Daniel Ortega, Dirk Väth, Gianna Weber, Lindsey Vanderlyn, Maximilian Schmidt, Moritz Völkel, Zorica Karacevic, Ngoc Thang VuACL 2019 · DOI

Abstract: ADVISER is an open-source, modular framework designed to bridge the gap between cutting-edge NLP research and interdisciplinary education. It supports the development of multi-domain, bilingual (English/German) task-oriented dialog systems using a flexible graph-based architecture. By accommodating both handcrafted rule-based modules and Deep Reinforcement Learning (RL) architectures, ADVISER provides a versatile platform for researchers to experiment with new algorithms while offering a user-friendly GUI for students and linguists.

Introduction

Building task-oriented dialog systems—such as those used for booking flights or navigating university course schedules—typically requires a rigid pipeline consisting of Natural Language Understanding (NLU), Belief State Tracking (BST), Policy Selection, and Natural Language Generation (NLG).

Existing toolkits often suffer from two major limitations: technical barriers that exclude non-engineers (like linguists) from contributing domain expertise, and architectural rigidity that struggles with fluid transitions between multiple domains. ADVISER addresses these gaps by providing a "white-box" framework that is accessible to students yet powerful enough for advanced research.

System Architecture

ADVISER’s architecture is built on a graph structure rather than a linear pipeline, allowing users to define complex data flows and branching scenarios. This modularity allows for hybrid support, where users can swap between rule-based modules (using regex and templates) and neural network-based modules, such as a recurrent neural network (RNN) for belief tracking and a Dueling Deep Q-Network (DQN) for dialog policy.

A central innovation of the framework is the Meta Policy. This component acts as the system's arbitrator for multi-domain interactions. Instead of forcing a single domain prediction, ADVISER runs multiple domain-specific subgraphs in parallel. The Meta Policy coordinates these outputs and can even trigger utterance rewriting to maintain a seamless user experience during topic shifts.

The ADVISER Meta Policy architecture handles parallel domain processing and utterance rewriting.

To further lower the barrier for non-technical users, ADVISER includes CrossTick, a gamified interface. In this environment, students earn points by finding "holes" in the NLU logic, effectively crowdsourcing the improvement of system coverage while learning complex linguistic concepts.

Evaluation and Datasets

The researchers evaluated ADVISER using two primary domains:

  1. Cambridge Restaurants: A standard benchmark for task-oriented dialog systems.
  2. IMS Modules: A new domain focused on the Institute for Natural Language Processing at the University of Stuttgart, containing data on lecturers, course locations, and module details.

Results and Impact

Automated evaluations show that ADVISER's RL-based policy is highly competitive with traditional handcrafted (HDC) systems. As shown in the performance table below, the RL policy achieved a 97.5% success rate on the Cambridge Restaurants domain and 91.6% on the IMS Modules domain.

Performance comparison of RL and Handcrafted (HDC) agents across two domains.

Human evaluations corroborated these findings, with the RL policy receiving a quality rating of 4.3/7, nearly identical to the rule-based policy's 4.48/7. Beyond technical performance, the framework's educational impact was significant: 84.6% of participants agreed that the gamified CrossTick interface was an effective tool for understanding NLU concepts.

Conclusion and Future Work

ADVISER successfully serves as a "dual-use" tool for both the classroom and the lab. While handcrafted policies currently maintain a slight edge in reliability and dialog brevity, the RL policies are robust and competitive. Future work will explore scaling the Meta Policy to handle a larger number of simultaneous domains and extending the modular architecture to support speech and vision modalities.

Summary updated

BibTeX
@inproceedings{ortega2019adviser,
  author = {Daniel Ortega and Dirk Väth and Gianna Weber and Lindsey Vanderlyn and Maximilian Schmidt and Moritz Völkel and Zorica Karacevic and Ngoc Thang Vu},
  title = {{ADVISER: A Dialog System Framework for Education \& Research}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-3016},
}
Figure