Abstract: ADVISER is an open-source, modular framework designed to bridge the gap between cutting-edge NLP research and interdisciplinary education. It supports the development of multi-domain, bilingual (English/German) task-oriented dialog systems using a flexible graph-based architecture. By accommodating both handcrafted rule-based modules and Deep Reinforcement Learning (RL) architectures, ADVISER provides a versatile platform for researchers to experiment with new algorithms while offering a user-friendly GUI for students and linguists.
Introduction
Building task-oriented dialog systems—such as those used for booking flights or navigating university course schedules—typically requires a rigid pipeline consisting of Natural Language Understanding (NLU), Belief State Tracking (BST), Policy Selection, and Natural Language Generation (NLG).
Existing toolkits often suffer from two major limitations: technical barriers that exclude non-engineers (like linguists) from contributing domain expertise, and architectural rigidity that struggles with fluid transitions between multiple domains. ADVISER addresses these gaps by providing a "white-box" framework that is accessible to students yet powerful enough for advanced research.
System Architecture
ADVISER’s architecture is built on a graph structure rather than a linear pipeline, allowing users to define complex data flows and branching scenarios. This modularity allows for hybrid support, where users can swap between rule-based modules (using regex and templates) and neural network-based modules, such as a recurrent neural network (RNN) for belief tracking and a Dueling Deep Q-Network (DQN) for dialog policy.
A central innovation of the framework is the Meta Policy. This component acts as the system's arbitrator for multi-domain interactions. Instead of forcing a single domain prediction, ADVISER runs multiple domain-specific subgraphs in parallel. The Meta Policy coordinates these outputs and can even trigger utterance rewriting to maintain a seamless user experience during topic shifts.
To further lower the barrier for non-technical users, ADVISER includes CrossTick, a gamified interface. In this environment, students earn points by finding "holes" in the NLU logic, effectively crowdsourcing the improvement of system coverage while learning complex linguistic concepts.
Evaluation and Datasets
The researchers evaluated ADVISER using two primary domains:
- Cambridge Restaurants: A standard benchmark for task-oriented dialog systems.
- IMS Modules: A new domain focused on the Institute for Natural Language Processing at the University of Stuttgart, containing data on lecturers, course locations, and module details.
Results and Impact
Automated evaluations show that ADVISER's RL-based policy is highly competitive with traditional handcrafted (HDC) systems. As shown in the performance table below, the RL policy achieved a 97.5% success rate on the Cambridge Restaurants domain and 91.6% on the IMS Modules domain.
Human evaluations corroborated these findings, with the RL policy receiving a quality rating of 4.3/7, nearly identical to the rule-based policy's 4.48/7. Beyond technical performance, the framework's educational impact was significant: 84.6% of participants agreed that the gamified CrossTick interface was an effective tool for understanding NLU concepts.
Conclusion and Future Work
ADVISER successfully serves as a "dual-use" tool for both the classroom and the lab. While handcrafted policies currently maintain a slight edge in reliability and dialog brevity, the RL policies are robust and competitive. Future work will explore scaling the Meta Policy to handle a larger number of simultaneous domains and extending the modular architecture to support speech and vision modalities.
Summary updated