Abstract
A major bottleneck in training end-to-end task-oriented dialog systems is the lack of data. To utilize limited training data more efficiently, we propose Modular Supervision Network (MOSS), an encoder-decoder training framework that incorporates supervision from various intermediate dialog system modules including natural language understanding (NLU), dialog state tracking (DST), dialog policy learning (DPL), and natural language generation (NLG). With only 60% of the training data, MOSS-all (i.e., MOSS with supervision from all four dialog modules) outperforms state-of-the-art models on CamRest676. Moreover, introducing modular supervision has even bigger benefits when the dialog task has a more complex dialog state and action space. With only 40% of the training data, MOSS-all outperforms the state-of-the-art model on a complex laptop network troubleshooting dataset, LaptopNetwork, that we introduced. MOSS is extremely flexible to update in real-world deployments as it can accommodate dialogs with supervision from different modules at both the framework and model levels.
Introduction
Building task-oriented dialog systems typically involves a trade-off between traditional modular systems and modern end-to-end (E2E) neural models. Traditional modular systems are data-efficient but suffer from error propagation across the pipeline. Conversely, E2E systems avoid error propagation by mapping user input directly to responses but are notoriously data-hungry. MOSS addresses this "data bottleneck" by creating a framework that leverages the structural benefits of modular systems within a jointly trainable, neural E2E architecture.
The MOSS Framework
The core of MOSS is a Modular Supervision Network. Unlike standard E2E models that only use the final system response as a training signal, MOSS utilizes four distinct decoders that share a single bidirectional GRU encoder.
Key Architectural Features
- Hidden State Connectivity: Instead of passing symbolic text between modules, MOSS passes hidden states. This allows the model to be optimized via backpropagation across all modules simultaneously, mitigating error propagation.
- Plug-and-Play Flexibility: MOSS can be trained even with missing annotations. If a dataset lacks specific labels (e.g., NLU), the corresponding decoder can be removed or its loss ignored. This allows for the integration of raw, unannotated dialogs to boost performance.
- Interpretability: By employing explicit decoders for intermediate steps, MOSS provides transparency. Developers can inspect NLU or DST outputs to diagnose logic failures.
Experimental Evaluation
The authors evaluated MOSS on two benchmarks:
- CamRest676: A standard dataset for restaurant reservations.
- LaptopNetwork: A complex, real-world dataset introduced in this study, consisting of 618 Chinese dialogs regarding laptop network troubleshooting. It features a significantly larger action space (60 dialog acts) compared to CamRest676 (9 acts).
Key Results
The most striking finding is that the quality of supervision acts as a data multiplier. MOSS-all outperformed state-of-the-art models on CamRest676 using only 60% of the training data. The benefits of modular supervision scale with task complexity; on the LaptopNetwork task, MOSS outperformed the baseline using only 40% of the data.
On the LaptopNetwork dataset, MOSS-all achieved a Success Accuracy of 0.61, significantly higher than the baseline TSCP score of 0.18. Furthermore, adding unannotated (raw) dialogs to a model already trained with modular supervision boosted language generation quality (BLEU scores), validating the framework's utility in real-world deployment scenarios where full annotation is scarce.
Conclusion and Future Work
MOSS demonstrates that incorporating intermediate modular supervision significantly improves the data efficiency and performance of E2E dialog systems. While the framework requires dense multi-level annotations, the resulting gains in accuracy and interpretability are substantial. Future work may explore the automatic generation of these modular annotations to reduce human overhead or apply the framework to non-task-oriented domains.
Summary updated