Abstract
Task-oriented dialog (TOD) systems rely on accurately modeling the relationship between dialog context and semantic states (belief and action states). While conventional approaches often aim for high similarity between these representations, this paper introduces Mars, a framework that leverages contrastive learning to explore the optimal relationship. We find that encouraging diversity between context and state representations—rather than similarity—significantly improves performance by reducing information redundancy. Mars achieves state-of-the-art results on MultiWOZ 2.0, CrossWOZ, and CamRest676.
Introduction
In task-oriented dialog (TOD) systems, the primary objective is to accurately map conversation history into structured semantic states: the belief state (user intent) and the action state (system response strategy). Traditional modeling paradigms often assume that the internal representation of the dialog context should be as similar as possible to these target states.
The Mars framework, developed by JD AI Research, challenges this intuition. Through the application of contrastive learning, the authors demonstrate that maintaining distinct representations for dialog context and semantic states—rather than forcing similarity—leads to superior performance in multi-turn conversations.
The Challenge of Representation Redundancy
A critical bottleneck in end-to-end TOD systems is representation quality. When a model’s internal encoding of the conversation history is too similar to the final structured state, it often suffers from information redundancy. This over-condensation can cause the model to "forget" subtle natural language nuances necessary for complex, multi-turn reasoning. The Mars framework investigates whether pushing these representations apart (increasing diversity) or pulling them together (increasing similarity) is more beneficial for downstream tasks.
Methodology: The Mars Framework
Mars is built upon a standard sequence-to-sequence architecture (utilizing a pre-trained T5 model) with a shared encoder and specialized decoders for belief and action states. The core innovation is the introduction of two contrastive learning (CL) strategies:
- Mars-P (Pair-aware): Follows traditional intuition by using a contrastive loss to minimize the distance between a dialog turn and its corresponding semantic state in the vector space.
- Mars-G (Group-aware): A novel "diversity" approach that aims to enlarge the margin between the dialog context and semantic states. By treating all semantic states in a batch as negative samples for a given context, it forces the model to learn context representations that are fundamentally distinct from the states.
Experimental Results and Insights
The authors evaluated Mars on three major benchmarks: MultiWOZ 2.0, CamRest676, and CrossWOZ.
State-of-the-Art Performance
Mars achieved state-of-the-art (SOTA) results across all datasets:
- MultiWOZ 2.0: Achieved a Combined Score of 103.4, outperforming previous leaders like GALAXY and MTTOD without requiring additional auxiliary corpora.
- CrossWOZ: Reached 59.8% Joint Goal Accuracy, a significant improvement over the previous baseline of 54.7%.
- CamRest676: Set a new benchmark with a Combined Score of 119.0.
The Diversity Advantage
The most significant finding was that Mars-G consistently outperformed Mars-P. For instance, on MultiWOZ, the "diversity" approach (Mars-G) achieved a score of 103.4 compared to 100.7 for the "similarity" approach (Mars-P). This confirms that keeping context and state representations distinct allows the model to retain richer information, which is particularly effective for handling long-range dependencies in extended dialogues.
Low-Resource Efficiency
Mars-G demonstrated remarkable data efficiency. When trained on only 10% of the available data, it outperformed several established models that were trained on the full 100% of the dataset.
Conclusion
The Mars framework provides a new perspective on representation modeling for task-oriented dialog. By demonstrating that "different is better," it highlights the importance of avoiding information redundancy between context and semantic states. While the current model relies on manual annotations for its contrastive learning objectives, future work will explore semi-supervised methods to reduce labeling costs and improve scalability for real-world applications.
Summary updated