Abstract
Building multi-domain Task-Oriented Dialogue (TOD) systems often requires extensive labeled data for each new domain. This paper introduces a novel domain-specific attention module leveraging "distributional signatures"—statistical fingerprints of word and pattern occurrences across domains. By integrating these signatures into a compact architecture, the proposed model effectively captures domain-specific nuances and identifies entities in low-resource settings. Experimental results on the In-Car Assistant and Multi-WOZ 2.1 benchmarks demonstrate state-of-the-art performance, particularly in few-shot scenarios, while maintaining a significantly smaller model size compared to existing multi-encoder approaches.
Introduction
Developing robust Task-Oriented Dialogue systems (e.g., Siri or Alexa) is traditionally resource-intensive. While data for common tasks like weather checking is abundant, transitioning to specialized domains like medical scheduling often necessitates starting data collection from scratch. Existing end-to-end models face a trade-off: large unified models often wash out domain-specific details, while multi-encoder architectures suffer from parameter explosion and poor generalization. The central challenge addressed here is how to effectively transfer knowledge between domains and adapt to new ones using limited data.
Methodology: The Signature Block
The core innovation of this work is the Signature Block, which operates in tandem with a standard Semantic Block. While the Semantic Block processes word meanings via traditional embeddings, the Signature Block analyzes how words behave statistically across different contexts.
The authors utilize three primary "Distributional Signatures":
- Inverse Word Frequency (IWF): Directs model attention away from common "stop words" and toward informative, rare terms.
- Domain Conditional Likelihood (DCL): Quantifies word specificity to a particular domain (e.g., "scheduled" in a Calendar domain vs. a Weather domain).
- Adjacent N-gram Patterns: Identifies fixed linguistic patterns surrounding entities (e.g., "away from [ENTITY]"), allowing the model to recognize novel entities based on their structural context.
These signatures are integrated via a Domain Feature Fusion layer. A joint training objective, including a specific "domain feature loss," ensures that semantic representations align with these statistical signatures.
Experimental Setup
The model was evaluated on two gold-standard multi-domain TOD benchmarks:
- In-Car Assistant (Stanford): Navigation, Weather, and Scheduling domains.
- Multi-WOZ 2.1: A large-scale dataset covering Restaurants, Attractions, and Hotels.
Key Insights and Results
Statistical Context and Pattern Recognition
A primary insight is that patterns are often more stable than specific entities in low-resource settings. By using signatures as prior knowledge, the model can identify the "slot" an entity fits into even if it has never seen that specific entity before.
Performance Benchmarks
- State-of-the-Art Results: On the In-Car Assistant dataset, the model achieved 18.0 BLEU and 63.0 Entity F1, surpassing previous benchmarks like CD-NET and DF-net.
- Few-Shot Efficiency: In a 1% data regime, the model outperformed the prior best by 1.4% in Entity F1 and 1.8% in BLEU.
- Architectural Compactness: The model is approximately one-third the size of DF-net (12.23MB vs. 34.07MB) because it avoids duplicating encoders for every new domain.
Limitations and Future Work
Despite its strengths, the approach has overhead costs. Calculating distributional signatures across large datasets is computationally intensive. Furthermore, in extremely low-resource scenarios, the statistical signatures themselves can become "noisy," potentially biasing the model. Future research will focus on optimizing signature calculation for real-time systems and improving robustness against small-sample noise.
Summary updated