TL;DR
This paper tackles "model churn" in conversational AI, where retraining can lead to inconsistent predictions. They introduce a compute-efficient method that pairs similar training examples based on their "function call signature" and uses a novel span similarity loss to encourage consistency, achieving significant churn reduction across academic, noisy, and industry datasets.
What Problem Does This Paper Address?
Conversational AI systems, like virtual assistants, rely on complex natural language understanding (NLU) models. A critical issue in deploying these models is "model churn": when a model is retrained (even with the same data and settings), its predictions can subtly change. This inconsistency erodes user trust, as users expect predictable behavior. Traditional solutions like ensembling or distillation often require significant computational resources for training or inference, making them impractical for many real-world applications with strict latency and cost constraints. This paper aims to reduce model churn efficiently, without increasing computational overhead.
Methods
The core of the proposed solution lies in a two-pronged approach applied during training:
- Pairing Examples by Function Call Signature: The authors observe that semantically similar queries often share the same "function call signature" (the non-terminal nodes in a semantic parse tree). They group training examples with identical signatures into batches. This encourages the model to learn more stable representations for these shared structures.
- Span Similarity Loss with Jensen-Shannon Divergence (JSD): Within these paired examples, they introduce a novel loss function. This loss encourages the model to produce similar representations for "spans" (e.g., entities like "midway airport" or "iowa state university") that appear in the same slot across the paired queries. They use JSD to measure the similarity between the contextual distributions of these spans, providing a "soft" similarity score. This is integrated into a pointer-generator network architecture, which is standard for structured prediction tasks like semantic parsing.
Datasets
The authors evaluate their method on several datasets:
- Academic Setting:
- TOP (Gupta et al., 2018)
- TOPv2 (Chen et al., 2020)
- MASSIVE (FitzGerald et al., 2022) - a large, multilingual dataset.
- Noisy Data Setting:
- "Simulated noisy" datasets created by Hidey et al. (2022), which involve adding systematic noise to training data.
- Industry Setting:
- An internal proprietary dataset collected from real system output.
Key Insights and Ideas
- Function Call Signature as a Proxy for Similarity: The insight that the structure of the semantic parse (the function call signature) can be a strong indicator of semantic similarity is key. This allows for pairing examples without needing explicit semantic similarity labels.
- JSD for Span Similarity: Using JSD to quantify the similarity between span distributions is a novel way to regularize representations in structured prediction, moving beyond simple exact match or hard label smoothing.
- Compute Efficiency is Paramount: The paper strongly emphasizes that their method achieves churn reduction without requiring additional compute for training or inference, making it highly practical for industry.
- Noise Robustness: The method shows particular gains on noisy datasets, suggesting that the span similarity regularization helps the model generalize better and become more robust to imperfect training data.
Key Results
The proposed method, "LS + Pairwise/JSD" (combining label smoothing with their approach), consistently reduces model churn across various settings:
- Academic: Achieved an average improvement of +3.93% in churn reduction metrics (EM@10) on TOP, TOPv2, and MASSIVE.
- Simulated Noisy Data: Showed a significant improvement of +8.09% on EM@10.
- Industry: Achieved +5.28% improvement on EM@10 on their internal dataset.
- The method often outperforms or matches co-distillation (a more compute-intensive approach) while using a single model.
- Ablation studies confirm that both the pairwise grouping and the JSD span loss are crucial for the observed improvements. Removing either component significantly degrades performance.
Limitations and Future Work
- Dataset Structure Dependency: The authors note that their approach is most effective on datasets with complex function call signatures and nested intents. For datasets with very shallow trees and simple intents, span alignment might not be beneficial and could even be detrimental to accuracy.
- Over-triggering: Encouraging span alignment can sometimes lead to "over-triggering," where the model predicts extraneous slots for argument-less intents.
- Future Work: The paper suggests exploring negative pairs to counter over-triggering and investigating the approach on datasets with simpler structures. They also highlight the potential for combining their method with co-distillation for even better performance when compute resources allow.
Summary updated