TL;DR
This paper tackles the challenge of efficiently retrieving relevant answers from business-specific FAQs in multi-tenant conversational AI systems. They propose a novel "last-layer weight-switching" strategy for fine-tuning Bi-Encoder models, achieving high accuracy with significantly reduced computational costs and memory footprints, making it practical for real-world deployments.
What Problem Does This Paper Address?
Task-oriented dialog systems often rely on Frequently Asked Questions (FAQs) to provide answers to user queries. However, business-specific FAQs present unique challenges compared to general community FAQs. Each FAQ question often represents a broad "intent" that can be phrased in numerous ways by users. Furthermore, business FAQs can have answers that are not direct text responses but rather system actions, and they need to handle out-of-scope (OOS) queries gracefully.
A significant hurdle arises when deploying such systems for multiple clients (tenants), each with their own set of FAQs. This requires optimizing for speed, accuracy, and cost, especially under production loads with strict latency requirements (under 100ms). Traditional approaches, while effective for single tenants, struggle to scale efficiently and cost-effectively across many. The paper specifically addresses the "few-shot" scenario, where only a limited number of examples are available for each intent, making robust intent detection and retrieval difficult.
Methods
The authors explore various approaches for FAQ retrieval, focusing on few-shot learning. They compare:
- Baselines: Traditional methods like BM25 and TF-IDF.
- Pre-trained and Fine-tuned Features: Utilizing pre-trained language models like BERT and ConvBERT, and specifically exploring Bi-Encoder models (like Sentence-BERT variants) for efficient embedding generation.
- Classifiers: Fine-tuning BERT as a direct intent classifier.
- Cross-Encoders: Models that process query-question pairs together for higher accuracy but with increased latency.
The core innovation is their "last-layer weight-switching" strategy. Instead of fine-tuning all layers of a pre-trained Bi-Encoder model for each tenant, they freeze most layers and only fine-tune the final layer. The weights of these final layers are tenant-specific, while the shared base model is loaded only once. During inference, the system switches the tenant-specific weights, significantly reducing memory requirements and allowing for efficient scaling across many tenants. They also explore contrastive and triplet loss functions for fine-tuning.
Datasets
The paper utilizes several publicly available intent detection datasets that are representative of task-oriented dialogues and business FAQs:
- HINT3: Reflects real-world FAQ intents and user queries.
- DialoGLUE: Includes datasets like HWU64, CLINC150, and BANKING77.
- Few-shot subsets of these datasets were specifically used, with training sets containing only 5 or 10 samples per class, to simulate real-world few-shot scenarios.
- CLINC150 and HINT3 were also used to evaluate Out-Of-Scope (OOS) performance.
Key Insights and Ideas
- Bi-Encoders are King for Few-Shot FAQ Retrieval: Fine-tuned Bi-Encoder models consistently outperform other approaches, even with a constrained last-layer fine-tuning strategy. This highlights their efficiency and effectiveness in low-data regimes.
- Last-Layer Fine-Tuning is a Game Changer for Multi-Tenancy: The proposed "weight-switching" strategy dramatically reduces memory overhead per tenant, making it feasible to support a large number of clients on the same infrastructure. This is crucial for cost-effective deployment.
- Pre-training Matters, Especially In-Domain: Models pre-trained on domain-specific data (like ConvBERT) show superior performance as feature extractors compared to general pre-trained models.
- Question Pair/Triplet Training Mitigates Few-Shot Effects: Training strategies that leverage question pairs or triplets help overcome the limitations of having very few examples per intent.
- Trade-offs are Inevitable: While larger models might offer slightly better accuracy, smaller models like MiniLM-L6 provide a better balance of accuracy, throughput, and latency for production systems.
Key Results
- Accuracy: Fine-tuned Bi-Encoder models (specifically SBERT variants like MiniLM-L6) achieved the highest Top-1 accuracy across various datasets, often exceeding 80-90% in few-shot settings.
- Memory Savings: The weight-switching strategy for MiniLM-L6 resulted in an 81.5% memory saving per tenant, and for the larger MPNet model, it was 92.5%.
- Latency: Bi-Encoder models offer a significant latency advantage due to cached embeddings, enabling them to meet strict production requirements.
- OOS Performance: The fine-tuned MiniLM-L6 model demonstrated strong OOS accuracy, with the best trade-off between in-scope and OOS accuracy varying by dataset.
Limitations and Future Work
- Model Size vs. Benefit: The benefits of the weight-switching approach are more pronounced for larger models; the advantage diminishes with lighter models.
- Inference Framework Integration: Implementing the strategy with inference frameworks like ONNX Runtime and NVIDIA Triton is not straightforward, currently limiting them to native PyTorch inference.
- Accuracy vs. Fine-tuning Depth: Fine-tuning only the last layer, while efficient, may constrain the model's potential accuracy compared to training all layers.
- Future Work: The authors suggest exploring other optimization strategies like quantization, pruning, ANN caching, and further investigation into integrating with specialized inference frameworks to enhance scalability and performance.
Summary updated