Abstract
In an enterprise Virtual Assistant (VA) system, intent classification is the crucial component that determines how a user input is handled based on what the user wants. The VA system is expected to be a cost-efficient SaaS service with low training and inference time while achieving high accuracy even with a small number of training samples. We pretrain a transformer-based sentence embedding model with a contrastive learning objective and leverage the embedding of the model as features when training intent classification models. Our approach achieves the state-of-the-art results for few-shot scenarios and performs better than other commercial solutions on popular intent classification benchmarks. However, generating features via a transformer-based model increases the inference time, especially for longer user inputs, due to the quadratic runtime of the transformer's attention mechanism. On top of model distillation, we introduce a practical multi-task adaptation approach that configures dynamic token pruning without the need for task-specific training for intent classification. We demonstrate that this approach improves the inference speed of popular sentence transformer models without affecting model performance.
Introduction
In the development of enterprise Virtual Assistants, developers face a persistent conflict between accuracy and efficiency. High-performing Transformer models typically require significant training data to achieve high accuracy in intent classification. However, new chatbot deployments often suffer from the "few-shot challenge," where only 1–5 training examples are available per intent. Furthermore, Transformers exhibit quadratic complexity, leading to increased latency as user inputs grow longer—a significant drawback for commercial SaaS products where response time is critical.
Existing token pruning methods, which attempt to mitigate latency by discarding less important tokens, often require expensive task-specific retraining. This research presents a "plug-and-play" solution that maintains high accuracy in few-shot scenarios while significantly reducing inference time across diverse domains.
Methodology
The proposed system utilizes a two-pronged approach to optimize both performance and speed:
- Contrastive Pre-training & Distillation: A sentence embedding model is pre-trained using a contrastive learning objective, treating examples of the same intent as positive pairs. To maintain a lean architecture, knowledge is distilled from a large teacher model into a smaller student model.
- Practical Multi-task Token Pruning: This novel approach identifies unimportant tokens using existing attention scores rather than retraining. The system is tuned via three parameters: minimum sentence length to trigger pruning ($s$), the percentage of tokens to keep ($q$), and the specific layer where pruning occurs ($l$). Research found that pruning at a single early layer (e.g., layer 1) provides maximum speedup by reducing computation in all subsequent layers.
Experimental Setup
The system was evaluated on three gold-standard intent classification benchmarks:
- BANKING77: 13,083 utterances across 77 banking intents.
- CLINC150: 22,500 queries across 150 intents and 10 domains.
- HWU64: 25,716 utterances for 64 intents across 21 domains.
Testing focused on few-shot variants (1, 2, 3, and 5-shot) and utilized "difficult" test sets based on Jaccard distance and TF-IDF to prevent the model from relying solely on keyword memorization.
Key Results and Discussion
Performance Against Commercial Solutions
In head-to-head comparisons, the IBM Watsonx(LM) system consistently outperformed major commercial rivals, including Google Dialogflow, Microsoft CLU, and Cognigy. The model achieved superior Precision, Recall, and F1 scores across all three primary datasets, validating the effectiveness of the contrastive pre-training approach.
Efficiency and Pruning Impact
The implementation of the token pruning technique resulted in significant efficiency gains. Applying pruning to a standard MiniLM-L12 model yielded inference speedups ranging from 20% to 34%. Notably, pruning often acted as a "noise filter"; by discarding linguistically unimportant tokens, the model occasionally saw accuracy improvements of 1-3% as the classifier focused on the most meaningful parts of the input.
Conclusion
This research demonstrates that task-agnostic token pruning can effectively balance the trade-off between latency and accuracy in production VA systems. By leveraging attention scores for pruning and contrastive learning for few-shot accuracy, the system achieves state-of-the-art performance while reducing operational costs. Future work will explore the impact of these techniques on massive, highly imbalanced production datasets and the challenges of "intent drift" in real-world environments.
Summary updated