Abstract
Transformer-based language models (LMs) pretrained on large text collections store a wealth of semantic knowledge. However, they are often ineffective as sentence encoders off-the-shelf and lag behind conversationally pretrained encoders on tasks like intent detection (ID). ConvFiT is a simple and efficient two-stage procedure that transforms any pretrained LM into a universal conversational encoder and task-specialized sentence encoder. We demonstrate that full-blown conversational pretraining is unnecessary; LMs can be quickly adapted with small amounts of unannotated data. Specializing these encoders allows treating ID as a semantic similarity task based on interpretable nearest neighbors retrieval, achieving state-of-the-art performance, especially in few-shot setups.
Introduction
While models like BERT are packed with semantic knowledge, they are notoriously poor sentence encoders "off-the-shelf." To make them useful for conversational tasks like Intent Detection (ID)—identifying user goals such as "check balance" or "report a lost card"—researchers typically rely on massive, expensive conversational pretraining from scratch.
ConvFiT addresses two main questions: Is full-scale conversational pretraining actually necessary, or can we "rewire" existing models more efficiently? And can we move away from standard classification to treat intent detection as a semantic similarity task?
Methodology
The ConvFiT framework consists of a two-stage process designed to "rewire" general-purpose LMs for conversational understanding.
- Stage 1: Adaptive Fine-Tuning (Behavioral): The model is trained on a small sample (as little as 1%) of Reddit data. Using a dual-encoder architecture, it learns to rank correct responses to a given context, teaching the model dialogue nuances without a massive compute budget.
- Stage 2: Task-Specialised Fine-Tuning: The model is refined on the target task using contrastive learning (specifically Online Contrastive Loss). This pulls utterances with the same intent closer in vector space and pushes different intents further apart.
- Similarity-Based Inference: Instead of a classification head, the model performs a 1-Nearest Neighbor (1-NN) search against the training set at test time. The intent of the most similar training example is the predicted label.
Experimental Setup
The study utilizes several diverse benchmarks for evaluation:
- BANKING77: 13,083 examples across 77 fine-grained banking intents.
- CLINC150: 23,700 examples covering 150 intents across 10 domains.
- HWU64: 25,716 examples across 64 intents and 21 domains.
- Reddit: A massive corpus of (context, response) pairs used for Stage 1 behavioral fine-tuning.
Analysis of Representations
The effectiveness of ConvFiT is best observed through the evolution of its embedding space. The authors describe this as a "Representation Specialisation Funnel."
Off-the-shelf RoBERTa embeddings appear as an overlapping "hairball" of points. Stage 1 begins to organize these points based on conversational patterns, and Stage 2 creates tight, distinct clusters corresponding to specific intents. This visualization demonstrates how the model learns to distinguish fine-grained semantics through the two-stage process.
Key Results
ConvFiT achieves State-of-the-Art (SotA) performance across all major ID datasets:
- Few-Shot Dominance: In the 10-shot BANKING77 setup, ConvFiT achieved 87.38% accuracy, significantly outperforming standard BERT fine-tuning and specialized models like ConveRT (83.32%).
- Efficiency: The authors show that fine-tuning on just 50k Reddit pairs (0.007% of the full corpus) is sufficient to transform a general LM into an effective conversational encoder.
- Synergy: The combination of Stage 1 and Stage 2 is crucial. Skipping Stage 1 (the conversational "rewiring") leads to a noticeable drop in performance, proving that general LMs need that initial dialogue-centric nudge.
Conclusion
ConvFiT offers a robust and interpretable approach to intent detection. By using nearest-neighbor retrieval, developers can see exactly which training sentence triggered a specific intent prediction, making the system easier to debug. While inference complexity scales with the training set size, efficient sampling strategies can mitigate this. Future work will explore if this "similarity-first" approach works for other tasks like slot filling or dialogue state tracking.
Summary updated