Abstract
Transformer-based language models (LMs) pretrained on large text collections store a wealth of semantic knowledge but are often ineffective as sentence encoders off-the-shelf. This typically results in a performance gap compared to conversationally pretrained encoders on tasks like intent detection (ID). We propose ConvFiT, a simple and efficient two-stage procedure that transforms any pretrained LM into a universal conversational encoder and then a task-specialized sentence encoder. Our framework demonstrates that full-blown conversational pretraining is unnecessary; LMs can be quickly adapted with small amounts of unannotated data. By treating ID as a semantic similarity task based on interpretable nearest neighbors retrieval, ConvFiT achieves state-of-the-art performance, particularly in challenging few-shot setups.
Introduction
While massive Transformer models like BERT and RoBERTa are packed with semantic knowledge, they are surprisingly poor at producing useful sentence embeddings without task-specific tuning. For conversational AI, tasks like Intent Detection (ID)—identifying whether a user wants to "check a balance" or "report a lost card"—are critical. Traditionally, achieving high performance required expensive, full-scale conversational pre-training from scratch.
ConvFiT (Conversational Fine-Tuning) challenges this status quo by addressing two key questions:
- Is it possible to "rewire" existing general models into conversational ones efficiently and cheaply?
- Can we move away from rigid "black-box" classifiers toward interpretable semantic similarity tasks (i.e., finding the nearest labeled example in vector space)?
The ConvFiT Framework
ConvFiT employs a "gradual representation specialization funnel" consisting of two distinct stages designed to refine the model's internal representations for conversational tasks.
Stage 1: Behavioral Fine-Tuning
In the first stage, the model undergoes adaptive fine-tuning to learn the general dynamics of natural dialogue. Using a dual-encoder architecture, the model is trained to rank correct responses to given conversation contexts. Crucially, this stage requires only a tiny fraction (approximately 2%) of a Reddit corpus. This "rewiring" process allows the model to capture the "vibe" and structure of natural conversation without the massive computational overhead of full pre-training.
Stage 2: Task-Based Specialization
The second stage focuses on specializing the model for a specific target task, such as banking or airline information. Instead of training a standard Multi-Layer Perceptron (MLP) classification head, ConvFiT utilizes contrastive learning (specifically Online Contrastive Learning or Cosine loss). This objective forces the model to pull sentences with the same intent closer together in the vector space while pushing different intents further apart. This creates highly distinct clusters for different semantic categories.
Inference via Semantic Similarity
Once specialized, classification is performed using a $k$-Nearest Neighbor ($k$-NN) approach. To classify a new user query, the system retrieves the most similar labeled examples from the training set. This makes the model's decisions inherently more interpretable, as developers can see exactly which training instances influenced a particular prediction.
Experimental Setup and Datasets
The framework was validated using the DialoGLUE benchmark, focusing on three diverse English datasets:
- BANKING77: 13,083 examples across 77 fine-grained banking intents.
- CLINC150: 23,700 examples covering 150 intents across 10 domains.
- HWU64: 25,716 examples with 64 intents across 21 domains.
The evaluation primarily focused on "few-shot" scenarios (e.g., 10 or 30 examples per intent), which are common in real-world industrial applications where labeled data is scarce.
Results and Visual Analysis
ConvFiT achieved state-of-the-art (SotA) performance across all evaluated datasets. In the 10-shot setup for BANKING77, the RoBERTa-based ConvFiT model reached 87.38% accuracy, outperforming established specialized encoders like ConveRT (83.32%) and the Universal Sentence Encoder (84.23%).
The effectiveness of the "representation funnel" is most clearly seen through visual analysis of the embedding space.
As shown in the t-SNE plots, the off-the-shelf RoBERTa model produces a disorganized cloud of points. After Stage 1, the model begins to group related intents, though significant overlap remains. After Stage 2, the intents collapse into tight, well-separated clusters, enabling highly accurate similarity-based retrieval.
Conclusion
ConvFiT provides a lightweight and effective "recipe" for building high-performance conversational agents. By combining efficient behavioral rewiring with task-specific contrastive learning, the framework enables general-purpose LMs to excel at intent detection, even with minimal labeled data. The shift from black-box classification to similarity-based retrieval not only improves performance but also enhances the transparency and interpretability of conversational AI systems.
Summary updated