ConvFiT: Conversational Fine-Tuning of Pretrained Language Models

Ivan Vulic, Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, Iñigo Casanueva, Nikola Mrksic, Tsung-Hsien Wen

Abstract

Transformer-based language models (LMs) pretrained on large text collections store a wealth of semantic knowledge but are often ineffective as sentence encoders off-the-shelf. This typically results in a performance gap compared to conversationally pretrained encoders on tasks like intent detection (ID). We propose ConvFiT, a simple and efficient two-stage procedure that transforms any pretrained LM into a universal conversational encoder and then a task-specialized sentence encoder. Our framework demonstrates that full-blown conversational pretraining is unnecessary; LMs can be quickly adapted with small amounts of unannotated data. By treating ID as a semantic similarity task based on interpretable nearest neighbors retrieval, ConvFiT achieves state-of-the-art performance, particularly in challenging few-shot setups.

Introduction

While massive Transformer models like BERT and RoBERTa are packed with semantic knowledge, they are surprisingly poor at producing useful sentence embeddings without task-specific tuning. For conversational AI, tasks like Intent Detection (ID)—identifying whether a user wants to "check a balance" or "report a lost card"—are critical. Traditionally, achieving high performance required expensive, full-scale conversational pre-training from scratch.

ConvFiT (Conversational Fine-Tuning) challenges this status quo by addressing two key questions:

  1. Is it possible to "rewire" existing general models into conversational ones efficiently and cheaply?
  2. Can we move away from rigid "black-box" classifiers toward interpretable semantic similarity tasks (i.e., finding the nearest labeled example in vector space)?

The ConvFiT Framework

ConvFiT employs a "gradual representation specialization funnel" consisting of two distinct stages designed to refine the model's internal representations for conversational tasks.

The two-stage ConvFiT architecture for conversational fine-tuning.

Stage 1: Behavioral Fine-Tuning

In the first stage, the model undergoes adaptive fine-tuning to learn the general dynamics of natural dialogue. Using a dual-encoder architecture, the model is trained to rank correct responses to given conversation contexts. Crucially, this stage requires only a tiny fraction (approximately 2%) of a Reddit corpus. This "rewiring" process allows the model to capture the "vibe" and structure of natural conversation without the massive computational overhead of full pre-training.

Stage 2: Task-Based Specialization

The second stage focuses on specializing the model for a specific target task, such as banking or airline information. Instead of training a standard Multi-Layer Perceptron (MLP) classification head, ConvFiT utilizes contrastive learning (specifically Online Contrastive Learning or Cosine loss). This objective forces the model to pull sentences with the same intent closer together in the vector space while pushing different intents further apart. This creates highly distinct clusters for different semantic categories.

Inference via Semantic Similarity

Once specialized, classification is performed using a $k$-Nearest Neighbor ($k$-NN) approach. To classify a new user query, the system retrieves the most similar labeled examples from the training set. This makes the model's decisions inherently more interpretable, as developers can see exactly which training instances influenced a particular prediction.

Experimental Setup and Datasets

The framework was validated using the DialoGLUE benchmark, focusing on three diverse English datasets:

  • BANKING77: 13,083 examples across 77 fine-grained banking intents.
  • CLINC150: 23,700 examples covering 150 intents across 10 domains.
  • HWU64: 25,716 examples with 64 intents across 21 domains.

The evaluation primarily focused on "few-shot" scenarios (e.g., 10 or 30 examples per intent), which are common in real-world industrial applications where labeled data is scarce.

Results and Visual Analysis

ConvFiT achieved state-of-the-art (SotA) performance across all evaluated datasets. In the 10-shot setup for BANKING77, the RoBERTa-based ConvFiT model reached 87.38% accuracy, outperforming established specialized encoders like ConveRT (83.32%) and the Universal Sentence Encoder (84.23%).

The effectiveness of the "representation funnel" is most clearly seen through visual analysis of the embedding space.

t-SNE plots illustrating the representation funnel from raw RoBERTa to fully specialized ConvFiT.

As shown in the t-SNE plots, the off-the-shelf RoBERTa model produces a disorganized cloud of points. After Stage 1, the model begins to group related intents, though significant overlap remains. After Stage 2, the intents collapse into tight, well-separated clusters, enabling highly accurate similarity-based retrieval.

Conclusion

ConvFiT provides a lightweight and effective "recipe" for building high-performance conversational agents. By combining efficient behavioral rewiring with task-specific contrastive learning, the framework enables general-purpose LMs to excel at intent detection, even with minimal labeled data. The shift from black-box classification to similarity-based retrieval not only improves performance but also enhances the transparency and interpretability of conversational AI systems.

Summary updated

Abstract

Transformer-based language models (LMs) pretrained on large text collections are proven to store a wealth of semantic knowledge. However, 1) they are not effective as sentence encoders when used off-the-shelf, and 2) thus typically lag behind conversationally pretrained (e.g., via response selection) encoders on conversational tasks such as intent detection (ID). In this work, we propose ConvFiT, a simple and efficient two-stage procedure which turns any pretrained LM into a universal conversational encoder (after Stage 1 ConvFiT-ing) and task-specialised sentence encoder (after Stage 2). We demonstrate that 1) full-blown conversational pretraining is not required, and that LMs can be quickly transformed into effective conversational encoders with much smaller amounts of unannotated data; 2) pretrained LMs can be fine-tuned into task-specialised sentence encoders, optimised for the fine-grained semantics of a particular task. Consequently, such specialised sentence encoders allow for treating ID as a simple semantic similarity task based on interpretable nearest neighbours retrieval. We validate the robustness and versatility of the ConvFiT framework with such similarity-based inference on the standard ID evaluation sets: ConvFiT-ed LMs achieve state-of-the-art ID performance across the board, with particular gains in the most challenging, few-shot setups.
BibTeX
@inproceedings{vulic2021convfit,
  author = {Ivan Vulic and Pei-Hao Su and Samuel Coope and Daniela Gerz and Pawel Budzianowski and Iñigo Casanueva and Nikola Mrksic and Tsung-Hsien Wen},
  title = {{ConvFiT: Conversational Fine-Tuning of Pretrained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ObZknrDyBi},
}
Figure