ConvFiT: Conversational Fine-Tuning of Pretrained Language Models

Ivan Vulic, Pei-Hao Su, Sam Coope, Daniela Gerz, Pawel Budzianowski, Iñigo Casanueva, Nikola Mrksic, Tsung-Hsien Wen

Abstract

Transformer-based language models (LMs) pretrained on large text collections store a wealth of semantic knowledge. However, they are often ineffective as sentence encoders off-the-shelf and lag behind conversationally pretrained encoders on tasks like intent detection (ID). ConvFiT is a simple and efficient two-stage procedure that transforms any pretrained LM into a universal conversational encoder and task-specialized sentence encoder. We demonstrate that full-blown conversational pretraining is unnecessary; LMs can be quickly adapted with small amounts of unannotated data. Specializing these encoders allows treating ID as a semantic similarity task based on interpretable nearest neighbors retrieval, achieving state-of-the-art performance, especially in few-shot setups.

Introduction

While models like BERT are packed with semantic knowledge, they are notoriously poor sentence encoders "off-the-shelf." To make them useful for conversational tasks like Intent Detection (ID)—identifying user goals such as "check balance" or "report a lost card"—researchers typically rely on massive, expensive conversational pretraining from scratch.

ConvFiT addresses two main questions: Is full-scale conversational pretraining actually necessary, or can we "rewire" existing models more efficiently? And can we move away from standard classification to treat intent detection as a semantic similarity task?

Methodology

The ConvFiT framework consists of a two-stage process designed to "rewire" general-purpose LMs for conversational understanding.

The ConvFiT framework: Stage 1 behavioral fine-tuning, Stage 2 task-based fine-tuning, and similarity-based classification.
  • Stage 1: Adaptive Fine-Tuning (Behavioral): The model is trained on a small sample (as little as 1%) of Reddit data. Using a dual-encoder architecture, it learns to rank correct responses to a given context, teaching the model dialogue nuances without a massive compute budget.
  • Stage 2: Task-Specialised Fine-Tuning: The model is refined on the target task using contrastive learning (specifically Online Contrastive Loss). This pulls utterances with the same intent closer in vector space and pushes different intents further apart.
  • Similarity-Based Inference: Instead of a classification head, the model performs a 1-Nearest Neighbor (1-NN) search against the training set at test time. The intent of the most similar training example is the predicted label.

Experimental Setup

The study utilizes several diverse benchmarks for evaluation:

  • BANKING77: 13,083 examples across 77 fine-grained banking intents.
  • CLINC150: 23,700 examples covering 150 intents across 10 domains.
  • HWU64: 25,716 examples across 64 intents and 21 domains.
  • Reddit: A massive corpus of (context, response) pairs used for Stage 1 behavioral fine-tuning.

Analysis of Representations

The effectiveness of ConvFiT is best observed through the evolution of its embedding space. The authors describe this as a "Representation Specialisation Funnel."

t-SNE visualization of embeddings: (a) off-the-shelf RoBERTa, (b) after Stage 1, and (c) after Stage 2.

Off-the-shelf RoBERTa embeddings appear as an overlapping "hairball" of points. Stage 1 begins to organize these points based on conversational patterns, and Stage 2 creates tight, distinct clusters corresponding to specific intents. This visualization demonstrates how the model learns to distinguish fine-grained semantics through the two-stage process.

Key Results

ConvFiT achieves State-of-the-Art (SotA) performance across all major ID datasets:

  • Few-Shot Dominance: In the 10-shot BANKING77 setup, ConvFiT achieved 87.38% accuracy, significantly outperforming standard BERT fine-tuning and specialized models like ConveRT (83.32%).
  • Efficiency: The authors show that fine-tuning on just 50k Reddit pairs (0.007% of the full corpus) is sufficient to transform a general LM into an effective conversational encoder.
  • Synergy: The combination of Stage 1 and Stage 2 is crucial. Skipping Stage 1 (the conversational "rewiring") leads to a noticeable drop in performance, proving that general LMs need that initial dialogue-centric nudge.

Conclusion

ConvFiT offers a robust and interpretable approach to intent detection. By using nearest-neighbor retrieval, developers can see exactly which training sentence triggered a specific intent prediction, making the system easier to debug. While inference complexity scales with the training set size, efficient sampling strategies can mitigate this. Future work will explore if this "similarity-first" approach works for other tasks like slot filling or dialogue state tracking.

Summary updated

Abstract

Transformer-based language models (LMs) pretrained on large text collections are proven to store a wealth of semantic knowledge. However, 1) they are not effective as sentence encoders when used off-the-shelf, and 2) thus typically lag behind conversationally pretrained (e.g., via response selection) encoders on conversational tasks such as intent detection (ID). In this work, we propose ConvFiT, a simple and efficient two-stage procedure which turns any pretrained LM into a universal conversational encoder (after Stage 1 ConvFiT-ing) and task-specialised sentence encoder (after Stage 2). We demonstrate that 1) full-blown conversational pretraining is not required, and that LMs can be quickly transformed into effective conversational encoders with much smaller amounts of unannotated data; 2) pretrained LMs can be fine-tuned into task-specialised sentence encoders, optimised for the fine-grained semantics of a particular task. Consequently, such specialised sentence encoders allow for treating ID as a simple semantic similarity task based on interpretable nearest neighbours retrieval. We validate the robustness and versatility of the ConvFiT framework with such similarity-based inference on the standard ID evaluation sets: ConvFiT-ed LMs achieve state-of-the-art ID performance across the board, with particular gains in the most challenging, few-shot setups.
BibTeX
@inproceedings{vulic2021convfit,
  author = {Ivan Vulic and Pei-Hao Su and Sam Coope and Daniela Gerz and Pawel Budzianowski and Iñigo Casanueva and Nikola Mrksic and Tsung-Hsien Wen},
  title = {{ConvFiT: Conversational Fine-Tuning of Pretrained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ObZknrDyBi},
}
Figure