ConveRT: Efficient and Accurate Conversational Representations from Transformers

Matthew Henderson, Iñigo Casanueva, Nikola Mrksic, Pei-Hao Su, Tsung-Hsien Wen, Ivan Vulic

Abstract: Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, Ivan Vulić. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

Building effective conversational agents, such as chatbots and virtual assistants, requires powerful sentence encoders. While general-purpose models like BERT or the Universal Sentence Encoder (USE) provide high-quality representations, they are often computationally heavy and expensive to train for specific dialogue applications. ConveRT (Conversational Representations from Transformers) addresses this by providing a lightweight, high-performance dual-encoder specifically optimized for conversational AI.

The Problem

Standard language modeling objectives (like predicting a missing word) often fail to capture the specific nuances of human dialogue, such as the mapping between questions and appropriate answers. Furthermore, large-scale models are slow in production and costly to train. The authors aimed to create a model that is effective, affordable (costing only $85 to train), and quick to deploy, specifically tailored for the unique structure of human dialogue.

Methods

ConveRT uses a dual-encoder architecture where the context and response are embedded separately into a joint semantic space. This allows for lightning-fast retrieval compared to cross-encoders, which must process context-response pairs together.

The single-context ConveRT architecture, illustrating the dual-encoder structure with shared Transformer layers and subword embeddings.

Key technical innovations include:

  • Response Selection Pretraining: Instead of standard language modeling, the model is trained to identify the correct response to a given context from a set of negatives.
  • Extreme Efficiency: The model uses 8-bit embedding quantization and quantization-aware training. It also employs subword-level parameterization and "pruned" self-attention to keep the footprint tiny (59MB).
  • Multi-Context Modeling: A variant of the model can ingest up to 10 previous turns of dialogue history, encoding the immediate context and past history separately to handle complex conversations.
  • Transfer Learning: The pretrained encoder can be easily adapted to other tasks, such as intent classification, by adding a simple linear layer.

Datasets and Evaluation

The model was pretrained on a massive Reddit dataset comprising 727 million input/response pairs. Evaluation was conducted across several benchmarks:

  • Response Selection: Tested on Reddit, AmazonQA (e-commerce), and DSTC7-Ubuntu (technical support).
  • Intent Classification: Evaluated on BANKING77, a Shopping dataset, and a Company FAQ dataset to test transferability.

Key Results

ConveRT achieved state-of-the-art results on several benchmarks while remaining significantly more efficient than larger models.

Performance comparison of ConveRT against other models on Reddit and AmazonQA response selection tasks.
  • State-of-the-Art Performance: ConveRT set new benchmarks for response selection on Reddit (71.8% R100@1) and AmazonQA (84.3%).
  • Efficiency: The model is only 59MB (or 73MB for the multi-context version), compared to the 444MB of previous state-of-the-art dual encoders.
  • Speed and Transferability: When used for intent classification, it reached high accuracy 40 times faster than a BERT-Large-based classifier and outperformed or matched BERT-Large and USE-Large across diverse datasets.
  • The Power of Context: In complex domains like technical support (Ubuntu), looking back at 10 turns of history improved the R100@1 score from 38.2% to 71.2%.

Conclusion

ConveRT demonstrates that task-specific architectural constraints and pretraining can outperform general-purpose models at a fraction of the size and cost. By democratizing state-of-the-art conversational AI, it makes high-performance dialogue systems accessible to researchers and startups without massive compute budgets. Future work may explore more sophisticated context fusion and expansion to other languages.

Summary updated

BibTeX
@inproceedings{henderson2020convert,
  author = {Matthew Henderson and Iñigo Casanueva and Nikola Mrksic and Pei-Hao Su and Tsung-Hsien Wen and Ivan Vulic},
  title = {{ConveRT: Efficient and Accurate Conversational Representations from Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ORaTvKWac5z},
}
Figure