ConveRT: Efficient and Accurate Conversational Representations from Transformers

Matthew Henderson, Iñigo Casanueva, Nikola Mrksic, Pei-Hao Su, Tsung-Hsien Wen, Ivan Vulic

Abstract

General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.

Introduction

Modern conversational AI systems increasingly rely on transfer learning from massive pretrained language models (LMs) like BERT or RoBERTa. However, these models pose significant challenges for production environments: they require massive computational resources to train and serve, and their standard pretraining objectives (like masked language modeling on Wikipedia) do not naturally capture the dynamics of dialog.

This paper introduces ConveRT, a highly compact, efficient dual-encoder Transformer pretrained specifically on conversational data. ConveRT aims to bridge the gap between performance and efficiency, achieving state-of-the-art results on dialog tasks at a fraction of the size (59MB) and training cost (~$85) of traditional models.

Methods

The authors propose a compact dual-encoder architecture. Instead of processing the input and response together in one massive Transformer, ConveRT encodes them separately using shared weights, making retrieval highly efficient.

Architecture of the ConveRT dual-encoder model showing the transformer layers and shared parameters.

Key innovations include:

  • Quantization-Aware Training: The model is trained using mixed precision. Embeddings are quantized to 8 bits and other parameters to 16 bits, drastically shrinking the memory footprint without sacrificing performance.
  • Coprime Positional Encodings: Instead of fixed embeddings for every position, ConveRT uses two smaller matrices of coprime dimensions (47 and 11), allowing it to represent 517 distinct positions with far fewer parameters.
  • Streamlined Architecture: The model uses single-headed attention and a square-root-of-N reduction to pool sequence embeddings, speeding up both training and inference.
  • Multi-Context Modeling: To handle multi-turn conversations, a variant concatenates up to 10 previous dialog turns, treating this history as an extra feature to help rank responses.

Key Results and Insights

Pretraining on a response selection task using organic conversational data (727 million Reddit pairs) yields superior representations for dialog systems compared to general-purpose LMs.

Comparison of model parameters and disk size, highlighting ConveRT's efficiency after quantization.
  • Extreme Efficiency: The quantized ConveRT model is only 59MB, compared to BERT-base at 438MB. Pretraining took only 18 hours on older K80 GPUs, costing roughly $85.
  • Response Selection SOTA: ConveRT achieved state-of-the-art results across the board. On the Reddit test set, the multi-context version hit 71.8% (R_100@1), significantly outperforming the previous SOTA of 61.3%. It also topped leaderboards for AmazonQA (84.3%) and DSTC7-UBUNTU (71.2%).
  • Intent Classification: ConveRT outperformed the Universal Sentence Encoder (USE-LARGE) on all tested datasets. Because the encoder is frozen and compact, training these intent classifiers was 40x faster than fine-tuning BERT-LARGE.

Conclusion

ConveRT demonstrates that dialog-specific pretraining and aggressive architectural optimization can produce high-performance models that are small enough for edge deployment. While the current multi-context approach relies on simple string concatenation, the team plans to explore more advanced architectures for integrating dialog history and investigating performance in low-data regimes.

Summary updated

Abstract

General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.
BibTeX
@inproceedings{henderson2020convert,
  author = {Matthew Henderson and Iñigo Casanueva and Nikola Mrksic and Pei-Hao Su and Tsung-Hsien Wen and Ivan Vulic},
  title = {{ConveRT: Efficient and Accurate Conversational Representations from Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ORaTvKWac5z},
}
Figure