ConVEx: Data-Efficient and Few-Shot Slot Labeling

Matthew Henderson, Ivan Vulic

Matthew Henderson, Ivan Vulić
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Introduction

ConVEx (Conversational Value Extractor) addresses the critical challenge of data scarcity in task-oriented dialog systems. By introducing a novel "pairwise cloze" pretraining task, ConVEx directly emulates the slot labeling process, allowing for significant performance gains in few-shot scenarios. This approach outperforms general-purpose models like BERT while remaining significantly more efficient to train and deploy.

Slot labeling—extracting specific entities like dates, times, or locations from user utterances—is foundational to task-oriented dialog. While transfer learning from large-scale models like BERT is the current standard, it faces two primary hurdles:

  1. Data Scarcity: Annotating specialized data for every new domain is prohibitively expensive.
  2. Task Mismatch: Standard pretraining objectives (e.g., masked language modeling) do not align well with span extraction. Consequently, new labeling layers must be trained from scratch, often failing in low-data regimes.

Methodology

ConVEx utilizes a Transformer-based architecture built on the lightweight ConveRT encoder, optimized through a new pretraining objective: Pairwise Cloze.

The Pairwise Cloze Task

In this task, the model processes pairs of sentences sharing a "keyphrase." One sentence serves as a template with the keyphrase masked (e.g., "I'd like to book for [BLANK] please"), while the other is the input containing the actual value (e.g., "Can you get me a table for 7pm?"). The model must learn to identify the span in the input that fits the template's blank.

ConVEx architecture for pretraining and fine-tuning.

Crucially, the sequence-decoding layers (CRF) are included during pretraining. During fine-tuning, the 30M-parameter encoder is frozen, and only the small (800K parameter) decoder layers are updated. This prevents overfitting and preserves the model's general extraction capabilities.

Datasets and Evaluation

The model was pretrained on 1.2 billion sentence pairs from Reddit (2015-2018) and validated on the CC100 web crawl corpus. Evaluation was conducted across several benchmarks:

  • RESTAURANTS-8K: Commercial restaurant booking data.
  • DSTC8 (Schema-Guided Dialog): Multi-domain data covering buses, events, homes, and rental cars.
  • SNIPS: A diverse NLU benchmark with seven domains (e.g., Weather, Music, Creative Work).

Key Results

ConVEx demonstrates exceptional performance in low-data regimes. On the RESTAURANTS-8K dataset with only 64 labeled examples, ConVEx achieved an F1 score of 71.7, compared to 40.5 for the previous state-of-the-art, Span-ConveRT.

Performance comparison of ConVEx against SpanConveRT and SpanBERT on RESTAURANTS-8K and DSTC8.

In a 5-shot setup on the SNIPS benchmark, ConVEx reached an average F1 of 76.8, significantly outperforming specialized few-shot models like TapNet (61.6). Furthermore, the model is highly efficient; pretraining took only 18 hours on 12 GPUs at a cost of roughly $85, and the final model size is a portable 59MB.

Discussion and Conclusion

The success of ConVEx stems from aligning pretraining directly with the downstream task. By learning to "find the value that fits the context," the model develops an inductive bias for what slots and values look like in natural language. Freezing the encoder during fine-tuning ensures that this learned knowledge is effectively transferred without being washed out by small target datasets.

Future work includes extending the model to multilingual settings and exploring zero-shot labeling capabilities, where slots are identified based solely on their names without explicit training examples.

Summary updated

Abstract

Matthew Henderson, Ivan Vulić. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
BibTeX
@inproceedings{henderson2021convex,
  author = {Matthew Henderson and Ivan Vulic},
  title = {{ConVEx: Data-Efficient and Few-Shot Slot Labeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=RClEoHdG-kN},
}
Figure