Abstract
We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx's pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model's parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx's reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.
Introduction
In task-oriented dialog systems, slot labeling (or slot filling) is the process of identifying specific pieces of information in a user's request—such as a date, a restaurant name, or the number of guests. Traditionally, building these systems for new domains requires a massive amount of expert-annotated data, which is expensive and time-consuming to produce.
While large-scale pretrained models like BERT have helped, they are often pretrained on general tasks (like predicting masked words) that aren't perfectly aligned with the "span extraction" nature of slot labeling. This misalignment means that when you only have a few examples (few-shot learning), these models often struggle to generalize. The authors of ConVEx set out to bridge this gap by designing a pretraining task specifically tailored for conversational slot labeling.
Methods
The core innovation is the Pairwise Cloze pretraining task. Instead of looking at single sentences, the model looks at pairs of sentences from Reddit that share a "keyphrase" (a potential slot value).
- Template vs. Input: One sentence acts as a template where the keyphrase is replaced by a
BLANKtoken. The other is the input sentence containing the actual value. - The Goal: The model must read both and predict exactly which tokens in the input sentence correspond to the
BLANKin the template. This teaches the model an implicit understanding of "slots" (the context around the blank) and "values" (the text that fits there). - Architecture: ConVEx uses a Transformer-based encoder (based on the lightweight ConveRT model) followed by a Conditional Random Field (CRF) layer for sequence labeling.
During fine-tuning for a specific task, the massive encoder (30M parameters) is kept frozen. Only the relatively small decoder layers (800K parameters) are updated. This makes fine-tuning incredibly fast and memory-efficient.
Evaluation and Results
ConVEx was evaluated on several benchmarks, including RESTAURANTS-8K, DSTC8 (Schema-Guided Dialog), and SNIPS. The model achieved state-of-the-art (SOTA) performance across all tested datasets, with its real strength appearing in few-shot scenarios:
- Massive Gains with Tiny Data: On the RESTAURANTS-8K dataset with only 64 training examples, ConVEx improved the F1 score from 40.5 (previous SOTA) to 71.7.
- Closing the Gap: As shown in the figure below, the performance gap between ConVEx and baselines like Span-BERT is widest when data is most scarce.
- SNIPS 5-Shot: ConVEx set a new SOTA on the SNIPS 5-shot evaluation, outperforming complex, specialized few-shot models.
Through pairwise cloze, the model learns a general representation of what "values" look like in conversational contexts before it ever sees a single human-labeled "date" or "location" slot. Furthermore, ConVEx is highly efficient, requiring only 18 hours on 12 GPUs for pretraining, costing roughly $85 on cloud platforms—orders of magnitude cheaper than training a BERT-large model.
Conclusion
ConVEx demonstrates that task alignment in pretraining is crucial for data-efficient slot labeling. By pretraining the CRF decoder itself, the model can effectively transfer knowledge to new domains with minimal fine-tuning. While pretraining on Reddit data introduces potential societal biases that require careful management, the underlying approach offers a scalable and cost-effective solution for sequence labeling. Future work will explore applying the pairwise cloze objective to other tasks like Named Entity Recognition (NER) and extractive Question Answering.
Summary updated