Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders

Ivan Vulic, Iñigo Casanueva, Georgios Spithourakis, Avishek Mondal, Tsung-Hsien Wen, Pawel BudzianowskiEMNLP 2022 · OpenReview

Abstract

Deploying task-oriented dialog (ToD) systems for new domains and tasks requires natural language understanding models that are 1) resource-efficient and work under low-data regimes; 2) adaptable, efficient, and quick-to-train; 3) expressive and can handle complex ToD scenarios with multiple user intents in a single utterance. Motivated by these requirements, we introduce a novel framework for multi-label intent detection (mID): MultI-ConvFiT (Multi-Label Intent Detection via Contrastive Conversational Fine-Tuning). While previous work on efficient single-label intent detection learns a classifier on top of a fixed sentence encoder (SE), we propose to 1) transform general-purpose SEs into task-specialized SEs via contrastive fine-tuning on annotated multi-label data, 2) where task specialization knowledge can be stored into lightweight adapter modules without updating the original parameters of the input SE, and then 3) we build improved mID classifiers stacked on top of fixed specialized SEs. Our main results indicate that MultI-ConvFiT yields effective mID models, with large gains over non-specialized SEs reported across a spectrum of different mID datasets, both in low-data and high-data regimes.

Introduction

In task-oriented dialog (ToD) systems, the primary objective is Intent Detection (ID)—identifying the user's goal from their utterance. While traditional systems assume a one-to-one mapping between utterances and intents, real-world interactions are frequently multi-faceted. For instance, a user might say, "I'd like to cancel my reservation and also check my account balance," necessitating multi-label Intent Detection (mID).

Existing mID solutions often face significant hurdles: they can be computationally "heavy," requiring full fine-tuning of large-scale models; they are often data-hungry, making them difficult to deploy in new domains with limited annotations; and they lack the modularity required to support diverse domains (e.g., banking and hospitality) simultaneously without incurring massive storage overhead.

The MULTI-CONVFIT Framework

To address these challenges, we propose MULTI-CONVFIT, a framework centered on "task specialization." Rather than training a classifier on top of a generic, static sentence encoder, MULTI-CONVFIT reshapes the encoder's internal semantic representation to better align with the specific requirements of the mID task.

The MULTI-CONVFIT framework architecture, illustrating the transition from general-purpose sentence encoders to specialized ones via contrastive fine-tuning and adapter modules.

The framework consists of three core components:

  1. Contrastive Specialization: Utilizing Online Contrastive Loss (OCL), the model is trained on pairs of utterances. Utterances sharing at least one intent are pulled closer in the vector space, while those with no overlapping intents are pushed apart. This process creates a more nuanced semantic map where multi-intent utterances naturally occupy subspaces between single-intent clusters.
  2. C-ADAPT (Modular Specialization): To maintain efficiency, we introduce lightweight adapter modules into the encoder. Instead of updating all parameters of the base model, only these small modules are trained while the original parameters remain frozen. This modularity allows for rapid adaptation to new domains with minimal storage requirements.
  3. Multi-Label Classification: A simple Multi-Layer Perceptron (MLP) is stacked on top of the specialized encoder. To prevent overfitting and overconfidence, we implement a corrected version of label smoothing tailored for multi-label scenarios.

Experimental Results

We evaluated MULTI-CONVFIT across four diverse datasets: BANKING (48 intents), HOTELS (40 intents), INSURANCEFAQ (118 intents), and MIXATIS (18 intents). Experiments were conducted in both low-data (simulating cold-start scenarios) and high-data regimes.

F1 scores for MULTI-CONVFIT across various datasets in both low-data (a) and high-data (b) scenarios, comparing full fine-tuning (C-FFT) with the adapter-based approach (C-ADAPT).

Our findings highlight several key advantages:

  • Performance Gains: MULTI-CONVFIT consistently outperforms non-specialized encoders. In low-data settings on the INSURANCEFAQ dataset, we observed F1 score improvements exceeding 12 points.
  • Efficiency of Adapters: The C-ADAPT approach, which trains only a fraction of the total parameters, achieves performance parity with full fine-tuning (C-FFT), as shown in the comparative results.
  • State-of-the-Art Results: The framework established new benchmarks across all tested datasets. For example, it achieved an F1 score of 94.3 on the high-data BANKING dataset, surpassing more complex existing architectures.
  • Data Efficiency: The specialization step is particularly potent when data is scarce, allowing the model to learn the underlying task logic much faster than standard supervised training.

Conclusion and Future Work

MULTI-CONVFIT demonstrates that contrastive task specialization is a powerful tool for efficient and effective multi-label intent detection. By leveraging modular adapters, the framework provides a scalable solution for multi-domain ToD systems.

However, limitations remain. The current model processes utterances in isolation, lacking the conversational context necessary to resolve ambiguities across multiple turns. Additionally, the model can struggle with subtle linguistic nuances, such as negations that completely invert an intent's meaning. Future research will explore incorporating dialogue history and implementing adaptive thresholding mechanisms to further refine intent detection accuracy.

Summary updated

Abstract

Deploying task-oriented dialog ToD systems for new domains and tasks requires natural language understanding models that are 1) resource-efficient and work under low-data regimes; 2) adaptable, efficient, and quick-to-train; 3) expressive and can handle complex ToD scenarios with multiple user intents in a single utterance. Motivated by these requirements, we introduce a novel framework for multi-label intent detection (mID): MultI-ConvFiT (Multi-Label Intent Detection via Contrastive Conversational Fine-Tuning). While previous work on efficient single-label intent detection learns a classifier on top of a fixed sentence encoder (SE), we propose to 1) transform general-purpose SEs into task-specialized SEs via contrastive fine-tuning on annotated multi-label data, 2) where task specialization knowledge can be stored into lightweight adapter modules without updating the original parameters of the input SE, and then 3) we build improved mID classifiers stacked on top of fixed specialized SEs. Our main results indicate that MultI-ConvFiT yields effective mID models, with large gains over non-specialized SEs reported across a spectrum of different mID datasets, both in low-data and high-data regimes.
BibTeX
@inproceedings{vulic2022multi,
  author = {Ivan Vulic and Iñigo Casanueva and Georgios Spithourakis and Avishek Mondal and Tsung-Hsien Wen and Pawel Budzianowski},
  title = {{Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=A1t9AnoGec},
}
Figure