ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection

Iftitahu Ni'mah, Meng Fang, Vlado Menkovski, Mykola Pechenizkiy

Abstract

The ability to detect Out-of-Domain (OOD) inputs has been a critical requirement in many real-world NLP applications, such as intent classification in dialogue systems. The inclusion of unsupported OOD inputs may lead to catastrophic failure of systems. However, it remains an empirical question whether current methods can tackle such problems reliably in a realistic scenario where zero OOD training data is available. In this study, we propose ProtoInfoMax, a new architecture that extends Prototypical Networks to simultaneously process in-domain and OOD sentences via a Mutual Information Maximization (InfoMax) objective. Experimental results show that our proposed method can substantially improve performance by up to 20% for OOD detection in low-resource settings of text classification. We also show that ProtoInfoMax is less prone to typical overconfidence errors of Neural Networks, leading to more reliable prediction results.

Introduction: The Challenge of Out-of-Domain Detection

In real-world dialogue systems, a common failure point occurs when a user provides a query that falls outside the system's trained capabilities. For instance, a chatbot designed for flight bookings might encounter a query about the weather or a sports score. This is known as the Out-of-Domain (OOD) detection problem.

Examples of In-Domain (ID) and Out-of-Domain (OOD) queries in a dialogue system.

Detecting these queries is particularly difficult due to two major hurdles:

  1. Low Resources: Systems often have very few examples of "In-Domain" (ID) data to learn from.
  2. Zero OOD Training Data: Developers cannot predict every possible OOD query, meaning models must identify "unknowns" without ever having seen them during training.

Standard Prototypical Networks, while effective for few-shot learning, tend to be overconfident. They often assign high similarity scores to OOD inputs that share superficial similarities with training data, leading to incorrect classifications rather than rejection.

Methodology: ProtoInfoMax and ProtoInfoMax++

To address these issues, the researchers introduced ProtoInfoMax and its enhanced version, ProtoInfoMax++.

Architecture diagrams for (a) ProtoInfoMax and (b) ProtoInfoMax++, illustrating the prediction flows and keyword context integration.

Prototypical Networks with InfoMax

At its core, the model uses Prototypical Networks to create a "prototype" (a mean vector) for each class in a metric space. New queries are classified based on their distance to these prototypes. The key innovation is the InfoMax Objective. By maximizing the Mutual Information (MI) between the ID support set and ID queries while penalizing OOD samples with lower MI estimates, the model learns a clearer boundary between "known" and "unknown." This is implemented using a binary cross-entropy (BCE) loss that acts as a robust rejection function.

Enhancing Features with ProtoInfoMax++

In extremely low-resource settings, sentence-level features can be sparse. ProtoInfoMax++ introduces a second prototype generator based on keyword context features (weighted by Tf-Idf). By merging standard sentence features with these keyword-based representations, the model creates a more discriminative metric space, allowing it to anchor its understanding of a domain even with only one or two example sentences.

Experimental Evaluation

The models were evaluated on two primary benchmarks: Amazon Product Reviews (sentiment classification) and AI Conversational Data (intent classification). The latter is particularly challenging as it requires distinguishing between many disjoint intent categories like "travel," "utility," and "kitchen."

Key Results

  • Performance Boost: ProtoInfoMax++ improved OOD detection performance by up to 20% in low-resource settings compared to state-of-the-art baselines like O-Proto.
  • Lower Error Rates: In intent classification tasks, the model achieved an Equal Error Rate (EER) of 0.388, significantly outperforming the baseline's 0.482.
  • Improved Calibration: Reliability is measured by Expected Calibration Error (ECE). ProtoInfoMax++ achieved an ECE of 16.40, nearly half that of the O-Proto baseline (28.74), indicating the model is much less prone to overconfidence.

Discussion and Future Work

The study demonstrates that the InfoMax objective acts as a powerful regularizer, preventing the latent space from "collapsing" and ensuring high confidence scores are only assigned when mutual information is truly high. Furthermore, the inclusion of keyword features provides a significant advantage when data is scarce.

However, limitations remain. The current keyword extraction relies on Tf-Idf, which could be replaced by more sophisticated automated methods. Additionally, while ProtoInfoMax significantly reduces overconfidence, reliability diagrams show it still occasionally overestimates accuracy for certain ID queries. Future research will focus on scaling these MI-based objectives to massive datasets and further refining calibration techniques during the training stage.

Summary updated

Abstract

The ability to detect Out-of-Domain (OOD) inputs has been a critical requirement in many real-world NLP applications. For example, intent classification in dialogue systems. The reason is that the inclusion of unsupported OOD inputs may lead to catastrophic failure of systems. However, it remains an empirical question whether current methods can tackle such problems reliably in a realistic scenario where zero OOD training data is available. In this study, we propose ProtoInfoMax, a new architecture that extends Prototypical Networks to simultaneously process in-domain and OOD sentences via Mutual Information Maximization (InfoMax) objective. Experimental results show that our proposed method can substantially improve performance up to 20% for OOD detection in low resource settings of text classification. We also show that ProtoInfoMax is less prone to typical overconfidence errors of Neural Networks, leading to more reliable prediction results.
BibTeX
@inproceedings{nimah2021protoinfomax,
  author = {Iftitahu Ni'mah and Meng Fang and Vlado Menkovski and Mykola Pechenizkiy},
  title = {{ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.138},
}
Figure