TL;DR
This paper introduces ICLER, a novel In-Context Learning (ICL) method for intent classification that tackles the limitations of existing approaches in understanding fine-grained intentions within enterprise vertical domains. ICLER enhances both the embedding model's ability to capture subtle semantic information and the Large Language Model's (LLM) reasoning capabilities, leading to significantly improved accuracy.
What Problem Does This Paper Address?
Intent classification is crucial for understanding user needs in various applications. While In-Context Learning (ICL) has shown promise, it struggles with identifying nuanced intentions, especially in specialized domains. The paper identifies two main culprits: 1) embedding models' limitations in capturing subtle, domain-specific details, leading to the retrieval of irrelevant examples, and 2) LLMs' insufficient reasoning abilities, causing them to focus on surface-level semantics rather than deeper connections and business logic. This results in misclassifications, particularly when dealing with complex or ambiguous user queries.
Methods
ICLER addresses these issues through a two-pronged approach:
- Reasoning-Driven Vector Representation Enhancement: The authors propose a mechanism to improve the embedding model. This involves integrating domain knowledge and employing a multi-task learning pipeline that jointly trains reasoning-generation and vector-optimization tasks. This aims to equip the embedding model with a better understanding of complex semantic patterns, thus improving the quality of retrieved examples.
- Reasoning-Augmented Intent Classification: The enhanced embedding model is then integrated into the ICL framework. This allows LLMs to leverage the improved embeddings and perform reasoning-augmented analysis, enhancing their ability to differentiate between semantically similar intents. The method uses a dual-channel retrieval mechanism to fetch relevant instances and reasoning data.
Datasets
The paper evaluates ICLER on several datasets:
- MARC (Multilingual Amazon Review Corpus): Including its en-US, zh-CN, and ja-JP subsets, for cross-lingual user review analysis.
- Banking_intent dataset: For evaluating performance in the financial domain.
- Mistral-intent-data-1816 dataset: For testing across diverse scenarios.
- A self-constructed PC domain dataset: With over 500 labels and industrial-domain intent classification characteristics.
- MTEB benchmark: Specifically six classification tasks (MTOPDomainClassification, MassiveIntentClassification, MassiveScenarioClassification, Banking77Classification, AmazonCounterfactualClassification, and EmotionClassification) for evaluating the multi-task fine-tuned embedding model.
Key Insights and Ideas
- The critical role of reasoning in ICL: The paper strongly emphasizes that for accurate intent classification, especially in complex domains, LLMs need to go beyond surface-level semantics and engage in deep logical reasoning.
- Bridging the gap between surface and deep semantics: ICLER's core innovation lies in its ability to connect superficial textual cues with deeper, domain-specific meanings and logical relationships.
- Multi-task learning for enhanced embeddings: Jointly training embedding models for both reasoning and representation tasks proves effective in capturing nuanced semantic information.
- The impact of reasoning samples: The study highlights that using only strongly relevant reasoning samples is crucial; weakly relevant ones can actually degrade performance due to LLM "hallucinations."
Key Results
ICLER demonstrates significant improvements across various datasets:
- Vertical Domains: ICLER significantly outperforms the original ICL method in intent identification within vertical domains.
- General Datasets: It achieves accuracy improvements ranging from 0.04% to 1.14% on general datasets.
- Embedding Model Performance: The fine-tuned embedding model shows an average performance gain of 5.56% on selected classification tasks in the MTEB benchmark.
- Specific Gains: For instance, on the PC-related domain datasets, ICLER achieves an absolute accuracy gain of 4.77 points compared to the best embedding-only baseline.
Limitations and Future Work
The authors acknowledge three limitations:
- Domain Generalization: The current experiments primarily focus on PC-related and general datasets. Further exploration is needed to validate ICLER's domain generalization ability in other professional fields.
- Memory Usage: The joint optimization process for generative and embedding tasks increases memory usage during training.
- Base Model Dependency: Model performance still relies on the scale of the base LLM. Future work aims to achieve similar optimization with smaller parameter models.
Summary updated