Introduction
General-purpose pre-trained language models (PLMs) like BERT and StructBert have fundamentally changed natural language processing. However, these models often underperform in specialized domains such as Intelligent Customer Service (ICS). In ICS environments, accurately identifying a user's specific intent—such as distinguishing a "complaint" from a "query" regarding a specific "order"—is critical for automated resolution.
Existing domain-adaptation methods typically rely on pre-defined sentiment lexicons or generic masking strategies that fail to capture the subtle nuances of customer service dialogue. The authors of this study identify a key gap: the need for a method that forces a model to prioritize domain-specific vocabulary and complex intent structures without the overhead of manually constructing massive dictionaries.
Methodology
The researchers propose icsPLMs, which optimizes domain adaptation through two primary strategies: automated lexicon construction and hierarchical multi-task training.
Domain Word Masking via Attention
Instead of masking words at random, icsPLMs identifies "domain words" automatically. The model is first fine-tuned on intent classification; the researchers then extract words that receive high attention scores in the final Transformer layer. These words (e.g., "refund," "delivery") form a domain lexicon. During pre-training, the model is tasked with predicting these domain-specific words 50% of the time, ensuring it learns their contextual nuances deeply.
Multi-task Training
The framework moves beyond binary matching to a three-level hierarchy for sentence-level tasks:
- Query-to-Query (QQ) Matching: Distinguishes between pairs with the same intent and sub-intent (Level 1), same intent but different sub-intent (Level 2), or entirely different intents (Level 3).
- Query-to-Answer (QA) Matching: Applies a similar three-level approach for matching user queries to potential agent responses.
- Generic Tasks: Retains original StructBert sentence-level tasks to maintain general linguistic capabilities.
Experimental Setup
The study utilizes data from the JD Instant Messaging intelligence platform (JIMI). For pre-training, 600,000 real-world utterances and 1,560 fine-grained intents were used to extract a 23,000-word domain lexicon. Evaluation was conducted across three tasks:
- QM (Query Matching): A dataset of ~188k samples.
- IntC (Intent Classification): 140k samples across 63 user intent labels.
- ComIntC (Complicated Intent Classification): A specialized subset of long utterances (over 30 characters) designed to test performance on complex, "wordy" customer queries.
Key Results
The researchers compared their model against the StructBert baseline. Two versions were tested: icsPLM (masking only) and icsPLM-MT (masking plus multi-task training).
As shown in the results, icsPLM-MT achieved the highest scores across all metrics. Notably, the ComIntC task saw the most significant improvement, jumping from 82.75% to 84.78% (+2.03%). This suggests that domain-specific pre-training is particularly effective at helping models filter through the "noise" of long-winded customer messages to identify core intents.
Conclusion
The icsPLM framework demonstrates that using a model's internal attention mechanism to build a domain lexicon is an effective shortcut for domain adaptation. By combining this with granular, hierarchical matching tasks, the model learns a more sophisticated understanding of the "distance" between concepts. Future work will explore whether this attention-based masking strategy can be generalized to other specialized fields like legal or medical services.
Summary updated