icsPLMs: Exploring Pre-trained Language Models in Intelligent Customer Service (Student Abstract)

Shixuan Liu, Chao Wang, Shuangyong Song

Introduction

General-purpose pre-trained language models (PLMs) like BERT and StructBert have fundamentally changed natural language processing. However, these models often underperform in specialized domains such as Intelligent Customer Service (ICS). In ICS environments, accurately identifying a user's specific intent—such as distinguishing a "complaint" from a "query" regarding a specific "order"—is critical for automated resolution.

Existing domain-adaptation methods typically rely on pre-defined sentiment lexicons or generic masking strategies that fail to capture the subtle nuances of customer service dialogue. The authors of this study identify a key gap: the need for a method that forces a model to prioritize domain-specific vocabulary and complex intent structures without the overhead of manually constructing massive dictionaries.

Methodology

The researchers propose icsPLMs, which optimizes domain adaptation through two primary strategies: automated lexicon construction and hierarchical multi-task training.

Domain Word Masking via Attention

Instead of masking words at random, icsPLMs identifies "domain words" automatically. The model is first fine-tuned on intent classification; the researchers then extract words that receive high attention scores in the final Transformer layer. These words (e.g., "refund," "delivery") form a domain lexicon. During pre-training, the model is tasked with predicting these domain-specific words 50% of the time, ensuring it learns their contextual nuances deeply.

Multi-task Training

The framework moves beyond binary matching to a three-level hierarchy for sentence-level tasks:

  • Query-to-Query (QQ) Matching: Distinguishes between pairs with the same intent and sub-intent (Level 1), same intent but different sub-intent (Level 2), or entirely different intents (Level 3).
  • Query-to-Answer (QA) Matching: Applies a similar three-level approach for matching user queries to potential agent responses.
  • Generic Tasks: Retains original StructBert sentence-level tasks to maintain general linguistic capabilities.
Overview of the icsPLM framework, illustrating the extraction of domain words via attention and the multi-task training architecture.

Experimental Setup

The study utilizes data from the JD Instant Messaging intelligence platform (JIMI). For pre-training, 600,000 real-world utterances and 1,560 fine-grained intents were used to extract a 23,000-word domain lexicon. Evaluation was conducted across three tasks:

  1. QM (Query Matching): A dataset of ~188k samples.
  2. IntC (Intent Classification): 140k samples across 63 user intent labels.
  3. ComIntC (Complicated Intent Classification): A specialized subset of long utterances (over 30 characters) designed to test performance on complex, "wordy" customer queries.

Key Results

The researchers compared their model against the StructBert baseline. Two versions were tested: icsPLM (masking only) and icsPLM-MT (masking plus multi-task training).

Performance comparison showing the additive benefits of domain-specific masking and multi-task training across QM, IntC, and ComIntC tasks.

As shown in the results, icsPLM-MT achieved the highest scores across all metrics. Notably, the ComIntC task saw the most significant improvement, jumping from 82.75% to 84.78% (+2.03%). This suggests that domain-specific pre-training is particularly effective at helping models filter through the "noise" of long-winded customer messages to identify core intents.

Conclusion

The icsPLM framework demonstrates that using a model's internal attention mechanism to build a domain lexicon is an effective shortcut for domain adaptation. By combining this with granular, hierarchical matching tasks, the model learns a more sophisticated understanding of the "distance" between concepts. Future work will explore whether this attention-based masking strategy can be generalized to other specialized fields like legal or medical services.

Summary updated

Abstract

Pre-trained language models have shown their high performance of text processing in intelligent customer service platforms. However, these models do not leverage domain specific information. In this paper, we propose icsPLMs optimized for intelligent customer service on both word and sentence levels. Our experimental results represent that using targeted strategies can further improve the performance of pre-trained language models in this field.
BibTeX
@inproceedings{liu2024icsplms,
  author = {Shixuan Liu and Chao Wang and Shuangyong Song},
  title = {{icsPLMs: Exploring Pre-trained Language Models in Intelligent Customer Service (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30475},
}
Figure