Abstract
Due to its potential impact on business efficiency, automated customer complaint labeling and classification are of great importance for management decisions and business-level applications. The majority of the current research on automated labeling uses large and well-balanced datasets. However, customer complaints' labels are hierarchical in structure, with many labels at the lowest hierarchy level. Relying on lower-level labels leads to small and imbalanced samples, thus rendering the current automatic labeling practices not applicable to customer complaints. This article proposes an automatic labeling model incorporating the BERT and Word2Vec methods. The model is validated on electric utility customer complaints data. Within the model, the BERT method serves to obtain shallow-level text tags. Further, text enhancement is used to mitigate the problem of uneven samples that emerges when the number of labels is large. Finally, the Word2Vec model is utilized for the deep-level text analysis. The experiments demonstrate the proposed model's efficiency in automating customer complaint labeling. Consequently, the proposed model supports the enterprises in improving their service quality while simultaneously reducing labor costs.
Introduction
Large enterprises face a significant challenge in managing thousands of customer complaints. While automated text classification has advanced, real-world data often presents hierarchical structures and severe class imbalances that traditional models struggle to handle. This research addresses these issues by proposing a "layer-by-layer" framework designed to navigate the complexities of hierarchical labeling in customer service datasets.
Methodology
The proposed framework splits the labeling task into two distinct phases, supported by a robust data preprocessing stage. This "divide and conquer" approach ensures that classification is used where data is plentiful, while semantic matching handles the sparser, more specific labels.
Data Preprocessing and Text Enhancement
To combat the "imbalance trap" where deep-level labels have very few samples, the authors employ text enhancement techniques. For categories with extremely low sample sizes, the model's accuracy was found to be 0% without these enhancements.
- Back-translation: Paraphrasing Chinese text by translating it to English and back.
- Synonym Substitution: Using the Harbin Institute of Technology (HIT) synonym thesaurus to swap words while preserving semantic meaning.
The Two-Stage Labeling Process
- Shallow-Level Classification (BERT): The model first identifies broad categories (Level 2) using BERT. BERT's bidirectional nature allows it to capture nuanced context—for instance, distinguishing between "booking a flight" and "reading a book"—which is essential for understanding nuanced complaints.
- Deep-Level Semantic Matching (Word2Vec): Once the broad category is established, the search space for the specific "leaf" label (Level 3) is narrowed. The model uses Word2Vec to calculate cosine similarity between the complaint text and potential labels within that specific branch.
Experimental Results
The model was tested on a real-world, anonymized dataset provided by a Chinese electric utility company, consisting of 3,438 initial complaints across 16 shallow-level labels, expanded to 3,698 after enhancement.
Performance Metrics
The text-enhanced BERT model achieved an overall accuracy of 0.9175, significantly higher than unenhanced BERT (0.9073) and traditional TF-IDF-LR models (0.8960).
Addressing Data Scarcity
A critical finding was the model's ability to solve the "zero-accuracy" problem. For categories with extremely low sample sizes, unenhanced models failed completely. Text enhancement provided the necessary data volume for the model to generalize and successfully assign labels to these sparse categories.
Efficiency of Layer-by-Layer Matching
The hierarchical approach proved far superior to direct matching. The layer-by-layer semantic matching achieved an accuracy of 0.7357, more than doubling the performance of a direct Word2Vec matching approach (0.3018) that attempted to find the deep label without first identifying the shallow category.
Conclusion
By combining context-aware classification (BERT) with semantic similarity matching (Word2Vec) and addressing data imbalance through enhancement, this research provides a practical solution for enterprise-scale customer complaint management. The proposed model supports enterprises in improving service quality while simultaneously reducing labor costs by automating a previously manual and error-prone process.
Summary updated