Abstract
Sentiment analysis in specialized domains like the automotive industry faces unique challenges, including technical terminology and informal "Internet-speak." This paper introduces ATWWM-BERT, a model that integrates Whole Word Masking (WWM) and Adversarial Training (AT) to enhance sentiment classification in car reviews. By preserving the semantic integrity of brand names and technical terms while improving robustness against textual noise, ATWWM-BERT achieves significant performance gains over general-purpose models like BERT and ERNIE.
Introduction
The automotive industry relies heavily on consumer feedback, yet analyzing car reviews is difficult due to domain-specific jargon and the messy nature of online comments. Standard NLP models often fail because they tokenize text at the character level, breaking apart critical entities like brand names (e.g., "BYD"). Furthermore, informal language—redundant punctuation and slang—introduces noise that degrades classification accuracy. Existing solutions often lack the contextual depth or robustness required for these specialized datasets.
Methodology: The ATWWM-BERT Architecture
The proposed ATWWM-BERT model addresses these issues through a multi-layered approach combining pre-training enhancements and structural robustness.
Whole Word Masking (WWM)
Unlike standard BERT, which masks individual characters, WWM identifies and masks entire technical terms or proper nouns (e.g., "gearbox"). This forces the model to learn these terms as cohesive semantic units, which is crucial for the automotive domain where brand identity and technical specifications carry significant sentiment weight.
Adversarial Training (AT)
To combat textual noise, the researchers implemented adversarial training. By injecting calculated "worst-case" perturbations into the word embeddings during training, the model is forced to minimize loss even in the presence of noise. This "vaccination" process makes the final classifier more resilient to the informalities of real-world user comments.
Sequence Modeling and Classification
The output from the BERT layer is processed by a Bidirectional Long Short-Term Memory (BiLSTM) network. This captures long-range dependencies and sequence features more effectively than a simple linear layer. Finally, a Multi-Layer Perceptron (MLP) performs the sentiment classification into positive, neutral, or negative categories.
Experimental Results
The authors curated a new dataset of 9,947 manually annotated car reviews from platforms like Autohome and Dongchedi. The dataset was split into training (5,000), validation (2,000), and testing (2,947) sets.
ATWWM-BERT demonstrated superior performance across all key metrics:
- Accuracy: 77.68% (vs. 68.56% for standard BERT and 70.23% for ERNIE).
- Macro-F1 Score: 74.75%, outperforming the next best model (ERNIE) by over 7 percentage points.
Ablation studies confirmed the value of each component: Whole Word Masking contributed a ~4% accuracy boost, while Adversarial Training added another ~2.7%.
Conclusion and Future Work
ATWWM-BERT effectively bridges the gap between general-purpose NLP and domain-specific sentiment analysis. While the model significantly improves accuracy and robustness, it remains limited by BERT's 512-character input ceiling and struggles with highly subjective or extremely noisy text. Future research will explore long-text optimization and more sophisticated adversarial strategies to further refine performance in the automotive domain.
Summary updated