A sentiment analysis model for car review texts based on adversarial training and whole word mask BERT

Xingchen Liu, Yawen Li, Yingxia Shao, Ang Li, Jian LiangarXiv 2022 · DOI · arXiv

Abstract

Sentiment analysis in specialized domains like the automotive industry faces unique challenges, including technical terminology and informal "Internet-speak." This paper introduces ATWWM-BERT, a model that integrates Whole Word Masking (WWM) and Adversarial Training (AT) to enhance sentiment classification in car reviews. By preserving the semantic integrity of brand names and technical terms while improving robustness against textual noise, ATWWM-BERT achieves significant performance gains over general-purpose models like BERT and ERNIE.

Introduction

The automotive industry relies heavily on consumer feedback, yet analyzing car reviews is difficult due to domain-specific jargon and the messy nature of online comments. Standard NLP models often fail because they tokenize text at the character level, breaking apart critical entities like brand names (e.g., "BYD"). Furthermore, informal language—redundant punctuation and slang—introduces noise that degrades classification accuracy. Existing solutions often lack the contextual depth or robustness required for these specialized datasets.

Methodology: The ATWWM-BERT Architecture

The proposed ATWWM-BERT model addresses these issues through a multi-layered approach combining pre-training enhancements and structural robustness.

The ATWWM-BERT architecture combining Whole-Word-BERT, BiLSTM, and a Feed Forward classification layer.

Whole Word Masking (WWM)

Unlike standard BERT, which masks individual characters, WWM identifies and masks entire technical terms or proper nouns (e.g., "gearbox"). This forces the model to learn these terms as cohesive semantic units, which is crucial for the automotive domain where brand identity and technical specifications carry significant sentiment weight.

Adversarial Training (AT)

To combat textual noise, the researchers implemented adversarial training. By injecting calculated "worst-case" perturbations into the word embeddings during training, the model is forced to minimize loss even in the presence of noise. This "vaccination" process makes the final classifier more resilient to the informalities of real-world user comments.

Sequence Modeling and Classification

The output from the BERT layer is processed by a Bidirectional Long Short-Term Memory (BiLSTM) network. This captures long-range dependencies and sequence features more effectively than a simple linear layer. Finally, a Multi-Layer Perceptron (MLP) performs the sentiment classification into positive, neutral, or negative categories.

Experimental Results

The authors curated a new dataset of 9,947 manually annotated car reviews from platforms like Autohome and Dongchedi. The dataset was split into training (5,000), validation (2,000), and testing (2,947) sets.

Comparison of training loss curves across different models, showing the superior convergence of the proposed ATWWM-BERT (Our Models).

ATWWM-BERT demonstrated superior performance across all key metrics:

  • Accuracy: 77.68% (vs. 68.56% for standard BERT and 70.23% for ERNIE).
  • Macro-F1 Score: 74.75%, outperforming the next best model (ERNIE) by over 7 percentage points.

Ablation studies confirmed the value of each component: Whole Word Masking contributed a ~4% accuracy boost, while Adversarial Training added another ~2.7%.

Conclusion and Future Work

ATWWM-BERT effectively bridges the gap between general-purpose NLP and domain-specific sentiment analysis. While the model significantly improves accuracy and robustness, it remains limited by BERT's 512-character input ceiling and struggles with highly subjective or extremely noisy text. Future research will explore long-text optimization and more sophisticated adversarial strategies to further refine performance in the automotive domain.

Summary updated

Abstract

In the field of car evaluation, more and more netizens choose to express their opinions on the Internet platform, and these comments will affect the decision-making of buyers and the trend of car word-of-mouth. As an important branch of natural language processing (NLP), sentiment analysis provides an effective research method for analyzing the sentiment types of massive car review texts. However, due to the lexical professionalism and large text noise of review texts in the automotive field, when a general sentiment analysis model is applied to car reviews, the accuracy of the model will be poor. To overcome these above challenges, we aim at the sentiment analysis task of car review texts. From the perspective of word vectors, pre-training is carried out by means of whole word mask of proprietary vocabulary in the automotive field, and then training data is carried out through the strategy of an adversarial training set. Based on this, we propose a car review text sentiment analysis model based on adversarial training and whole word mask BERT(ATWWM-BERT).
BibTeX
@misc{liu2022sentiment,
  author = {Xingchen Liu and Yawen Li and Yingxia Shao and Ang Li and Jian Liang},
  title = {{A sentiment analysis model for car review texts based on adversarial training and whole word mask BERT}},
  year = {2022},
  eprint = {2206.02389},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.02389},
}
Figure