CharAs-CBert: Character Assist Construction-Bert Sentence Representation Improving Sentiment Classification

Bo Chen, Weiming Peng, Jihua SongSensors 2022 · DOI

TL;DR

This paper introduces CharAs-CBert, a novel sentence representation framework that enhances sentiment classification by integrating character-level information and syntactic construction patterns with BERT. By capturing both the internal structure of words and the grammatical relationships within sentences, CharAs-CBert aims to overcome the limitations of traditional methods that often lose crucial semantic details.

What Problem Does This Paper Address?

Traditional sentence representation methods, even those based on powerful pre-trained models like BERT, often struggle to fully capture the nuances of language. They tend to overlook the internal structure of words and the grammatical "construction" of sentences, leading to a loss of global and contextual semantics. This ambiguity can arise from words having different meanings in different contexts or from sentences with similar grammatical structures but vastly different underlying meanings. For tasks like sentiment classification, where subtle semantic distinctions are critical, this loss of information can significantly impact accuracy. The paper aims to address this by developing a representation that explicitly incorporates character-level details and syntactic construction information.

Methods

The proposed CharAs-CBert framework is built upon BERT but augments it with several key components:

  • Character-Assisted Construction-BERT (CharAs-CBert): This is the overarching framework.
  • Initial Embedding Module: This module embeds words, their constituent characters, and sentence construction information into a unified low-dimensional space. It leverages BERT for word and character embeddings and introduces a "construction vector" to capture syntactic patterns.
  • Construction Vector Generation: To address the ambiguity of words in different contexts and to distinguish sentences with similar structures, a construction vector is generated. This is achieved by analyzing the grammatical roles (e.g., "CC" for coordinating conjunction, "NP" for noun phrase) within a sentence and computing an average vector feature for these constructions.
  • Character Graph Convolution Module (CharGCM): This module focuses on the internal structure of words by treating characters as nodes in a graph. It uses graph convolutions to capture the relationships between adjacent characters, thereby enriching the representation of salient words.
  • Bidirectional Independent Recurrent Neural Network (BIRM): To capture long-term dependencies and model sentence construction information from both forward and backward directions, a BIRM module is employed. This aims to improve the model's ability to perceive contextual details and overcome issues like gradient disappearance common in traditional RNNs.
  • Slice Attention Module (SAM): This module enhances word representations by considering both channel and spatial dependencies. It assigns higher weights to salient words, further refining the sentence representation.
  • Fusion and Classification: Finally, the enriched character, word, and construction features are fused. This combined representation is then fed into a SoftMax classifier for sentiment classification. A triple loss function, incorporating focal loss and class weights, is used to optimize the model, especially for imbalanced datasets.

Datasets

The paper evaluates its framework on two widely used benchmark datasets for sentiment classification:

  • SemEval 2014: This dataset is used for fine-grained sentiment classification and includes domains like "Laptop" and "Restaurant," with categories for "positive," "negative," and "neutral."
  • ACL 2014: This dataset comprises reviews of celebrity ties, products, and companies, with sentiment categories of "negative," "neutral," and "positive."

Key Insights and Ideas

  • The Power of Structure: The core insight is that simply using word embeddings, even from advanced models like BERT, is insufficient. Explicitly modeling the structure of language—both at the character level within words and at the syntactic level between words (constructions)—is crucial for robust sentence representation.
  • Construction Vectors as Ambiguity Resolvers: The introduction of "construction vectors" is a novel approach to disambiguate words that might have different meanings in different grammatical contexts. By encoding the syntactic role of words, the model can better differentiate between sentences.
  • Character-Level Granularity Matters: The CharGCM module highlights that the internal structure of words, captured through character relationships, provides valuable semantic information that can differentiate sentences with similar word compositions but different internal word structures.
  • Complementary Features: The fusion of character, word, and construction features is key. The paper argues that these different feature types are complementary, each capturing different aspects of sentence meaning, and their combination leads to a more comprehensive representation.

Key Results

The CharAs-CBert framework demonstrates strong performance across the evaluated datasets:

  • On the ACL14 dataset, the model achieved an F₁ score of 87.54% and an ACC of 92.88%.
  • On the Laptop dataset (SemEval 2014), it achieved an F₁ score of 87.03% and an ACC of 92.19%.
  • On the Restaurant dataset (SemEval 2014), it achieved an F₁ score of 82.96% and an ACC of 86.22%.
  • In comparisons with other state-of-the-art models (e.g., SBERT-att, TE-DCNN, TG-HTreeLSTM), CharAs-CBert consistently outperformed them, often by a noticeable margin in F₁ scores. For instance, it was 1.17% higher in F₁ than SBERT-att on ACL14.
  • Ablation studies confirmed the positive contribution of each proposed component (SAM, BIRM, CharGCM) to the overall performance.

Limitations and Future Work

  • Model Complexity and Redundancy: The authors acknowledge that the model structure is complex and that there might be some redundant information in the fusion stage of different feature vectors.
  • Time Efficiency: The paper notes that there is significant room for improvement in the model's time efficiency.
  • Future Directions: The authors plan to design a more efficient and concise semantic-guided sentence representation framework. This will involve exploring simpler model structures and more effective ways to capture detail semantics while ensuring accuracy and improving overall efficiency.

Summary updated

Abstract

In the process of semantic capture, traditional sentence representation methods tend to lose a lot of global and contextual semantics and ignore the internal structure information of words in sentences. To address these limitations, we propose a sentence representation method for character-assisted construction-Bert (CharAs-CBert) to improve the accuracy of sentiment text classification. First, based on the construction, a more effective construction vector is generated to distinguish the basic morphology of the sentence and reduce the ambiguity of the same word in different sentences. At the same time, it aims to strengthen the representation of salient words and effectively capture contextual semantics. Second, character feature vectors are introduced to explore the internal structure information of sentences and improve the representation ability of local and global semantics. Then, to make the sentence representation have better stability and robustness, character information, word information, and construction vectors are combined and used together for sentence representation. Finally, the evaluation and verification are carried out on various open-source baseline data such as ACL-14 and SemEval 2014 to demonstrate the validity and reliability of sentence representation, namely, the <i>F</i><sub>1</sub> and ACC are 87.54% and 92.88% on ACL14, respectively.
BibTeX
@inproceedings{chen2022charas,
  author = {Bo Chen and Weiming Peng and Jihua Song},
  title = {{CharAs-CBert: Character Assist Construction-Bert Sentence Representation Improving Sentiment Classification}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22135024},
}
Figure