Scalable End-to-End Training of Knowledge Graph-Enhanced Aspect Embedding for Aspect Level Sentiment Analysis

Sk Mainul Islam, Sourangshu Bhattacharya

Introduction

Aspect-Level Sentiment Classification (ALSC) is a critical task in natural language processing that involves identifying the sentiment expressed toward specific entities or aspects within a sentence. For example, in the sentence "The battery is great, but the software is buggy," a model must distinguish between the positive sentiment for "battery" and the negative sentiment for "software."

While BERT-based models have become the standard for ALSC, they frequently encounter the "data sparsity" problem. Models often struggle to generalize when an aspect has very few training examples (the "long-tail" problem). This paper introduces AR-BERT (Aspect-Relation BERT), which addresses this by integrating external relational data from Knowledge Graphs (KGs) like DBpedia. By leveraging the fact that KGs already contain relationships between entities (e.g., "Windows 8" is related to "Windows 7"), the model can "borrow" sentiment signals from better-represented neighbors to improve predictions on rare aspects.

Methodology

Integrating massive KGs into deep learning models presents two primary challenges: scalability and noise. AR-BERT addresses these through three core innovations:

1. Two-Level Global-Local Embedding

To handle the millions of nodes in DBpedia, the authors employ a hierarchical embedding scheme. First, the entire KG is partitioned into a "cluster graph" to capture global structure. Second, a specific "subgraph" of entities relevant to the target dataset is extracted to capture local context. These structures are processed using GraphSAGE to generate compact, informative entity embeddings.

2. Incorrect Disambiguation Detection (IDD)

Automated "Wikifiers" used to link text to KG entities are often noisy. To prevent incorrect links from corrupting the model, AR-BERT uses a "probing function." This function compares the similarity of aspects in the BERT embedding space with their similarity in the KG space. If the contexts are too contradictory, the model "zeros out" the graph contribution.

The AR-BERT architecture, illustrating the two-level embedding, incorrect disambiguation detection, and the final ALSC model integration.

3. Multi-modal Explanations

The framework includes a post-hoc explanation model that identifies whether the text, the graph, or both modalities were most significant for a specific prediction, providing transparency into the model's decision-making process.

Experimental Setup

The model was evaluated on three standard ALSC benchmarks: LAPTOP (SemEval 2014), REST (Restaurant reviews), and TWITTER. The knowledge source utilized was the DBpedia page links graph, containing approximately 22 million nodes and 173 million edges.

Results and Discussion

AR-BERT consistently outperformed state-of-the-art baselines, such as BERT-ADA and SDGCN-BERT, by 2.5 to 4.1 percentage points in Macro-F1 scores.

Solving Data Scarcity

The most significant performance gains were observed for "long-tail" aspects with fewer than 20 training examples. By leveraging KG neighbors, the model effectively bypassed the lack of direct training data, demonstrating the power of external knowledge transfer.

Comparison of accuracy between the baseline and AR-BERT across different numbers of training examples, highlighting superior performance on long-tail data.

Robustness to Noise

The IDD technique proved highly effective, correctly identifying over 80% of incorrect entity disambiguations across the datasets. This ensures that the integration of external knowledge remains a net positive, even when the underlying linking tools are imperfect.

Conclusion

AR-BERT demonstrates that integrating Knowledge Graphs into BERT-based models can significantly mitigate data sparsity in ALSC. While the model currently relies on static KG snapshots and specific fusion techniques, future work will explore dynamic graph updates and more sophisticated cross-attention mechanisms to further bridge the gap between text and structured knowledge.

Summary updated

BibTeX
@inproceedings{islam2022scalable,
  author = {Sk Mainul Islam and Sourangshu Bhattacharya},
  title = {{Scalable End-to-End Training of Knowledge Graph-Enhanced Aspect Embedding for Aspect Level Sentiment Analysis}},
  booktitle = {The Web Conference},
  year = {2022},
  doi = {10.1145/3485447.3511941},
}
Figure