Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis

Yi Tay, Anh Tuan Luu, Siu Cheung Hui

Abstract

Aspect-based sentiment analysis (ABSA) tries to predict the polarity of a given document with respect to a given aspect entity. While neural network architectures have been successful in predicting the overall polarity of sentences, aspect-specific sentiment analysis still remains as an open problem. In this paper, we propose a novel method for integrating aspect information into the neural model. More specifically, we incorporate aspect information into the neural model by modeling word-aspect relationships. Our novel model, Aspect Fusion LSTM (AF-LSTM) learns to attend based on associative relationships between sentence words and aspect which allows our model to adaptively focus on the correct words given an aspect term. This ameliorates the flaws of other state-of-the-art models that utilize naive concatenations to model word-aspect similarity. Instead, our model adopts circular convolution and circular correlation to model the similarity between aspect and words and elegantly incorporates this within a differentiable neural attention framework. Finally, our model is end-to-end differentiable and highly related to convolution-correlation (holographic like) memories. Our proposed neural model achieves state-of-the-art performance on benchmark datasets, outperforming ATAE-LSTM by 4%-5% on average across multiple datasets.

Introduction

Standard sentiment analysis typically focuses on the overall polarity of a sentence. However, Aspect-Based Sentiment Analysis (ABSA) requires a more granular approach, predicting sentiment toward specific entities or categories within a sentence. For instance, in the review "I love the user interface but this app is practically useless," a model must distinguish the positive sentiment toward "user interface" from the negative sentiment toward "functionality."

To achieve this, neural models must effectively incorporate target "aspect" information. Previous state-of-the-art architectures, such as ATAE-LSTM, relied on naive concatenation of aspect vectors with context word vectors. This approach has two primary drawbacks: it forces the LSTM to simultaneously learn sequential dependencies and word-aspect relationships, and it doubles the input dimensionality, leading to an inflated parameter count and increased risk of overfitting.

The Aspect Fusion LSTM (AF-LSTM)

The proposed Aspect Fusion LSTM (AF-LSTM) addresses these issues by decoupling sequence modeling from word-aspect association. The architecture consists of several key layers:

  1. Standard Sequence Modeling: The input sentence is processed by a standard LSTM layer to generate hidden states for each word.
  2. Word-Aspect Fusion Attention Layer: Instead of concatenation, AF-LSTM employs an association layer inspired by holographic reduced representations. This layer uses circular convolution or circular correlation to fuse LSTM hidden states with aspect embeddings.
  3. Efficient Associative Operators: These operators function as compressed tensor products, capturing complex second-order interactions without adding learnable parameters. By computing these in the frequency domain via Fast Fourier Transforms (FFTs), the model maintains high efficiency ($O(n \log n)$).
  4. Attention Mechanism: Attention weights are calculated based on these fused "memory traces," allowing the model to focus adaptively on words relevant to the specific aspect query.
The AF-LSTM architecture separates sequence modeling from word-aspect fusion using circular convolution/correlation.

Experimental Setup

The model was evaluated on benchmark ABSA datasets, including SemEval 2014 Task 4 (Laptops and Restaurants) and SemEval 2015 Task 12. The evaluation covered both Aspect Term Classification (explicit mentions) and Aspect Category Classification (implicit categories). To ensure rigorous comparison, the authors introduced a standardized development split, addressing a common issue in prior work where models were tuned directly on test sets.

Results and Discussion

AF-LSTM demonstrates significant improvements over existing baselines. The circular convolution (CONV) variant consistently outperformed the ATAE-LSTM baseline by 4% to 5% on average across all datasets and tasks.

Performance comparison of AF-LSTM variants against baselines on benchmark datasets.

Key Insights

  • Parameter Efficiency: By avoiding input doubling and using parameter-less association operators, AF-LSTM reduces the parameter count from approximately 1.4M (ATAE-LSTM) to 810K.
  • Operator Comparison: Circular convolution was found to be more effective than circular correlation. The authors suggest this is because correlation's inherent asymmetry may not align well with the word-aspect fusion task.
  • Qualitative Attention: Visualizations reveal that AF-LSTM successfully shifts its attention focus based on the aspect. In sentences with mixed sentiment, the model correctly isolates the relevant descriptors for each specific aspect, whereas baseline models often exhibit "leaky" attention across the entire sentence.

Conclusion

AF-LSTM introduces a principled way to fuse aspect information into neural sentiment models using associative memory operators. By separating sequence modeling from association, the model achieves higher accuracy with fewer parameters. Future work may explore more sophisticated ways to generate aspect embeddings for multi-word phrases and further refine the normalization techniques used in the fusion layer.

Summary updated

Abstract

Aspect-based sentiment analysis (ABSA) tries to predict the polarity of a given document with respect to a given aspect entity. While neural network architectures have been successful in predicting the overall polarity of sentences, aspect-specific sentiment analysis still remains as an open problem. In this paper, we propose a novel method for integrating aspect information into the neural model. More specifically, we incorporate aspect information into the neural model by modeling word-aspect relationships. Our novel model, \textit{Aspect Fusion LSTM} (AF-LSTM) learns to attend based on associative relationships between sentence words and aspect which allows our model to adaptively focus on the correct words given an aspect term. This ameliorates the flaws of other state-of-the-art models that utilize naive concatenations to model word-aspect similarity. Instead, our model adopts circular convolution and circular correlation to model the similarity between aspect and words and elegantly incorporates this within a differentiable neural attention framework. Finally, our model is end-to-end differentiable and highly related to convolution-correlation (holographic like) memories. Our proposed neural model achieves state-of-the-art performance on benchmark datasets, outperforming ATAE-LSTM by $4\%-5\%$ on average across multiple datasets.
BibTeX
@inproceedings{tay2018learning,
  author = {Yi Tay and Anh Tuan Luu and Siu Cheung Hui},
  title = {{Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=HkZytyb_ZH},
}
Figure