MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers

You Zhang, Jin Wang, Liang-Chih Yu, Xuejie ZhangACL 2021

TL;DR

This paper introduces MA-BERT, a novel Transformer-based model that enhances sentiment analysis by effectively integrating external attribute knowledge (like user and product information) into both text representations and attention mechanisms. MA-BERT outperforms existing methods by leveraging a new "multi-attribute transformer" encoder, allowing for richer, more context-aware text representations without requiring additional pre-training.

What Problem Does This Paper Address?

Sentiment analysis, a crucial task in understanding user opinions, traditionally relies on plain text. However, the authors argue that incorporating external information, such as user preferences or product characteristics, can significantly improve performance. Previous attempts to integrate this "attribute knowledge" have primarily involved either concatenating attributes to text representations or using them as simple biases in attention mechanisms. The former can lead to a lack of nuanced interaction, while the latter might not fully update word representations, potentially losing valuable attribute information. This paper aims to bridge this gap by proposing a more sophisticated method for integrating multi-attribute knowledge into deep learning models for sentiment analysis.

Methods

The core innovation is the Multi-Attribute BERT (MA-BERT) model. It builds upon the powerful BERT architecture by adding custom MA-Transformer encoders as extra layers. The key novelty lies within these MA-Transformer encoders:

  • Multi-Attribute Attention (MAA): Instead of just biasing attention, MAA injects attribute information directly into both the query, key, and value projections of the attention mechanism. This allows for a more dynamic and interactive incorporation of attributes.
  • Bilinear Interaction: The MA-Transformer uses bilinear interactions to combine text and attribute representations, enabling a deeper fusion of information.
  • Universal Layer Design: The MA-Transformer is designed as a universal layer that can be stacked on top of a pre-trained BERT model. This allows MA-BERT to be initialized from a BERT checkpoint and fine-tuned for downstream tasks without costly additional pre-training.
  • Multi-Head Mechanism: Similar to standard Transformers, MA-Transformers employ a multi-head mechanism to capture diverse relational patterns between text and attributes.

Datasets

The paper evaluates MA-BERT on three standard benchmark datasets for sentiment polarity classification:

  • IMDB
  • Yelp-2013
  • Yelp-2014

Key Insights and Ideas

  • Attribute knowledge is crucial for nuanced sentiment: Simply adding attribute information isn't enough; how it's integrated matters significantly. MA-BERT's approach of fusing attributes into both representations and attention is key.
  • Beyond bias: Deeper attribute integration: The paper demonstrates that going beyond using attributes as simple biases in attention (as in some prior work) leads to better performance. The bilinear interaction and direct injection into Q, K, V projections are more effective.
  • Leveraging pre-trained models efficiently: The MA-Transformer's design as a plug-and-play layer on top of BERT allows for efficient transfer learning, inheriting BERT's language understanding capabilities without re-training from scratch.
  • Distinguishing user preferences: The authors highlight how current attention models might struggle to differentiate sentiment based on individual user preferences (e.g., liking spicy food). MA-BERT's richer attribute integration aims to address this by allowing for more personalized interpretations.

Key Results

MA-BERT consistently outperformed various baseline models across all three benchmark datasets.

  • Overall Superiority: MA-BERT achieved the best results on IMDB, Yelp-2013, and Yelp-2014 for sentiment polarity classification.
  • Outperforming BERT: It surpassed pre-trained BERT models that did not incorporate external attributes.
  • Advantage over Attribute-Aware Methods: MA-BERT also outperformed existing methods that incorporated user and product information, including those using concatenation or simpler attention bias strategies. For instance, on the IMDB dataset, MA-BERT achieved an accuracy of 57.3% and an RMSE of 1.042, which were the best among all tested methods.

Limitations and Future Work

  • Pre-training Phase: The authors mention that MA-BERT is fine-tuned on downstream tasks. Future work could explore incorporating similar multiple attributes into PTMs during their pre-training phases for even deeper integration.
  • Generalizability: While MA-BERT is presented as a universal layer, its application to tasks beyond sentiment analysis is a natural extension. The authors suggest that MA-Transformers could be used for other tasks by stacking them on BERT-based models.
  • Attribute Types: The current work focuses on user and product attributes. Exploring a wider range of attribute types and their integration could be a future direction.

Summary updated

BibTeX
@inproceedings{zhang2021ma,
  author = {You Zhang and Jin Wang and Liang-Chih Yu and Xuejie Zhang},
  title = {{MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure