MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction

Han Jiang, Junwen Duan, Zhe Qu, Jianxin Wang

Abstract

Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation. Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method.

Introduction: The Challenge of Multi-Aspect Rationalization

In the field of unsupervised rationale extraction, the objective is to identify specific text segments (rationales) that justify a model's prediction without relying on human-labeled ground truth. For instance, in a beer review, if a model predicts a positive sentiment for "appearance," a high-quality rationale might be the phrase "pours a murky orangish-brown color."

Historically, most models have been uni-aspect, meaning they are designed to process and explain only one aspect at a time. To analyze multiple aspects—such as appearance, aroma, and palate—researchers typically had to train and deploy three separate models. This approach is not only computationally expensive and memory-intensive but also fails to capture the natural correlations that exist between different aspects of the same review.

Comparison of the traditional uni-aspect encoding approach versus the unified multi-aspect extraction in MARE.

Furthermore, previous attempts to build multi-aspect models often suffered from "information leakage." Standard Transformer attention mechanisms can struggle to strictly isolate information, leading to situations where words relevant to one aspect bleed into the representation of another, resulting in noisy or confusing rationales.

The MARE Framework

To address these limitations, the authors introduce MARE (Multi-Aspect Rationale Extractor). MARE is designed to extract explanatory snippets for multiple aspects simultaneously while maintaining strict isolation between them. The framework relies on three primary innovations:

1. Multi-Aspect Multi-Head Attention (MAMHA)

MAMHA serves as the core engine of the model. Instead of a single classification token, MARE prepends multiple special tokens (e.g., [C1], [C2], [C3]) to the input text. Each token is dedicated to a specific aspect, acting as a collector for relevant information across the document.

Detailed architecture of the Multi-Aspect Multi-Head Attention (MAMHA) mechanism, featuring the Multi-Aspect Controller (MAC).

2. Hard Deletion via MAC

A critical component of MAMHA is the Multi-Aspect Controller (MAC). Unlike previous models that used "soft" masking—which often leaves residual traces of "deleted" tokens—MARE implements a Hard Deletion mechanism. By utilizing an outer product operation on token masks and binarizing the results, MAC ensures that tokens assigned to one aspect are completely invisible to others. This prevents the "incomplete deletion" problem and ensures that each aspect's rationale is distinct and accurate.

3. Multi-Task Training

Real-world datasets are often incomplete; a hotel review might rate "Service" and "Cleanliness" but omit "Location." To handle this, MARE employs a round-robin multi-task training strategy. This allows the model to learn from all available labels across different reviews without requiring a complete set of labels for every single input, significantly reducing training overhead.

Experimental Results and Efficiency

The researchers evaluated MARE on two standard benchmarks: BeerAdvocate (Appearance, Aroma, Palate) and Hotel Reviews (Location, Service, Cleanliness).

  • State-of-the-Art Performance: MARE outperformed all established baselines, including RNP, FR, and YOFO. On the BeerAdvocate decorrelated dataset, MARE achieved a token-level F1 score of 88.8%, surpassing the previous best of 86.5%. In Hotel Reviews, it showed a 5.4% improvement in F1 score over existing methods.
  • Efficiency Gains: By consolidating multiple aspects into a single model, MARE achieved a 17.9% reduction in memory usage and a 25.2% reduction in training time compared to collaborative multi-aspect baselines.
  • Ablation Studies: Tests confirmed that removing either the Hard Deletion mechanism or the Multi-Task training component led to significant performance degradation, validating these architectural choices.

Conclusion

MARE demonstrates that leveraging internal correlations between aspects, rather than treating them as isolated tasks, leads to superior rationale extraction. By solving the "information leakage" problem through Hard Deletion and optimizing training via a multi-task approach, MARE provides a more accurate and efficient path for explaining complex, multi-dimensional model predictions. Future work will focus on adapting this encoder-based architecture for use with decoder-only Large Language Models (LLMs).

For more information and access to the source code, visit the official repository.

Summary updated

Abstract

Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation.Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method. Our codes have been available at https://github.com/CSU-NLP-Group/MARE.
BibTeX
@inproceedings{jiang2024mare,
  author = {Han Jiang and Junwen Duan and Zhe Qu and Jianxin Wang},
  title = {{MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MNKF1pBGw0},
}
Figure