MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction

Han Jiang, Junwen Duan, Zhe Qu, Jianxin Wang

Abstract

Unsupervised rationale extraction aims to identify text snippets that support model predictions without requiring explicit rationale annotations. While previous research has focused on encoding aspects independently or mitigating spurious correlations, our work introduces the Multi-Aspect Rationale Extractor (MARE). MARE leverages beneficial internal correlations between aspects to improve extraction performance. We propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on "hard deletion" to encode multiple text chunks simultaneously. By prepending aspect-specific special tokens and deploying multi-task training, MARE reduces training overhead while achieving state-of-the-art performance on standard benchmarks. Ablation studies further validate the effectiveness of our hard deletion approach.

Introduction

Deep learning models are frequently criticized for being "black boxes," making interpretability a critical requirement for their deployment in sensitive domains. Rationale extraction—the process of identifying specific input snippets that drive a model's prediction—serves as a primary method for providing this transparency. In unsupervised settings, these rationales must be discovered without human-labeled ground truth.

Existing approaches typically treat rationale extraction as a "uni-aspect" task. For instance, in a multi-criteria review (e.g., a hotel review covering Location, Service, and Cleanliness), standard models must be trained independently for each aspect. This approach is computationally expensive and ignores the inherent correlations between aspects. A mention of a "spotless room" in a cleanliness context often correlates with positive sentiment in service. MARE addresses these limitations by moving toward a unified, multi-aspect framework that captures these internal correlations while preventing "information leakage" between aspects.

The MARE Framework

MARE introduces three primary innovations to enable efficient and accurate multi-aspect rationale extraction in a single forward pass.

1. Multi-Aspect Multi-Head Attention (MAMHA)

To handle multiple aspects simultaneously, MARE prepends multiple special tokens to the input text, where each token corresponds to a specific aspect (e.g., [ASP_1], [ASP_2]). The MAMHA mechanism allows the model to collaboratively encode these aspects, using the special tokens as anchors to gather relevant information from the text for each specific criterion.

2. Hard Deletion via Multi-Aspect Controller (MAC)

Information leakage is a significant challenge in multi-aspect models, where data relevant to one aspect (e.g., "Aroma") might accidentally influence the rationale for another (e.g., "Appearance"). Previous methods often used "soft" masking or broadcast masking, which can allow deleted tokens to indirectly influence the model through the transformer's attention layers.

MARE employs a Multi-Aspect Controller (MAC) that utilizes an outer product operation to create a strict attention mask. This "Hard Deletion" ensures that irrelevant tokens are completely erased from the attention calculation for a specific aspect, providing a cleaner signal and more precise rationales.

3. Multi-Task Training

To minimize training overhead, MARE utilizes a round-robin multi-task training strategy. This allows the model to iterate through different aspects during training, making it compatible with datasets where only a subset of aspects might be labeled for any given sample. This approach significantly reduces the memory and time required compared to training multiple independent models.

Experimental Results

Performance Benchmarks

We evaluated MARE on two classic rationale extraction benchmarks: BeerAdvocate (Appearance, Aroma, Palate) and Hotel Review (Location, Service, Cleanliness).

MARE established a new state-of-the-art (SOTA) across several metrics:

  • BeerAdvocate: MARE outperformed the previous leading model (YOFO) by 3.3%, 1.0%, and 2.8% in token-level F1 scores for the Appearance, Aroma, and Palate aspects, respectively.
  • High-Sparsity Scenarios: The model achieved a record-high average F1 score of 88.8% on high-sparse beer reviews, demonstrating its robustness even when rationales are brief.

Efficiency Gains

By consolidating multiple aspects into a single model, MARE offers substantial computational benefits. Compared to traditional multi-aspect collaborative training, MARE reduced memory usage by 17.9% and training time by 25.2%.

The Criticality of Hard Deletion

Ablation studies highlighted the necessity of the MAC mechanism. When Hard Deletion was replaced with standard Attention Mask Deletion (AMD), performance dropped precipitously. In the Palate aspect of the BeerAdvocate dataset, the F1 score plummeted from 82.0% to 3.9%, proving that "soft" masking is insufficient for maintaining aspect-specific integrity in multi-task environments.

Conclusion

MARE demonstrates that leveraging internal correlations between aspects, rather than treating them as isolated variables, leads to superior rationale extraction. By combining Multi-Aspect Multi-Head Attention with a rigorous Hard Deletion mechanism, the framework achieves SOTA results while remaining more efficient than its predecessors.

Currently, MARE is optimized for encoder-based architectures like BERT. Future work will focus on adapting this framework for decoder-only Large Language Models (LLMs), ensuring that the next generation of generative AI can provide equally robust and interpretable explanations for their outputs.

Summary updated

Abstract

Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation. Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method. Our codes have been available at https://github.com/CSU-NLP-Group/MARE.
BibTeX
@inproceedings{jiang2024mare,
  author = {Han Jiang and Junwen Duan and Zhe Qu and Jianxin Wang},
  title = {{MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MNKF1pBGw0},
}
Figure