Abstract
Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation. Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method.
Introduction: The Challenge of Multi-Aspect Rationalization
In the field of unsupervised rationale extraction, the objective is to identify specific text segments (rationales) that justify a model's prediction without relying on human-labeled ground truth. For instance, in a beer review, if a model predicts a positive sentiment for "appearance," a high-quality rationale might be the phrase "pours a murky orangish-brown color."
Historically, most models have been uni-aspect, meaning they are designed to process and explain only one aspect at a time. To analyze multiple aspects—such as appearance, aroma, and palate—researchers typically had to train and deploy three separate models. This approach is not only computationally expensive and memory-intensive but also fails to capture the natural correlations that exist between different aspects of the same review.
Furthermore, previous attempts to build multi-aspect models often suffered from "information leakage." Standard Transformer attention mechanisms can struggle to strictly isolate information, leading to situations where words relevant to one aspect bleed into the representation of another, resulting in noisy or confusing rationales.
The MARE Framework
To address these limitations, the authors introduce MARE (Multi-Aspect Rationale Extractor). MARE is designed to extract explanatory snippets for multiple aspects simultaneously while maintaining strict isolation between them. The framework relies on three primary innovations:
1. Multi-Aspect Multi-Head Attention (MAMHA)
MAMHA serves as the core engine of the model. Instead of a single classification token, MARE prepends multiple special tokens (e.g., [C1], [C2], [C3]) to the input text. Each token is dedicated to a specific aspect, acting as a collector for relevant information across the document.
2. Hard Deletion via MAC
A critical component of MAMHA is the Multi-Aspect Controller (MAC). Unlike previous models that used "soft" masking—which often leaves residual traces of "deleted" tokens—MARE implements a Hard Deletion mechanism. By utilizing an outer product operation on token masks and binarizing the results, MAC ensures that tokens assigned to one aspect are completely invisible to others. This prevents the "incomplete deletion" problem and ensures that each aspect's rationale is distinct and accurate.
3. Multi-Task Training
Real-world datasets are often incomplete; a hotel review might rate "Service" and "Cleanliness" but omit "Location." To handle this, MARE employs a round-robin multi-task training strategy. This allows the model to learn from all available labels across different reviews without requiring a complete set of labels for every single input, significantly reducing training overhead.
Experimental Results and Efficiency
The researchers evaluated MARE on two standard benchmarks: BeerAdvocate (Appearance, Aroma, Palate) and Hotel Reviews (Location, Service, Cleanliness).
- State-of-the-Art Performance: MARE outperformed all established baselines, including RNP, FR, and YOFO. On the BeerAdvocate decorrelated dataset, MARE achieved a token-level F1 score of 88.8%, surpassing the previous best of 86.5%. In Hotel Reviews, it showed a 5.4% improvement in F1 score over existing methods.
- Efficiency Gains: By consolidating multiple aspects into a single model, MARE achieved a 17.9% reduction in memory usage and a 25.2% reduction in training time compared to collaborative multi-aspect baselines.
- Ablation Studies: Tests confirmed that removing either the Hard Deletion mechanism or the Multi-Task training component led to significant performance degradation, validating these architectural choices.
Conclusion
MARE demonstrates that leveraging internal correlations between aspects, rather than treating them as isolated tasks, leads to superior rationale extraction. By solving the "information leakage" problem through Hard Deletion and optimizing training via a multi-task approach, MARE provides a more accurate and efficient path for explaining complex, multi-dimensional model predictions. Future work will focus on adapting this encoder-based architecture for use with decoder-only Large Language Models (LLMs).
For more information and access to the source code, visit the official repository.
Summary updated