Abstract
Unsupervised rationale extraction aims to identify text snippets that support model predictions without requiring explicit rationale annotations. While previous research has focused on encoding aspects independently or mitigating spurious correlations, our work introduces the Multi-Aspect Rationale Extractor (MARE). MARE leverages beneficial internal correlations between aspects to improve extraction performance. We propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on "hard deletion" to encode multiple text chunks simultaneously. By prepending aspect-specific special tokens and deploying multi-task training, MARE reduces training overhead while achieving state-of-the-art performance on standard benchmarks. Ablation studies further validate the effectiveness of our hard deletion approach.
Introduction
Deep learning models are frequently criticized for being "black boxes," making interpretability a critical requirement for their deployment in sensitive domains. Rationale extraction—the process of identifying specific input snippets that drive a model's prediction—serves as a primary method for providing this transparency. In unsupervised settings, these rationales must be discovered without human-labeled ground truth.
Existing approaches typically treat rationale extraction as a "uni-aspect" task. For instance, in a multi-criteria review (e.g., a hotel review covering Location, Service, and Cleanliness), standard models must be trained independently for each aspect. This approach is computationally expensive and ignores the inherent correlations between aspects. A mention of a "spotless room" in a cleanliness context often correlates with positive sentiment in service. MARE addresses these limitations by moving toward a unified, multi-aspect framework that captures these internal correlations while preventing "information leakage" between aspects.
The MARE Framework
MARE introduces three primary innovations to enable efficient and accurate multi-aspect rationale extraction in a single forward pass.
1. Multi-Aspect Multi-Head Attention (MAMHA)
To handle multiple aspects simultaneously, MARE prepends multiple special tokens to the input text, where each token corresponds to a specific aspect (e.g., [ASP_1], [ASP_2]). The MAMHA mechanism allows the model to collaboratively encode these aspects, using the special tokens as anchors to gather relevant information from the text for each specific criterion.
2. Hard Deletion via Multi-Aspect Controller (MAC)
Information leakage is a significant challenge in multi-aspect models, where data relevant to one aspect (e.g., "Aroma") might accidentally influence the rationale for another (e.g., "Appearance"). Previous methods often used "soft" masking or broadcast masking, which can allow deleted tokens to indirectly influence the model through the transformer's attention layers.
MARE employs a Multi-Aspect Controller (MAC) that utilizes an outer product operation to create a strict attention mask. This "Hard Deletion" ensures that irrelevant tokens are completely erased from the attention calculation for a specific aspect, providing a cleaner signal and more precise rationales.
3. Multi-Task Training
To minimize training overhead, MARE utilizes a round-robin multi-task training strategy. This allows the model to iterate through different aspects during training, making it compatible with datasets where only a subset of aspects might be labeled for any given sample. This approach significantly reduces the memory and time required compared to training multiple independent models.
Experimental Results
Performance Benchmarks
We evaluated MARE on two classic rationale extraction benchmarks: BeerAdvocate (Appearance, Aroma, Palate) and Hotel Review (Location, Service, Cleanliness).
MARE established a new state-of-the-art (SOTA) across several metrics:
- BeerAdvocate: MARE outperformed the previous leading model (YOFO) by 3.3%, 1.0%, and 2.8% in token-level F1 scores for the Appearance, Aroma, and Palate aspects, respectively.
- High-Sparsity Scenarios: The model achieved a record-high average F1 score of 88.8% on high-sparse beer reviews, demonstrating its robustness even when rationales are brief.
Efficiency Gains
By consolidating multiple aspects into a single model, MARE offers substantial computational benefits. Compared to traditional multi-aspect collaborative training, MARE reduced memory usage by 17.9% and training time by 25.2%.
The Criticality of Hard Deletion
Ablation studies highlighted the necessity of the MAC mechanism. When Hard Deletion was replaced with standard Attention Mask Deletion (AMD), performance dropped precipitously. In the Palate aspect of the BeerAdvocate dataset, the F1 score plummeted from 82.0% to 3.9%, proving that "soft" masking is insufficient for maintaining aspect-specific integrity in multi-task environments.
Conclusion
MARE demonstrates that leveraging internal correlations between aspects, rather than treating them as isolated variables, leads to superior rationale extraction. By combining Multi-Aspect Multi-Head Attention with a rigorous Hard Deletion mechanism, the framework achieves SOTA results while remaining more efficient than its predecessors.
Currently, MARE is optimized for encoder-based architectures like BERT. Future work will focus on adapting this framework for decoder-only Large Language Models (LLMs), ensuring that the next generation of generative AI can provide equally robust and interpretable explanations for their outputs.
Summary updated