LLM Explainability via Attributive Masking Learning

Oren Barkan, Yonatan Toib, Yehonatan Elisha, Jonathan Weill, Noam KoenigsteinEMNLP 2024 · DOI

Abstract

Attributive Masking Learning (AML) is a novel framework for explaining Large Language Model (LLM) predictions by training an auxiliary model to learn input masks. By optimizing a dual-masking objective—preserving predictions with minimal tokens while ensuring failure when those tokens are removed—AML achieves state-of-the-art performance across encoder and decoder models. It offers a flexible trade-off between inference speed and explanation accuracy through its pretrained (pAML) and finetuned (fAML) variants.

Introduction

As Large Language Models (LLMs) grow in complexity, they increasingly function as "black boxes," making it difficult to discern which specific input tokens drive a particular output. Existing explainability methods typically fall into three categories: gradient-based methods, which are fast but often unfaithful to decision logic; perturbation-based methods, which are accurate but computationally expensive; and decomposition-based methods, which are often architecture-specific.

Furthermore, there is no universal metric for explainability. Users may prioritize Sufficiency (the smallest subset of words maintaining the prediction) or Comprehensiveness (the prediction drop when important words are removed). AML addresses these challenges by providing a learnable, metric-agnostic framework that generalizes across different architectures.

Methodology

The core of AML is an auxiliary attribution model ($G_\theta$) that generates an attribution map (a mask) for a given input. This model is trained using a dual-masking loss function comprising three components:

  1. Preservation Loss ($L_p$): Ensures the LLM's prediction on the masked input matches the original.
  2. Inversion Loss ($L_{inv}$): Ensures the prediction diverges significantly when "important" tokens are removed.
  3. Sparsity Loss ($L_a$): Encourages the mask to be as small as possible.

The framework offers two operational modes:

  • pAML (Pretrained AML): The attribution model is trained offline. At inference, it generates explanations in a single forward pass, matching the speed of gradient methods.
  • fAML (Finetuned AML): The pretrained model is refined for a specific input instance during inference, yielding highly precise, metric-adapted explanations.

Notably, the attribution model can be significantly smaller than the target LLM (e.g., using a RoBERTa model to explain Llama2-7B), reducing computational overhead.

Experimental Setup

AML was evaluated on four text classification benchmarks: SST2, RTN, and IMDB (sentiment analysis), and EMR (emotion recognition). Additionally, the ERASER Movie Reviews dataset was used to assess alignment with human-annotated rationales.

Results

Empirical results demonstrate that fAML consistently outperforms 13 state-of-the-art methods, including SHAP, Integrated Gradients, and LIME, across nearly all datasets and metrics.

Quantitative comparison of AML variants against baseline explainability methods.

Key findings include:

  • Efficiency: While search-based methods like Solvex can take over an hour for long reviews, pAML provides competitive explanations in approximately 0.01 seconds.
  • Dual-Masking Efficacy: Ablation studies confirmed that optimizing for both preservation and inversion simultaneously is significantly more effective than optimizing for either alone.
  • Human Alignment: AML explanations showed higher agreement (F1 scores) with human rationales compared to top-performing baselines like DCMP or SIG.
  • LLM Self-Explanation: Testing revealed that LLMs are poor at self-explanation, often hallucinating tokens or failing to rank them accurately.

Conclusion

AML provides a robust and efficient framework for LLM explainability. By leveraging a dual-masking objective and a learnable attribution model, it achieves superior performance and flexibility. Future work will focus on extending AML to generative tasks and exploring more sophisticated masking strategies, such as contextually plausible token replacement.

Summary updated

BibTeX
@inproceedings{barkan2024llm,
  author = {Oren Barkan and Yonatan Toib and Yehonatan Elisha and Jonathan Weill and Noam Koenigstein},
  title = {{LLM Explainability via Attributive Masking Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.556},
}
Figure