Introduction
Transformer-based models are central to modern NLP, yet their internal mechanisms for determining text similarity remain largely opaque. While standard Explainable AI (XAI) tools provide first-order heatmaps highlighting individual tokens, similarity is fundamentally a relational property between two inputs. This paper introduces a framework using BiLRP (Bilinear Layer-wise Relevance Propagation) to uncover second-order interactions between token pairs, providing a higher-fidelity explanation of how models like BERT and SBERT compute similarity.
Methodology: Beyond First-Order Explanations
The core contribution is the application of BiLRP to Transformer architectures. Unlike traditional methods that ask which word is important, BiLRP identifies which pair of words—one from each sentence—drives the similarity score.
To ensure faithfulness, the authors implemented specific propagation rules for the non-linear components of Transformers, such as attention heads, layer normalization, and GeLU activations. This maintains the "conservation of relevance," ensuring that the sum of token-pair attributions matches the model's final similarity output.
Experimental Setup and Datasets
The framework was evaluated across several benchmarks, including the Semantic Textual Similarity benchmark (STSb), its multilingual counterpart (mSTSb), SICK (compositional knowledge), and BIOSSES (biomedical text). A synthetic noun-matching task was also used to validate explanations against a known ground truth.
Key Findings
1. Faithfulness and Sensitivity
Perturbation tests demonstrate that BiLRP is significantly more faithful to the model's internal logic than baseline methods. By removing token interactions identified as highly relevant by BiLRP, the model's internal representations collapse much faster than when using random or first-order baselines.
2. The "Bag of Interactions" Strategy
The analysis reveals that non-finetuned models often rely on a "bag of interactions" strategy, focusing on simple token-to-token matches (e.g., matching [SEP] or [CLS] tokens) rather than deep semantic structures. In contrast, finetuned models like SBERT prioritize meaningful NOUN-NOUN and NOUN-VERB interactions.
3. Domain-Specific Insights
In specialized domains like biomedicine, high-performing models (e.g., SGPT) were found to focus on task-relevant interactions, such as specific gene names, whereas general-purpose models relied on more generic terminology.
Conclusion
BiLRP provides a powerful lens for auditing Transformer-based similarity models. By revealing the second-order interactions that drive model predictions, researchers can better understand where models rely on semantic understanding versus superficial shortcuts. Future work may leverage these interaction-based rationales to improve model alignment and training data quality.
Summary updated