Explanations that Transfer: Interpreting Medical Retrieval with Sparse AutoEncoders

Diane Bouchacourt, Gaëtan Brison

TL;DR

This paper introduces a framework for making medical information retrieval (IR) both interpretable and robust by applying Sparse Autoencoders (SAEs) to dense text embeddings. By training SAEs with a dual objective—reconstruction and contrastive relevance—the authors produce sparse, human-readable features that not only improve retrieval performance but also provide explanations that generalize across diverse medical domains.

What Problem Does This Paper Address?

Modern medical AI systems often rely on "black-box" dense embeddings to retrieve relevant knowledge from massive document collections. While these systems are powerful, they suffer from two major drawbacks:

  1. Opacity: It is nearly impossible for a clinician to understand why a specific document was retrieved, which is a significant barrier to trust and safety in healthcare.
  2. Overfitting: Fine-tuning a model on a specific medical dataset (e.g., clinical notes) often causes it to lose "transferability," performing poorly when faced with a different medical context (e.g., a biology research paper).

The authors aim to bridge this gap by transforming opaque dense vectors into a higher-dimensional sparse space where individual "neurons" correspond to distinct, interpretable medical concepts.

Methods

The authors propose a pipeline that moves from dense, uninterpretable vectors to sparse, meaningful features:

  1. Embedding: Queries and documents are first processed by a frozen, state-of-the-art text encoder (like BGE or Gemini) to produce dense embeddings.
  2. Top-K SAE: These embeddings are fed into a Sparse Autoencoder. Unlike standard SAEs that only focus on reconstruction, this model uses a dual training objective:
  3. Reconstruction Loss: Ensures the sparse code retains the information from the original dense vector.
  4. InfoNCE Contrastive Loss: Explicitly "pulls" query embeddings closer to their relevant document embeddings in the sparse space, preserving retrieval accuracy.
  5. Interpretability Phase: To understand what the learned sparse features mean, the authors use three methods:
  6. Snippets: Looking at the top documents that activate a neuron.
  7. TF-IDF: Identifying the most distinctive words associated with a neuron's activation.
  8. LLM Interpretation: Using a Large Language Model (LLM) to generate a natural language description of the concept a neuron represents.
Figure 1: Overview of the proposed pipeline. (1) Medical queries and documents are embedded by (2) a frozen text encoder. (3) A top-k sparse autoencoder (SAE) turns each embedding into a k-sparse code, trained with a joint objective (reconstruction and InfoNCE) to match queries to their relevant documents. (4) After training, documents are ranked by cosine similarity using sparse codes. (5) Neurons are interpreted in three ways that can be used to predict activation on texts and explain retrieval dynamics. (6) SAEs and neuron interpretations trained on CURE are evaluated on 5 unseen medical corpora to assess their transferability.

Datasets

The study utilizes six distinct medical retrieval datasets to test both in-domain performance and out-of-domain (OOD) generalization:

  • CURE (In-domain): A clinical retrieval benchmark used for training the SAE.
  • OOD Evaluation Sets: MIRIAD (biomedical literature), NFCorpus (medical nutrition), and three subsets of R2MED (Biology, Bioinformatics, and Medical-Sciences).

Key Insights and Ideas

  • Sparsity as a Regularizer: One of the most surprising findings is that "low capacity" (higher sparsity) acts as a form of regularization. While dense models often overfit the training data, sparse models maintain better performance when transferred to "distant" datasets (like Wikipedia-based medical queries).
  • Explanations that Transfer: The semantic meaning of a neuron (e.g., a feature representing "kidney stones") remains consistent even when the model is applied to a dataset it has never seen before. This suggests that SAEs are capturing fundamental medical concepts rather than just dataset-specific patterns.
  • Debugging Retrieval: The sparse features allow researchers to pinpoint exactly why a model failed. For example, a model might fail to retrieve the best answer because it over-weighted a general concept like "laboratory" while ignoring a crucial specific concept like "infant health."
An example of how sparse concepts explain retrieval success and failure. In the right panel, the model favors the wrong passage because it explicitly mentions "laboratory" and "retinoids," even though "infant health" was the key population required by the query.

Key Results

  • Performance Boost: On the in-domain CURE dataset, the sparse codes significantly outperformed the original dense embeddings. For the BGE model, NDCG@10 improved from 53.06% to 60.50% (at $k=256$).
  • Robust Generalization: On out-of-domain datasets, sparse models lost less performance than their dense counterparts. In several R2MED subsets, sparse representations actually matched or beat the dense models.
  • Human-Level Interpretability: In a blind test with five medical experts, there was a high correlation ($r = 0.81$) between the experts' predictions of when a neuron would fire and the actual model activations. The aggregated human-LLM system correctly classified 90% of the test passages.

Limitations and Future Work

  • Capacity Constraints: While sparse models generalize better, they still face a "capacity bottleneck" and do not always reach the peak performance of the most massive dense models on every single dataset.
  • LLM Bias: The automated evaluation of interpretability relies on an "Activation Prediction LLM," which may introduce its own biases or calibration errors.
  • Scale of Human Validation: The human expert evaluation, while successful, was conducted on a relatively small sample of neurons (30).
  • Future Directions: The authors suggest exploring "dead neuron" reduction techniques and better calibration of LLM-predicted probabilities to further refine the system's reliability.

Summary updated

Abstract

Medical AI systems that answer users' queries rely on retrieving medical knowledge from large document collections. In this context, finetuning general models on medical datasets can improve retrieval relevance. Yet the dense embeddings underlying modern retrieval systems remain largely opaque, making it difficult to understand the mechanics behind their results. Moreover, adapting to a specific medical dataset risks losing transferability to new clinical corpora. In this work, we tackle these issues by applying Sparse Autoencoders (SAEs) to medical information retrieval, transforming dense text embeddings into semantically interpretable representations. To do so, our model combines a reconstruction objective with an InfoNCE contrastive loss that directly preserves query-document relevance. We evaluate three embedding models across six medical retrieval datasets, both in-domain and out-of-domain. Trained on a medical corpus, our sparse codes improve in-domain retrieval over the original embeddings, and lose less out-of-domain performance than their dense counterparts, especially when the target corpus is distant from the training one. Importantly, our experiments show that the method yields interpretable features whose semantic content can be inferred from the queries and documents that activate them most strongly. The interpretation of these features sheds light on why a particular document was retrieved, and transfers beyond the training distribution.
BibTeX
@misc{bouchacourt2026explanations,
  author = {Diane Bouchacourt and Gaëtan Brison},
  title = {{Explanations that Transfer: Interpreting Medical Retrieval with Sparse AutoEncoders}},
  year = {2026},
  eprint = {2611.18179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18179},
}
Figure