Do Learned Sparse Retrievers Mean What Their Words Say?

Learned sparse retrievers such as SPLADE represent every query and document as a short list of weighted vocabulary words, for example bankruptcy: 1.4, insolvency: 0.9, filed: 0.7. Those lists live in an ordinary inverted index, the word-to-documents lookup table that keyword search engines use. Because each coordinate carries a word as its label, these models are routinely called interpretable. Nobody has checked whether the label is true. A coordinate named bank might fire for riverbanks and for lenders alike. It might fire on a subword fragment like ##ph that means nothing. Worst of all, it might act as an arbitrary "hash code" that links documents for reasons unrelated to the word. STAIR's authors had to build a three-stage training curriculum precisely to stop their sparse multimodal model from doing this. If vocabulary labels can lie, then the engineer auditing a ranking error, or clamping a term weight to fix it, is working from a misleading map.

This note proposes treating the vocabulary coordinates of a learned sparse retriever the way interpretability research treats neurons and sparse-autoencoder features: as hypotheses about meaning that must pass behavioral and causal tests. The work has two parts. The first is a faithfulness benchmark that measures, coordinate by coordinate, whether a sparse retriever's word labels predict its retrieval behavior. The benchmark is comparable across real-vocabulary models (SPLADE), unlabeled "pseudo-token" models, and sparse decompositions of dense retrievers. The second part is a targeted repair. A sparse autoencoder (a model that rewrites a hidden vector as a few active features drawn from a large learned dictionary) is trained on the retriever's own hidden states. It identifies which vocabulary coordinates are shared by several unrelated features. Those coordinates are then split into sense-specific coordinates that remain inverted-index compatible and keep readable labels.

Core Claim

Calling a vocabulary-grounded sparse representation "interpretable" is an empirical claim that can be tested, and it is probably false for a measurable fraction of coordinates. That fraction is concentrated in identifiable places: polysemous tokens, subword fragments, and high-weight expansion terms with no semantic tie to the text. Dictionary learning on the retriever's pre-projection states can locate these coordinates. Splitting only those coordinates by sense should raise faithfulness substantially without lowering ranking quality on MS MARCO or BEIR.

Where This Sits

Three lines of work motivate the gap. SPLADE, its successors SPLADE v2 and SPLADE-v3, and the decoder-LLM version in Scaling Sparse and Dense Retrieval in Decoder-Only LLMs project contextual token states through the language-model prediction head onto the vocabulary, then max-pool over the sequence. All of them assume, rather than test, that a coordinate's weight reflects the meaning of its word. Deep Bag-of-Words goes furthest in operationalizing that assumption: Taobao engineers manually clamp term weights to fix ranking errors. That workflow is only sound if labels are faithful, and it is our main motivating use case.

STAIR maps image and text into WordPiece tokens and reports shortcut "hash-code" tokens. This is direct evidence that vocabulary grounding can fail silently. STAIR addresses the problem with a training curriculum but never measures how many tokens remain unfaithful after training. Match Your Words! supplies a related diagnostic: the Robertson–Sparck Jones term-weight gap. It shows that SPLADE underweights rare query terms out of domain. That is a weighting failure. We treat it as one axis of our audit, but our concern is broader: whether the coordinate means its word at all.

The method comes from interpretability. Interpret and Control Dense Retrieval with Sparse Latent Features trains a sparse autoencoder on dense retriever embeddings. It adds a ranking-preserving KL loss, labels features with Neuron-to-Graph descriptions, and steers retrieval with them. It works post hoc on dense vectors, and its raw latents retrieve worse than the dense model. We instead apply dictionary learning inside an already-sparse lexical model. The goal is to audit and refine its native coordinates, not to replace them.

No More K-means (SSR) uses sparse autoencoders to build 16,384 inverted-index "pseudo-tokens" for multi-vector retrieval. It explicitly performs no interpretability analysis, so its pseudo-tokens are a natural comparison point in our benchmark. SPLATE maps frozen ColBERTv2 states into SPLADE vocabulary with a small adapter. Its authors never ask whether the adapter's words reflect ColBERT's actual matching, which makes it another subject for the audit.

Transformer visualization via dictionary learning and SpLiCE supply the decomposition tools. The first shows that early transformer layers decompose into word-sense factors. The second shows that a non-negative sparse decomposition over a named concept dictionary can be faithful and even denoising for CLIP. Category Builder shows that explicit sparse context spaces handle polysemy better than collapsed dense vectors, which motivates sense-splitting. Two cautions constrain the design. The RAGLens paper (Toward Faithful RAG with Sparse Autoencoders) notes that sparse autoencoder features are model-specific, so every analysis is per checkpoint. The broader interpretability literature reports that interventions on latent features cause semantic drift, so our causal tests must measure side effects as well as target effects.

Research Program

The first component is the faithfulness benchmark. It scores each coordinate $j$ of a sparse retriever, whether that coordinate has a word label or a generated label, on three tests.

The first test is label agreement. An LLM judge, spot-checked by humans, sees passages ranked by $w_j$ and decides whether they are about the label's meaning. A random-label control gives the chance rate.

The second test is causal selectivity. We zero coordinate $j$ in all document vectors and measure the drop in reciprocal rank of relevant documents. One group of queries is chosen so that their relevance depends on the labeled concept, identified by query and judgment text containing the word or its synonyms. A second group is matched on query frequency and does not depend on the concept. We define selectivity as

$$S_j = \Delta_{\text{concept}}(j) - \Delta_{\text{other}}(j),$$

where each $\Delta$ is the mean reciprocal-rank drop in that query group. A faithful coordinate has large positive $S_j$. A hash-code coordinate damages unrelated queries or nothing at all.

The third test is simulatability. An LLM receives only the top-20 labeled coordinates of a document vector and must pick, from five candidate queries, the one for which the document is relevant. This measures whether the labels let a reader predict the model's behavior, which is what an auditing engineer needs.

Aggregating these tests yields a per-model faithfulness profile, broken down by coordinate type: surface term versus expansion term, whole word versus subword, and high- versus low-frequency word.

The second component is diagnosis by dictionary learning. On a SPLADE-v3 checkpoint, we train a TopK sparse autoencoder on the encoder's final hidden states $\mathbf{h}_t$, the states that feed the vocabulary head. For each vocabulary coordinate $j$, we attribute its pooled activation in each document to the autoencoder features that drove it, through the linear head $\mathbf{W}_j^\top$. A coordinate counts as multi-sense if its activation mass is split across two or more features whose top contexts are dissimilar. It counts as a shortcut if its driving features are unrelated to the label according to the label-agreement judge.

The third component is sense-split retraining. Each multi-sense coordinate $j$ is replaced by up to $s$ coordinates $j\#1,\dots,j\#s$. Each new coordinate gets its own head row, initialized from the attributed feature directions. Routing to a split coordinate is computed by the same linear head, so the model stays a standard sparse encoder and the index stays a standard inverted index. We fine-tune briefly with the SPLADE-v3 distillation objective and FLOPS regularizer. Each split coordinate is labeled by its parent token plus the top co-activating tokens, for example bank [river, shore]. Splitting only lengthens the vocabulary. Postings for split terms get shorter, so query latency should not rise, and we measure it.

Baselines

The naive reference points are BM25, whose terms are faithful by construction, and unmodified SPLADE-v3 as a "do nothing" baseline. On MS MARCO, SPLADE reports BM25 at 0.184 MRR@10 (SPLADE). BM25 is the faithfulness ceiling for surface terms and the floor for effectiveness.

The strongest effectiveness competitor in the same model family is SPLADE-v3. It reports 40.2 MRR@10 on MS MARCO dev (SPLADE-v3) and is reported at 51.2 average BEIR nDCG@10 in No More K-means. The sense-split model is fine-tuned from the same checkpoint, on the same MS MARCO data and hard negatives, with the same number of fine-tuning steps given to a control run of plain SPLADE-v3. That control separates the effect of splitting from the effect of extra training. Beating this baseline means significantly higher faithfulness on all three tests, MRR@10 within 0.3 points of the control, and BEIR nDCG@10 no lower than the control. A gain on queries containing polysemous terms would count as a bonus, not a requirement.

The strongest interpretable-by-decomposition competitor is the dense-retriever sparse autoencoder of Interpret and Control. It reports BGE-base at 0.3605 MRR on MS MARCO, falling to 0.3306 when retrieving on raw latents. We run it with Neuron-to-Graph labels through the same three faithfulness tests. That shows whether real-word labels or generated labels are more faithful at matched effectiveness.

SSR is the strongest sparse-autoencoder-based inverted-index retriever, at 53.4 BEIR nDCG@10 for SSR-CLS (No More K-means). We give its pseudo-tokens LLM-generated labels from top-activating contexts and audit them identically. SPLATE's adapter vocabulary is audited as well.

Two ablations isolate the key ingredient. The first splits the same number of coordinates chosen at random rather than by autoencoder diagnosis. The second splits by k-means clustering of each token's contexts instead of autoencoder features. If either matches the full method on faithfulness, the dictionary-learning diagnosis is not doing the work.

Why not just…?

Why not just read the top-activating passages for each SPLADE coordinate and call that an audit? That is our label-agreement test, and it is cheap. Correlation alone misses the case STAIR warns about, where a coordinate looks topical on its top examples while its retrieval effect comes from linking unrelated documents. The causal selectivity score $S_j$ exists to catch this. We will report how often the two tests disagree. If they rarely do, the cheap audit suffices, and that would itself be a useful finding.

Why not just use the lexical-only SPLADE variant with no expansion? SPLADE-v3 reports a lexical version that halves retrieval FLOPS while beating SPLADE++, and restricting coordinates to words present in the text removes most shortcut risk. It does not remove polysemy or subword fragmentation, and it gives up expansion, which is what bridges vocabulary mismatch. This is a real threat to the repair half of the proposal. We include lexical-only SPLADE-v3 in the audit. If it matches the sense-split model's faithfulness with no BEIR loss, we recommend it and drop the repair.

Why not just use sparse-autoencoder pseudo-tokens like SSR and label them afterwards? SSR retrieves well, but its dimensions have no fixed meaning that an engineer can type into a query or clamp. Generated labels can also be unfaithful in their own way. The benchmark answers this question directly by scoring SSR and the dense-autoencoder features with the same tests. If generated labels prove as faithful as word labels, vocabulary grounding loses its main argument, and we will say so.

Why not just switch to a word-level or n-gram vocabulary? DeepBoW's n-gram hashing fixes fragmentation for multi-word entities. Even so, whole words remain polysemous, and hashing introduces collisions that are themselves unfaithful. We include a word-level-vocabulary ablation to measure how much of the unfaithfulness comes from fragments alone.

Proposed Experiments and Decisive Checks

All models are trained or fine-tuned on MS MARCO passage ranking. Effectiveness is measured as MRR@10 on MS MARCO dev and nDCG@10 on the 13 BEIR datasets, with latency and index size measured on a CPU inverted-index engine. Faithfulness is measured on a 2,000-coordinate stratified sample per model with the three tests. Human validation covers 300 coordinates to calibrate the LLM judge.

Polysemy is probed on a BEIR subset of queries containing ambiguous terms, identified by the sense-diagnosis step and verified by annotators. Rare-term weighting is probed on TREC-COVID and FiQA with the Robertson–Sparck Jones gap from Match Your Words!. No reported numeric gap value for SPLADE was found on Lacuna, so we recompute the baseline value ourselves.

The first decisive check concerns the audit. If more than a small fraction of high-weight SPLADE-v3 coordinates (we set 10% in advance) fail causal selectivity, then the field's interpretability claim needs qualification. If almost none fail, the audit is a negative result worth publishing, and the repair is unnecessary.

The second decisive check concerns the repair. Sense-splitting must beat both the random-split and k-means-split ablations on selectivity and simulatability. It must also hold MS MARCO and BEIR effectiveness against the equal-steps SPLADE-v3 control.

The third check is the practical one. We simulate the Deep Bag-of-Words workflow: on held-out ranking errors, an annotator clamps the labeled coordinate they believe is responsible. We measure how often the clamp fixes the error without displacing other relevant results, comparing original against sense-split labels. A clear improvement there is the result that would justify the method to people who run search engines.