Abstract
Multimodal representations that enable cross-modal retrieval are widely used but often lack interpretability, making it difficult to explain or control retrieved results. We propose an approach that generates fixed-size, sparse, and disentangled embeddings. Unlike previous sparse models that scale with vocabulary size, our method maintains a compact dimensionality while offering superior control for retrieval tasks. We demonstrate the utility of these representations using challenging "exclusion queries" (e.g., "A but not B") on the MSCOCO and Conceptual Captions benchmarks. Our experiments show that our approach outperforms traditional dense models like CLIP and BLIP by up to 11% in AP@10, and sparse models like VDR by up to 21%.
Introduction: The Problem of Entanglement
In multimodal retrieval, users often need to filter results using negation. However, popular dense models like CLIP or BLIP are "entangled"—their internal features are fused in a way that makes it impossible to isolate specific concepts like "basketball" from broader categories like "sports." If a user attempts to subtract a "basketball" vector from a "sports" vector in a dense space, the resulting vector often loses its semantic meaning entirely.
Existing "sparse" models, such as Vocabulary Disentangled Retrieval (VDR), attempt to solve this by mapping every word in a vocabulary to its own dimension. While effective for disentanglement, this results in massive, computationally expensive embeddings. This paper addresses the challenge of creating compact, fixed-size embeddings that remain interpretable and disentangled enough to handle complex exclusion logic.
Methodology: The Three-Step Pipeline
The authors propose a three-step training process to generate sparse representations ($SR_{clip}$ and $SR_{blip}$):
- Sparse Word Embeddings: The process begins by training a Sparse Autoencoder on pretrained word embeddings (like GloVe). This forces the model to represent words using only a few active dimensions, ensuring that semantically similar words activate overlapping dimensions while distinct concepts remain separated.
- Sentence Aggregation: Sparse sentence embeddings are then created by aggregating the sparse word vectors from the first step.
- Biencoder-Decoder Training: This is the core innovation. The model uses a biencoder-decoder architecture where encoders take standard pretrained image/text features (from CLIP or BLIP) and map them to a sparse latent space. Crucially, the sparse sentence embeddings from Step 2 act as a "guiding bias," ensuring the model learns to associate specific visual and textual concepts with specific dimensions in the fixed-size vector.
Exclusion Retrieval Mechanism
Because the resulting dimensions are disentangled, exclusion becomes a simple algebraic problem. To perform a query like "A but not B," the model identifies the active dimensions for "B" (the concept to be excluded) and masks or subtracts those specific dimensions from the query vector for "A."
This mechanism allows for precise filtering without degrading the signal of the primary query. For instance, in a search for "sports but not basketball," the dimensions representing "basketball" are silenced, while dimensions representing "competition," "athletes," or "ball games" remain active for other sports.
Key Results and Qualitative Analysis
The proposed $SR_{clip}$ and $SR_{blip}$ models significantly outperformed both dense baselines and previous sparse models on exclusion tasks.
- Superiority over Dense Models: On the MSCOCO dataset, $SR_{clip}$ achieved an AP@10 of 0.7865, compared to 0.5504 for the standard CLIP approach—a massive gain of 11% to 23% depending on the specific metric used.
- Beating Sparse Baselines: The model outperformed VDR, the previous state-of-the-art for sparse disentangled retrieval, by up to 21% in AP@10.
- Qualitative Success: Visual results demonstrate the model's interpretability. While CLIP often returns basketball images for a "sports but not basketball" query due to the high similarity between the concepts, the SR models successfully filter them out while retaining other sports.
Conclusion
By learning sparse, disentangled representations, this research provides a path toward more controllable and explainable multimodal retrieval. The approach balances the efficiency of fixed-size embeddings with the logical flexibility of sparse models. Future work will focus on extending this logic to more complex constraints, such as multi-label exclusions or conjunctive queries.
Summary updated