TL;DR
Image embeddings are widely shared under the assumption that their compressed, lossy nature protects user privacy. This paper introduces SLImE, a lightweight framework demonstrating that exact pixel reconstruction isn't necessary to breach privacy; instead, the semantic neighborhood structure preserved by retrieval-optimized models allows adversaries to extract detailed captions, objects, and scene graphs using off-the-shelf LLMs and VLMs.
What Problem Does This Paper Address?
Multimodal AI ecosystems frequently share fixed-length image embeddings via APIs (such as Pinecone or Weaviate) for downstream tasks like semantic search. Because these embeddings compress high-dimensional images into compact vectors, they are generally assumed to pose minimal privacy risks.
The authors challenge this assumption by investigating semantic leakage—the ability to recover structured semantic information directly from standalone compressed image embeddings. They explore whether an adversary with zero access to a target model's internals can exploit the core property of these embeddings (their optimization for semantic retrieval) to infer private visual content without ever reconstructing the original pixels.
Methods
The authors propose SLImE (Semantic Leakage from Image Embeddings), a lightweight, model-agnostic framework that operates in two main stages without requiring expensive task-specific decoders:
- Embedding Space Alignment: Using a few-shot, training-free linear mapping matrix ($W$) derived via least-squares minimization, victim embeddings from a target space are mapped into the attacker's space.
- Local Retriever Training: The attacker extracts semantic tags (such as relational triples like
<subject, verb, object>) from a public caption dataset using the Stanza NLP toolkit. They then train a lightweight local retriever using a contrastive objective followed by a Deep & Cross Network (DCN v2) ranker to map the aligned image embeddings to these semantic tags. - Generative Amplification: Off-the-shelf LLMs (e.g., DeepSeek-V3.2) turn the retrieved tags into coherent natural language descriptions. For advanced "adaptive attacks," an off-the-shelf diffusion model generates a low-fidelity image from the aligned embedding, which is then fed alongside retrieved tags into Vision-Language Models (VLMs) to progressively extract objects, relations, and full scene graphs.
Datasets
- COCO: The standard Microsoft COCO dataset was used for training the local retriever and evaluating text reconstruction performance (subsampling 500 images each for validation and testing).
- nocaps: A subset of 500 samples (250 near-domain, 250 out-domain) was utilized to evaluate how well the semantic leakage generalizes to unseen, out-of-distribution image domains.
Key Insights and Ideas
- Leakage Without Reconstruction: Traditional privacy attacks focus on pixel-level image reconstruction. SLImE proves that privacy can be completely undermined at the semantic level without ever recovering the original pixels accurately.
- The Retrieval Paradox: The very feature that makes image embeddings useful—preserving local semantic similarity so similar concepts cluster together—is what makes them vulnerable. Alignment preserves these "local semantic neighborhoods," allowing information to flow through multiple lossy transformations.
- Semantic Amplification: Even if tag retrieval is noisy or incomplete, modern generative models (LLMs/VLMs) act as powerful semantic priors that can "decompress" and amplify these partial cues into highly accurate, structured descriptions.
Key Results
- Neighborhood Preservation: While exact tag matching remains low (under 0.25 F1), neighborhood preservation F1 scores reach up to 0.8, showing that aligned embeddings reliably land in the correct semantic ballpark.
- Text Reconstruction: Using 10k alignment samples, open-source LLMs like DeepSeek-V3.2 achieved a ROUGE-L score of 54.36, closely rivaling proprietary models like Gemini-3-Pro (60.53) in reconstructing human-like descriptions from embeddings alone.
- Adaptive VLM Attacks: Low-fidelity images generated from embeddings alone recovered nearly half of the objects in the original scene. Combining these images with retrieved tags boosted object and relation identification significantly across various VLMs.
- Cross-Domain Robustness: The attack successfully generalized to the out-of-domain nocaps dataset, showing only a moderate performance drop (e.g., CLIP ROUGE-L dropping from 38.27 to 34.29).
Limitations and Future Work
- Domain Shift Sensitivity: There is a noticeable performance degradation when moving to completely out-of-domain datasets (nocaps out-domain), indicating that the alignment matrix $W$ is somewhat sensitive to the alignment data distribution.
- Dependence on Retrieval Optimization: The attack exploits embeddings explicitly optimized for semantic retrieval; it may be less effective against self-supervised representations trained on purely pixel-level pretext tasks.
- Defense and Mitigation: The paper focuses entirely on exposing the vulnerability rather than fixing it. The authors note that future work must explore representation learning techniques that can preserve retrieval utility while actively defending against semantic inference attacks.
Summary updated