A General Concept-based Decomposition for Vision-Language Embeddings

Simone Alberto Peirone, Ortal Senouf, Francesca Pistilli, Giuseppe Averta, Pascal FrossardarXiv 2026 · arXiv

TL;DR

GRACE is a novel framework that makes Vision-Language Models (VLMs) like CLIP interpretable by decomposing their complex embeddings into human-readable concepts. By prioritizing the preservation of global geometric relationships between samples over simple local reconstruction, GRACE achieves superior alignment with human perception and maintains high performance across diverse image and video tasks.

What Problem Does This Paper Address?

Vision-Language Models are remarkably effective at aligning images and text, but they remain "black boxes." We don't truly know if a model's internal representation of a "dog playing in a park" actually relies on the concepts of "dog," "playing," and "park" in the same way a human would.

Existing methods for concept-based decomposition often suffer from two main issues:

  1. Instance-level Overfitting: They optimize for each image individually, which can lead to "shortcut learning"—mathematical tricks that reconstruct the embedding without capturing true semantic meaning.
  2. The Modality Gap: Visual and textual features live in different geometric spaces. Forcing a strict reconstruction of visual features using only text concepts often fails because these spaces aren't perfectly interchangeable.

Methods

The authors introduce GRACE (Geometry-based Reconstruction and Concept-based Embedding), which learns a global, task-agnostic concept structure.

Figure 1: The GRACE framework. Given a visual sample, either an image or video, and a vocabulary C of human-interpretable concepts, a geometry-based reconstruction pipeline decomposes the visual embedding into semantically relevant concept contributions. These reconstructed embeddings can replace the original ones, improving interpretability and enabling model debugging.

Key technical innovations include:

  • Geometry-Preserving Reconstruction: Instead of standard $\ell_2$ reconstruction, GRACE uses a dual objective. Retrieval Consistency ensures the reconstructed embedding remains the closest match to the original, while Isometry ensures that the pairwise similarities between different samples are preserved in the concept space.
  • Calibrated Visual Evidence: The model matches visual embeddings against a large concept vocabulary and uses a gating mechanism (an MLP denoted as $H$) to filter out irrelevant or redundant concepts.
  • Orthogonal Transformation ($Q$): A learnable matrix maps text concepts into a subspace compatible with visual embeddings. Being orthogonal, it preserves the internal relationships (dot products) between concepts.
  • Sparsity and Consistency Regularization: A Dirichlet prior forces the model to use only a few significant concepts per sample (sparsity), while a Jensen-Shannon Divergence loss ensures that semantically similar inputs use similar concepts (consistency).
Figure 2: Geometry-based reconstruction and regularization objectives in GRACE. Retrieval consistency ensures that the reconstructed visual embeddings are close to their original counterparts, Isometry preserves the same pairwise similarities between the samples in the batch before and after the reconstruction, while Sparsity and Consistency regularization favor sparse reconstruction weights and consistency across different sample categories.

Datasets

  • Training: The model is trained on large-scale datasets: EgoClip (3.8M video-text pairs) for video tasks and ImageNet for image tasks.
  • Evaluation:
    • Video: EgoMCQ (retrieval), Charades-Ego (action recognition), and ConVisBench (a benchmark for semantic similarity alignment with human judgment).
  • Image: COCO, Flickr30k, Flowers102, and Food101.
  • Concept Vocabularies: Concepts are mined from captions or generated using LLMs (like GPT-4) to ensure a broad, human-understandable set of terms.

Key Insights and Ideas

  • Global Structure > Local Accuracy: The "aha moment" is that preserving the relative positions of samples in the feature space (geometry) is more important for true interpretability than perfectly reconstructing the coordinates of a single sample.
  • VLMs are Naturally Linear: The paper reinforces the "linear representation hypothesis," showing that high-level concepts are already represented as linear directions in VLM spaces; GRACE simply provides the right "lens" to see them.
  • Interpretability as a Regularizer: Forcing the model to reason through a sparse set of human concepts acts as a form of regularization that actually helps the model generalize better to new, unseen domains.

Key Results

  • Human Alignment: On the ConVisBench benchmark, GRACE improved alignment with human perception by +10.6% compared to previous state-of-the-art methods like SpLiCE.
  • Faithfulness: The reconstructed embeddings are highly "faithful," meaning they preserve the original model's behavior. On EgoMCQ, GRACE produced +7.50% more faithful reconstructions than prior work.
  • Robustness to Shift: The authors demonstrated that a GRACE model trained on one set of ImageNet classes could still produce accurate concept decompositions for entirely different, unseen classes.
  • Qualitative Superiority: Unlike earlier methods that often "collapsed" onto a single concept, GRACE distributes weights across all relevant parts of a scene (e.g., identifying "fishing," "ocean," and "boat" simultaneously).

Limitations and Future Work

  • Subjectivity of Concepts: The definition of a "concept" remains somewhat subjective and is limited by the quality and coverage of the initial vocabulary.
  • Temporal Modeling: For video, GRACE currently produces clip-level summaries. The authors suggest that modeling how concepts evolve second-by-second (temporal grounding) is a key next step.
  • Causal Links: While the model aligns well with human judgment, the authors note they haven't yet established a strict causal link—proving that the model made a specific decision because of a specific concept activation.

Summary updated

Abstract

Vision-Language Models are designed to capture the compositional structure of the world through multi-modal alignment; however, it remains unclear whether their representations truly align with the way a human would naturally describe the same visual input. Prior works attempt to decompose visual embeddings into their concept-level contributions, but their instance-level optimization is inherently local and does not capture the model's global semantic structure, which is essential for compositional generalization across different inputs, tasks, and domains. We introduce GRACE, a method that learns this concept-based structure at the feature space level, producing semantic and general concept decompositions. To achieve this, we propose a novel training objective that preserves the geometric relations of different samples in the concept space, enabling a better understanding of how concepts compose and relate to each other. We validate GRACE on several image and video benchmarks, showing that its concept decomposition is aligned with human perception and achieves faithful decompositions across different tasks and domains. With minimal overhead, GRACE makes existing Vision-Language Models more interpretable, while preserving most of their original performance.
BibTeX
@misc{peirone2026general,
  author = {Simone Alberto Peirone and Ortal Senouf and Francesca Pistilli and Giuseppe Averta and Pascal Frossard},
  title = {{A General Concept-based Decomposition for Vision-Language Embeddings}},
  year = {2026},
  eprint = {2611.15195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15195},
}
Figure