Image Reconstruction as a Tool for Feature Analysis

Eduard Allakhverdov, Dmitrii Tarasov, Elizaveta Goncharova, Andrey KuznetsovarXiv 2025 · DOI · arXiv

Introduction

Vision encoders serve as the backbone of modern AI, yet they are often treated as "black boxes." While their utility in downstream tasks is undeniable, the specific information they prioritize and the internal organization of their feature spaces remain poorly understood. This paper explores a novel diagnostic approach: using image reconstruction to "peek under the hood" of these encoders. By training a decoder to reconstruct original images from hidden feature representations, we can evaluate what visual data is preserved and how it is geometrically structured.

Methodology: The Reconstructor Pipeline

The researchers propose a Reconstructor Pipeline consisting of a frozen vision encoder (e.g., CLIP or SigLIP) and a small, trainable reconstructor module. Trained primarily on the COCO (Common Objects in Context) dataset, this module—composed of four transformer blocks followed by upsampling layers—is optimized to invert the encoder's features back into the original pixel space using an L2 loss.

The Reconstructor Pipeline architecture.

This framework allows for the empirical testing of the Feature Transformation Hypothesis, which posits that mathematical operations within the latent feature space correspond to logical, predictable changes in the reconstructed image.

The Geometry of Feature Space

A significant finding of this research is the linear and geometric organization of the feature space, particularly regarding color. By solving the Orthogonal Procrustes problem, the authors identified a rotation matrix ($Q$) that maps the features of an original image to those of a color-modified version (e.g., swapping red and blue channels).

Calculation and application of the Q matrix for feature space transformation.

When this rotation is applied to the features of new, unseen images, the resulting reconstructions exhibit the expected color shifts. This demonstrates that color information is not randomly distributed but is encoded such that orthogonal rotations in high-dimensional space correspond to physical color permutations. Additionally, the model's ability to "colorize" grayscale images suggests that encoders capture inherent semantic knowledge about the world, such as the typical colors of sky or grass.

Comparative Analysis: SigLIP vs. SigLIP2

The study highlights how training objectives dictate memory. Purely contrastive models like SigLIP tend to discard "irrelevant" pixel data to optimize for image-text matching. Conversely, SigLIP2—which utilizes multitask learning including captioning, self-distillation, and masked prediction—retains significantly higher-fidelity structural and color information. Quantitative tests using CLIP-Score and SigLIP2-Score confirmed that SigLIP2 reconstructions are statistically superior ($p < 0.01$) across various resolutions, preserving details like fur texture that are lost in standard CLIP models.

Conclusion

By transforming the "black box" of vision encoders into a "projector," this research provides a practical framework for feature interpretability. The findings underscore the importance of multitask pre-training for information preservation and reveal a surprisingly structured geometry within latent spaces. While vision encoders remain inherently lossy, this diagnostic tool allows developers to choose encoders based on the visual detail required for specific applications.

Summary updated

Abstract

Vision encoders are increasingly used in modern applications, from vision-only models to multimodal systems such as vision-language models. Despite their remarkable success, it remains unclear how these architectures represent features internally. Here, we propose a novel approach for interpreting vision features via image reconstruction. We compare two related model families, SigLIP and SigLIP2, which differ only in their training objective, and show that encoders pre-trained on image-based tasks retain significantly more image information than those trained on non-image tasks such as contrastive learning. We further apply our method to a range of vision encoders, ranking them by the informativeness of their feature representations. Finally, we demonstrate that manipulating the feature space yields predictable changes in reconstructed images, revealing that orthogonal rotations (rather than spatial transformations) control color encoding. Our approach can be applied to any vision encoder, shedding light on the inner structure of its feature space. The code and model weights to reproduce the experiments are available in GitHub.
BibTeX
@misc{allakhverdov2025image,
  author = {Eduard Allakhverdov and Dmitrii Tarasov and Elizaveta Goncharova and Andrey Kuznetsov},
  title = {{Image Reconstruction as a Tool for Feature Analysis}},
  year = {2025},
  eprint = {2506.07803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07803},
}
Figure