Introduction
Vision encoders serve as the backbone of modern AI, yet they are often treated as "black boxes." While their utility in downstream tasks is undeniable, the specific information they prioritize and the internal organization of their feature spaces remain poorly understood. This paper explores a novel diagnostic approach: using image reconstruction to "peek under the hood" of these encoders. By training a decoder to reconstruct original images from hidden feature representations, we can evaluate what visual data is preserved and how it is geometrically structured.
Methodology: The Reconstructor Pipeline
The researchers propose a Reconstructor Pipeline consisting of a frozen vision encoder (e.g., CLIP or SigLIP) and a small, trainable reconstructor module. Trained primarily on the COCO (Common Objects in Context) dataset, this module—composed of four transformer blocks followed by upsampling layers—is optimized to invert the encoder's features back into the original pixel space using an L2 loss.
This framework allows for the empirical testing of the Feature Transformation Hypothesis, which posits that mathematical operations within the latent feature space correspond to logical, predictable changes in the reconstructed image.
The Geometry of Feature Space
A significant finding of this research is the linear and geometric organization of the feature space, particularly regarding color. By solving the Orthogonal Procrustes problem, the authors identified a rotation matrix ($Q$) that maps the features of an original image to those of a color-modified version (e.g., swapping red and blue channels).
When this rotation is applied to the features of new, unseen images, the resulting reconstructions exhibit the expected color shifts. This demonstrates that color information is not randomly distributed but is encoded such that orthogonal rotations in high-dimensional space correspond to physical color permutations. Additionally, the model's ability to "colorize" grayscale images suggests that encoders capture inherent semantic knowledge about the world, such as the typical colors of sky or grass.
Comparative Analysis: SigLIP vs. SigLIP2
The study highlights how training objectives dictate memory. Purely contrastive models like SigLIP tend to discard "irrelevant" pixel data to optimize for image-text matching. Conversely, SigLIP2—which utilizes multitask learning including captioning, self-distillation, and masked prediction—retains significantly higher-fidelity structural and color information. Quantitative tests using CLIP-Score and SigLIP2-Score confirmed that SigLIP2 reconstructions are statistically superior ($p < 0.01$) across various resolutions, preserving details like fur texture that are lost in standard CLIP models.
Conclusion
By transforming the "black box" of vision encoders into a "projector," this research provides a practical framework for feature interpretability. The findings underscore the importance of multitask pre-training for information preservation and reveal a surprisingly structured geometry within latent spaces. While vision encoders remain inherently lossy, this diagnostic tool allows developers to choose encoders based on the visual detail required for specific applications.
Summary updated