TL;DR
Modern visual embedding models inherently organize continuous, ordinal attributes (like age, crowd count, or image aesthetics) along linear directions called "rank axes." Remarkably, these global ranking directions can be recovered with minimal supervision—sometimes using just two extreme example images—enabling efficient image sorting in vector databases without full-scale model training.
What Problem Does This Paper Address?
When we use vector databases for image retrieval, we rely on the embedding space's local similarity structure: similar images are clustered close to a query image. However, databases also need to support ranking—sorting a collection globally based on continuous attributes (e.g., sorting faces from youngest to oldest, or products by visual quality).
While prior work has heavily explored local similarity, the global rankability of embedding spaces has remained largely underexplored. This paper investigates whether modern vision encoders inherently map continuous, ordinal concepts along straight lines, and how easily we can discover these "rank axes" to sort images globally.
Methods
The authors formally define rankability: an embedding function is rankable for an attribute if there exists a linear direction (a unit vector $v_A$) such that projecting embeddings onto this axis preserves the correct attribute order.
To evaluate this, they use Spearman’s rank correlation coefficient (SRCC, $\rho$) to measure how well the projected order matches the ground truth. They establish three reference baselines:
- No-train (Lower Bound): The performance of a randomly initialized version of the encoder.
- Nonlinear (Embedding Upper Bound): A two-layer Multilayer Perceptron (MLP) trained on the frozen embeddings to see how much total ordinal information can be extracted nonlinearly.
- Finetuning (Architecture Upper Bound): End-to-end training of the encoder.
To find the rank axis efficiently, they experiment with standard linear regression, few-shot learning, and a highly practical "extreme pairs" approach. In the extreme pairs setup, the axis is calculated simply by subtracting the average embedding of a few "low" attribute images from a few "high" attribute images: $$v_A = \frac{f(x_h) - f(x_l)}{\|f(x_h) - f(x_l)\|_2}$$
Datasets
The authors evaluate 7 modern vision encoders (including ResNet50, ViT, CLIP, and DINOv2) across 9 datasets covering 7 distinct continuous attributes:
- Age: UTKFace (exact ages 21–60) and Adience (8 age groups).
- Crowd Count: UCF-QNRF, ShanghaiTech-A, and ShanghaiTech-B.
- Head Pose (Pitch, Yaw, Roll): BIWI Kinect dataset.
- Aesthetics: AVA and KonIQ-10k (human-annotated quality ratings).
- Image Modernness/Recency: Historical Color Images (HCI), which categorizes photos by the decade they were taken.
Key Insights and Ideas
- Ordinal Information is Strikingly Linear: There is an incredibly small performance gap between a simple linear rank axis and a complex, nonlinear MLP regressor. This means that if an embedding model knows about an ordinal attribute, it stores that information linearly.
- CLIP Excels at Semantics, DINOv2 at Geometry: CLIP-based models generally outperform image-only models on semantic and abstract attributes like age, aesthetics, and recency. However, for purely geometric attributes like head yaw and roll, self-supervised models like DINOv2 massively outperform CLIP.
- Two Extreme Examples Can Be Enough: In data-scarce regimes, the "extreme pairs" method outperforms standard few-shot linear regression. For instance, using just one image of a child and one of an elderly person can yield a highly effective age-sorting axis ($\sim 0.75$ SRCC on the Adience dataset), bypassing the need for expensive continuous annotations.
Key Results
- High Global Rankability: Across most attributes, the learned linear axes achieve high SRCC scores. For example, on the Adience age dataset, the average linear rankability across models is $0.861$, which is remarkably close to the nonlinear upper bound of $0.878$.
- Transferability: Rank axes learned on one dataset transfer surprisingly well to entirely different datasets containing the same attribute. An age axis trained on Adience achieves an SRCC of $0.680$ when evaluated on UTKFace, proving the existence of a "universal" attribute direction in the embedding space.
- Zero-Shot Prompting Lags Behind: While text prompts from Vision-Language Models (like CLIP) can define zero-shot rank axes, they perform significantly worse than image-based linear regression (e.g., an SRCC of $0.449$ vs $0.790$ for image recency).
Limitations and Future Work
The current study is entirely empirical and limited to the selected 7 attributes and 9 datasets. Furthermore, certain geometric attributes remain challenging; most encoders struggled to linearly rank head yaw and roll angles effectively compared to pitch.
For future work, the authors suggest developing theoretical frameworks to explain why these spaces are inherently rankable. They also propose exploring whether embedding spaces can be entirely characterized as interpretable collections of latent ordinal subspaces.
Summary updated