Abstract
Learning rich and compact representations is an open topic in many fields such as object recognition or image retrieval. Deep neural networks have made a major breakthrough during the last few years for these tasks but their representations are not necessary as rich as needed nor as compact as expected. To build richer representations, high order statistics have been exploited and have shown excellent performances, but they produce higher dimensional features. While this drawback has been partially addressed with factorization schemes, the original compactness of first order models has never been retrieved, or at the cost of a strong performance decrease. Our method, by jointly integrating codebook strategy to factorization scheme, is able to produce compact representations while keeping the second order performances with few additional parameters. This formulation leads to state-of-the-art results on three image retrieval datasets.
Introduction
In computer vision, "first-order" representations, such as Global Average Pooling, are ubiquitous due to their simplicity and compactness. However, they often fail to capture the complex, non-linear interactions between features necessary for fine-grained recognition. "Second-order" methods, like Bilinear Pooling, address this by computing the outer product of feature vectors, creating a much richer representation.
Despite their power, second-order methods suffer from a "dimensionality explosion." For instance, a 512-dimensional feature vector expands to 262,144 dimensions when squared. While factorization techniques can compress these features, they often lose the benefits of codebook pooling—the ability to group similar features before aggregation. This paper introduces Joint Codebook and Factorization (JCF), a method that bridges this gap by providing the richness of second-order codebooks with the efficiency of first-order models.
Methodology: Joint Codebook and Factorization
The authors propose a two-step optimization to maintain model compactness without sacrificing performance:
- JCF-N (Joint Codebook and Factorization): Instead of computing a massive covariance matrix and subsequently projecting it, JCF-N utilizes a rank-one factorization. The model learns a specific projection matrix for each of the $N$ entries in a codebook. This allows the network to pool "like with like" (e.g., separating "eye" features from "wing" features) without incurring quadratic memory costs.
- JCF-N-R (Sharing Projectors): To further optimize, the authors hypothesize that projection matrices for different codewords are not entirely independent. Instead of $N$ unique projectors, they learn a small set of $R$ "basis" projectors. Each codeword is then represented as a linear combination of these shared bases. This reduces the parameter count by a factor of $R/N$.
As shown in the ablation study, the JCF-N-R approach significantly reduces the parameter footprint compared to standard bilinear pooling (BP) while maintaining superior Recall@1 (R@1) performance.
Experimental Results
The JCF model was evaluated on three standard image retrieval benchmarks: Stanford Online Products (SOP), CUB-200-2011 (birds), and Cars-196. The method consistently outperformed existing state-of-the-art (SOTA) methods across these diverse tasks.
- Stanford Online Products: JCF-32 achieved a 77.4% Recall@1, a significant improvement over the previous SOTA (HTL at 74.8%).
- CUB-200-2011: The model reached 60.1% R@1, surpassing the previous best of 57.1%.
- Efficiency Gains: The shared version, JCF-32-8, utilized approximately 4x fewer parameters than the full JCF-32 while remaining competitive with or exceeding previous SOTA results.
Discussion and Insights
A key insight of this work is that codebooks prevent "destructive averaging." By ensuring second-order statistics are calculated only over semantically similar regions, the model preserves critical signals that global pooling might dilute. Furthermore, the success of the shared projector model (JCF-N-R) suggests significant redundancy in the mathematical "lenses" required for different feature clusters.
However, limitations were observed on the Cars-196 dataset, where the shared projector version (JCF-32-8) saw a performance drop (74.2% vs 82.6% for JCF-32). This suggests that high visual variety—such as the diverse shapes and angles found in automotive data—may require a larger set of unique projections than more uniform categories like birds. Future work may explore non-linear recombinations of shared projectors to better capture these complex relationships.
Summary updated