TL;DR
This paper introduces Sparse Coding based Fisher Vector Coding (SCFVC), a novel method for turning high-dimensional local features (like CNN activations) into compact, discriminative image representations. By replacing the traditional Gaussian Mixture Model (GMM) with a sparse coding generative model, the authors achieve state-of-the-art results in object, scene, and fine-grained image classification while overcoming the computational bottlenecks of high-dimensional data.
What Problem Does This Paper Address?
In computer vision, "Fisher Vector Coding" (FVC) has long been a gold standard for image classification. Traditionally, it works by fitting a Gaussian Mixture Model (GMM) to local features (like SIFT) and then describing how an image's features deviate from that model.
However, we have entered the era of Deep Learning. Instead of 128-dimensional SIFT descriptors, we now use 4096-dimensional features from Convolutional Neural Networks (CNNs). This creates a "curse of dimensionality" for GMMs:
- Volume Explosion: High-dimensional space is vast. To "cover" it effectively with a GMM, you would need an impractically large number of Gaussians.
- Dimensionality Bloat: The resulting Fisher Vector's size is proportional to the number of Gaussians times the feature dimension. With high-dimensional features, the final image representation becomes too massive to store or process efficiently.
The authors tackle this by asking: How can we model the distribution of high-dimensional features without needing thousands of Gaussians?
Methods
The core innovation is a shift in the underlying generative model.
From Points to Subspaces
Instead of assuming local features are drawn from a fixed set of $m$ Gaussian "points" (the GMM approach), the authors propose that each feature is drawn from a Gaussian whose mean vector is sampled from a subspace. This subspace is spanned by a set of bases (a dictionary).
The Sparse Coding Link
The authors demonstrate that under certain approximations, calculating the likelihood of a feature in this new model is mathematically equivalent to solving a Sparse Coding problem.
- Generative Model: $P(\mathbf{x}|\mathbf{u}, \mathbf{B}) = \mathcal{N}(\mathbf{B}\mathbf{u}, \sigma^2 \mathbf{I})$
- Inference: Finding the best latent variable $\mathbf{u}$ (the code) becomes the standard Lasso-style optimization: $\min_{\mathbf{u}} \|\mathbf{x} - \mathbf{B}\mathbf{u}\|_2^2 + \lambda \|\mathbf{u}\|_1$.
The SCFVC Descriptor
By taking the gradient of this log-likelihood with respect to the dictionary $\mathbf{B}$, they derive the new Fisher Vector. Interestingly, the descriptor for a single local feature turns out to be the outer product of the reconstruction residual $(\mathbf{x} - \mathbf{B}\mathbf{u}^*)$ and the sparse code $\mathbf{u}^*$. These are then sum-pooled and normalized to create the final image-level representation.
Datasets
The authors evaluated their method on three diverse benchmarks to prove its versatility:
- Pascal VOC 2007: The classic benchmark for generic object recognition (20 classes).
- MIT Indoor Scene-67: A challenging dataset for scene classification where global context matters.
- Caltech-UCSD Birds-200-2011: A fine-grained classification task requiring the model to distinguish between very similar bird species.
Key Insights and Ideas
- Efficiency in High Dimensions: As dimensionality increases, the average distance between a feature and its nearest GMM mean grows rapidly. However, the sparse coding model maintains a much higher "partition resolution" because a subspace can represent a high-dimensional volume much more efficiently than discrete points.
- The Power of the Residual: Traditional sparse coding often just uses the code $\mathbf{u}$. By using the Fisher Vector framework, this method also captures the reconstruction error (the residual), which contains rich information about how a specific feature differs from the general dictionary.
- CNNs as Local Features: Instead of using the CNN as a global classifier, they treat the activations of the 6th layer (FC6) of a pre-trained AlexNet as "local" descriptors by extracting them from overlapping patches. This combines the representational power of deep learning with the robust aggregation of Fisher Vectors.
Key Results
The SCFVC consistently outperformed both traditional GMM-based Fisher Vectors and standard global CNN features:
- Pascal VOC 2007: Achieved 76.9% mAP, outperforming GMM-FVC (73.8%) and the standard CNN-SVM approach (73.9%).
- MIT Indoor-67: Reached 68.2% accuracy, a significant jump over GMM-FVC (64.3%).
- Birds-200: Achieved 66.4% accuracy, beating GMM-FVC by nearly 5%.
- Scaling: The performance gap between SCFVC and GMM-FVC widened as the dimensionality of the local features increased, validating the authors' hypothesis that sparse coding is superior for high-dimensional modeling.
Limitations and Future Work
- Computational Complexity: Solving a sparse coding optimization for every local patch is computationally heavier than the simple membership assignment used in GMMs. While the authors suggest using fast solvers like FISTA, this remains a trade-off.
- Dictionary Learning: The dictionary is currently learned separately from the classifier. A potential future direction is "end-to-end" learning where the dictionary and the SVM (or a final neural layer) are optimized simultaneously.
- Spatial Information: While the method aggregates features well, it doesn't explicitly encode the spatial arrangement of those features beyond standard pooling techniques. Integrating spatial pyramids or attention mechanisms could further boost performance.
Summary updated