Abstract
We investigate the scalable image classification problem with a large number of categories. Hierarchical visual data structures are helpful for improving the efficiency and performance of large-scale multi-class classification. We propose a novel image classification method based on learning hierarchical inter-class structures. Specifically, we first design a fast algorithm to compute the similarity metric between categories, based on which a visual tree is constructed by hierarchical spectral clustering. Using the learned visual tree, a test sample label is efficiently predicted by searching for the best path over the entire tree. The proposed method is extensively evaluated on the ILSVRC2010 and Caltech 256 benchmark datasets. Experimental results show that our method obtains significantly better category hierarchies than other state-of-the-art visual tree-based methods and, therefore, much more accurate classification.
Introduction
As image datasets expand to thousands of categories, traditional "flat" classification models—such as One-vs-All classifiers—encounter significant computational bottlenecks during inference. Hierarchical classification, which organizes classes into a tree structure to achieve logarithmic search time, offers a solution. However, traditional hierarchical methods face three main challenges: the semantic gap (where human-defined taxonomies do not align with visual similarity), high construction costs for visual trees, and error propagation in greedy search algorithms.
Proposed Framework
This paper introduces a fully visual, taxonomy-independent hierarchical learning framework designed to overcome these hurdles. The pipeline consists of three core components:
- Fast Inter-Class Similarity Metric: To avoid the $O(N^2)$ complexity of pairwise image comparisons, the authors derive a metric using class means and variances, reducing complexity to $O(N)$.
- Visual Tree Construction: Hierarchical spectral clustering is applied to the affinity matrix generated by the fast similarity metric. This groups visually similar categories without the need to train thousands of intermediate classifiers.
- N-best Path Algorithm: To mitigate error propagation, inference is treated as a path-searching problem. Instead of a greedy single-branch search, the algorithm maintains the top $N$ candidate paths with the highest joint probabilities at each layer, allowing recovery from early misclassifications.
Experimental Evaluation and Results
The framework was evaluated on the ILSVRC2010 (1.2 million images, 1,000 classes) and Caltech 256 datasets using both hand-crafted (SIFT, VLAD) and deep learning features (AlexNet, Inception V3).
Key Findings
- Superiority of Visual Hierarchies: Trees built on visual feature clustering significantly outperform those based on semantic taxonomies like WordNet. Visual grouping simplifies the task for classifiers at each node.
- Impact of N-best Path Search: The N-best path algorithm improved top-1 accuracy by 3.2% to 7.1% compared to standard greedy search on the same tree structures. This demonstrates the critical role of path search in hierarchical models.
- Performance Benchmarks: On ILSVRC2010, the method achieved 76.2% top-1 accuracy using Inception V3 features and N-best path search. This outperformed prior state-of-the-art hierarchical methods by a wide margin and even surpassed a flat One-vs-All SVM baseline by 1.5%.
- Efficiency Gains: The hierarchical approach required approximately half the CPU memory of a flat CNN approach and offered faster inference by only evaluating classifiers along active search paths.
Conclusion
The proposed method demonstrates that learning hierarchical structures based on visual similarity, combined with robust path-searching algorithms, can significantly enhance both the accuracy and efficiency of large-scale image classification. Future work may involve end-to-end training of feature representations and tree construction to further optimize performance.
Summary updated