Abstract
In this paper, a level-wise mixture model (LMM) is developed by embedding visual hierarchy with deep networks to support large-scale visual recognition (i.e., recognizing thousands or even tens of thousands of object classes), and a Bayesian approach is used to adapt a pre-trained visual hierarchy automatically to the improvements of deep features (that are used for image and object class representation) when more representative deep networks are learned along the time. Our LMM model can provide an end-to-end approach for jointly learning: (a) the deep networks to extract more discriminative deep features for image and object class representation; (b) the tree classifier for recognizing large numbers of object classes hierarchically; and (c) the visual hierarchy adaptation for achieving more accurate indexing of large numbers of object classes hierarchically. By supporting joint learning of the tree classifier, the deep networks and the visual hierarchy adaptation, our LMM algorithm can provide an effective approach for controlling inter-level error propagation effectively, thus it can achieve better accuracy rates on large-scale visual recognition. Our experiments are carried on ImageNet1K and ImageNet10K image sets, and our LMM algorithm can achieve very competitive results on both the accuracy rates and the computation efficiency as compared with the baseline methods.
Introduction
Large-scale visual recognition, involving thousands of object categories, presents significant challenges for traditional "flat" classification approaches. Standard deep learning models typically employ an $N$-way softmax classifier, which treats all classes as equally distinct. This approach ignores natural visual correlations—for instance, failing to recognize that a "Husky" is more similar to a "Wolf" than to a "Toaster." Furthermore, it leads to gradient imbalances during training, as the model may struggle to learn subtle features necessary for fine-grained discrimination amidst the noise of broad category differences.
While hierarchical "divide-and-conquer" structures have been proposed to address these issues, they often rely on static trees. If the initial hierarchy is constructed using suboptimal features, it can lead to "inter-level error propagation," where early classification mistakes cannot be recovered.
Methodology: The Level-wise Mixture Model (LMM)
To overcome these limitations, this paper introduces the Level-wise Mixture Model (LMM), an end-to-end framework that embeds visual hierarchies directly into deep neural networks.
The LMM architecture replaces the single terminal softmax with a tree of classifiers. Each node in the tree represents a level of granularity, guiding the network to learn "common" features in early layers and "specific" features in deeper layers.
A key innovation is the Bayesian Hierarchy Adaptation. Recognizing that a static hierarchy is a bottleneck, the authors use a Bayesian Dirichlet-categorical model to estimate an object-group assignment matrix ($\Psi$). This allows the model to automatically re-assign object classes to more appropriate groups as the deep network learns more representative features. The entire system—feature extractors, node-specific classifiers, and the hierarchical structure itself—is trained jointly through a path-based loss function.
Experimental Results
The LMM was evaluated on the ImageNet1K (1,000 classes) and ImageNet10K (over 10,000 classes) benchmarks.
On ImageNet1K, the LMM with adaptation (LMM w/P, TC) achieved a Top-1 error rate of 44.29%, a notable improvement over the baseline flat softmax error of 45.13%. Approximately 56% of object classes showed improved accuracy, particularly those within visually similar clusters. On the larger ImageNet10K dataset, the model achieved a Top-10 prediction error of 38.07%, compared to the baseline's 39.20%. Qualitative analysis indicates that the LMM produces more "reasonable" errors, such as confusing similar breeds, rather than semantically unrelated categories.
Conclusion
The LMM framework demonstrates that embedding visual hierarchy into deep networks can effectively scale visual recognition. By allowing the hierarchy to evolve alongside feature learning, the model mitigates inter-level error propagation and improves fine-grained discrimination. Future work will focus on optimizing training for even larger datasets and exploring the impact of deeper hierarchical structures.
Summary updated