DeepME: Deep Mixture Experts for Large-scale Image Classification

Ming He, Guangyi Lv, Weidong He, Jianping Fan, Guihua Zeng

Introduction

Standard convolutional neural networks (CNNs) like ResNet or VGG are typically optimized for datasets with around 1,000 classes. Scaling these architectures to 10,000 or more categories introduces significant optimization difficulties and computational complexity. Monolithic networks often fail to capture the varying learning complexities of diverse object types. DeepME addresses these challenges using a "Mixture of Experts" (MoE) framework that semantically clusters categories into overlapping task groups.

Methods

The DeepME framework operates in three distinct stages to manage large-scale classification efficiently.

The DeepME architecture, illustrating the flow from semantic clustering to expert networks and gate-driven fusion.
  1. Soft Spectral Clustering (SSC): Categories are grouped based on semantic correlations derived from WordNet. Unlike hard clustering, SSC allows categories to belong to multiple task groups, facilitating information sharing across experts.
  2. Learning Base Deep Networks: Specialized base networks (VGG16) are trained for each task group. The training process utilizes a loss function that incorporates visual and semantic similarity regularization.
  3. Gate Network Training: A central gate network learns to assign confidence scores to each expert. It ensures consistency across overlapping classes and suppresses irrelevant experts through a specialized hinge loss function.

Datasets and Benchmarking

The model was evaluated on ImageNet10K, a subset of the full ImageNet database containing approximately 9 million images across 10,184 categories. The dataset was split into training (85%), validation (5%), and testing (10%) sets.

Key Results

DeepME achieves state-of-the-art performance on ImageNet10K, significantly outperforming standard monolithic scaling and previous mixture models.

Comparison of classification accuracy across 10,000 classes for DeepME and the previous state-of-the-art DMDE.
Model Top-1 Accuracy Top-5 Accuracy
DeepME (Proposed) 32.13% 59.79%
DMDE (Previous SOTA) 29.51% 56.49%
VGG16-Ext (Standard) 27.62% 53.17%
DeepME-NoGate (Ablation) 7.84% 29.85%

Ablation studies highlight the critical role of the gate network; without it, accuracy drops to 7.84%, demonstrating that adaptive fusion is essential for large-scale tasks.

Conclusion

DeepME demonstrates that semantic grouping and adaptive expert fusion are powerful tools for scaling image classification. While training multiple experts is computationally intensive, the gains in accuracy—especially for "hard" classes—are substantial. Future work will focus on end-to-end integration and more advanced fusion mechanisms like attention.

Summary updated

Abstract

Although deep learning has demonstrated its outstanding performance on image classification, most well-known deep networks make efforts to optimize both their structures and their node weights for recognizing fewer (e.g., no more than 1000) object classes. Therefore, it is attractive to extend or mixture such well-known deep networks to support large-scale image classification. According to our best knowledge, how to adaptively and effectively fuse multiple CNNs for large-scale image classification is still under-explored. On this basis, a deep mixture algorithm is developed to support large-scale image classification in this paper. First, a soft spectral clustering method is developed to construct a two-layer ontology (group layer and category layer) by assigning large numbers of image categories into a set of groups according to their inter-category semantic correlations, where the semantically-related image categories under the neighbouring group nodes may share similar learning complexities. Then, such two-layer ontology is further used to generate the task groups, in which each task group contains partial image categories with similar learning complexities and one particular base deep network is learned. Finally, a gate network is learned to combine all base deep networks with fewer diverse outputs to generate a mixture network with larger outputs. Our experimental results on ImageNet10K have demonstrated that our proposed deep mixture algorithm can achieve very competitive results (top 1 accuracy: 32.13%) on large-scale image classification tasks.
BibTeX
@inproceedings{he2021deepme,
  author = {Ming He and Guangyi Lv and Weidong He and Jianping Fan and Guihua Zeng},
  title = {{DeepME: Deep Mixture Experts for Large-scale Image Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/100},
}
Figure