Deep Mixture of Diverse Experts for Large-Scale Visual Recognition

Tianyi Zhao, Qiuyu Chen, Zhenzhong Kuang, Jun Yu, Wei Zhang, Jianping Fan

Abstract

This paper presents a Deep Mixture of Diverse Experts (DMDE) algorithm designed to scale visual recognition to tens of thousands of classes. By organizing object classes into a two-layer semantic ontology, the framework generates specialized task groups. Each group is handled by a base deep CNN expert trained to recognize specific atomic classes alongside a "not-in-group" category. Through multi-task learning and a novel stacking function for fusion, DMDE effectively leverages inter-class similarities and manages the computational complexities of large-scale datasets like ImageNet10K.

Introduction

Training neural networks for standard benchmarks like ImageNet-1K is a well-understood task. However, scaling recognition to 10,000 or more classes introduces significant hurdles. A monolithic "flat" classifier with a massive softmax layer often suffers from high computational costs, overfitting, and the "confused gradient" problem, where the model struggles to distinguish between highly diverse categories.

The DMDE framework adopts a "divide and conquer" strategy. Instead of one giant network, it utilizes a mixture of specialized experts, each focusing on a semantically related subset of the label space. This approach allows for the use of proven architectures, like AlexNet, as modular building blocks for a much larger system.

Methodology

1. Semantic Ontology and Task Assignment

The label space is organized using a two-layer ontology (category layer and object class layer) derived from WordNet. This ensures that semantically related classes—which often share visual features and learning complexities—are grouped together. Unlike traditional clustering, DMDE employs tree-guided task assignment with significant inter-group overlap. This overlap is crucial for calibrating the experts' confidence scores against one another during the final fusion stage.

2. Specialized Expert Training

Each expert is a CNN trained using a deep multi-task learning (MTL) algorithm. Two key features define the expert training:

  • The "Not-in-group" Class: Every expert is trained to recognize its specific classes plus an additional category for objects outside its domain. This acts as a built-in confidence mechanism.
  • Multi-Task Softmax: Laplacian regularization is used to ensure that visually similar classes within a group share internal feature representations, enhancing discriminative power.

3. Stacking and Fusion

To produce a final classification, the outputs from all experts are combined via a stacking function. This function uses the "not-in-group" scores to weigh the contributions of each expert; if an expert identifies an image as "not-in-group" with high confidence, its specific class predictions are suppressed in the final output.

The DMDE architecture: specialized task groups are processed by individual CNN experts and combined via a stacking function to produce final object class predictions.

Experimental Results

The DMDE framework was evaluated on the ImageNet10K dataset, specifically targeting 7,756 atomic object classes.

Performance Benchmarks

DMDE demonstrated a clear advantage over standard deep learning baselines. On ImageNet10K, it achieved a 38.65% top-1 accuracy, significantly outperforming a standard AlexNet extension (31.70%). The performance gap was even more pronounced in top-K metrics, with DMDE reaching 64.32% top-10 accuracy compared to 52.18% for the baseline.

Component Analysis

Ablation studies highlighted the importance of the multi-task learning and the "not-in-group" safety valve. Removing these components led to a marked decrease in accuracy across the board, confirming that the synergy between semantic grouping and specialized training is vital for large-scale success.

Ablation study results comparing accuracy across different model configurations, demonstrating the performance gains from multi-task learning (MT) and overlapping task groups.

Discussion and Conclusion

The Deep Mixture of Diverse Experts offers a robust framework for large-scale visual recognition by leveraging semantic hierarchies to simplify the learning task. While the current implementation relies on AlexNet, the modular nature of DMDE allows for the integration of more advanced base networks. Future research may focus on optimizing the inference speed of multiple parallel experts and exploring visual-similarity-based ontologies to further refine task grouping.

Summary updated

Abstract

In this paper, a deep mixture of diverse experts algorithm is developed to achieve more efficient learning of a huge (mixture) network for large-scale visual recognition application. First, a two-layer ontology is constructed to assign large numbers of atomic object classes into a set of task groups according to the similarities of their learning complexities, where certain degrees of inter-group task overlapping are allowed to enable sufficient inter-group message passing. Second, one particular base deep CNNs with <inline-formula><tex-math notation="LaTeX">$M+1$</tex-math></inline-formula> outputs is learned for each task group to recognize its <inline-formula><tex-math notation="LaTeX">$M$</tex-math></inline-formula> atomic object classes and identify one special class of “not-in-group”, where the network structure (numbers of layers and units in each layer) of the well-designed deep CNNs (such as AlexNet, VGG, GoogleNet, ResNet) is directly used to configure such base deep CNNs. For enhancing the separability of the atomic object classes in the same task group, two approaches are developed to learn more discriminative base deep CNNs: (a) our deep multi-task learning algorithm that can effectively exploit the inter-class visual similarities; (b) our two-layer network cascade approach that can improve the accuracy rates for the hard object classes at certain degrees while effectively maintaining the high accuracy rates for the easy ones. Finally, all these complementary base deep CNNs with diverse but overlapped outputs are seamlessly combined to generate a mixture network with larger outputs for recognizing tens of thousands of atomic object classes. Our experimental results have demonstrated that our deep mixture of diverse experts algorithm can achieve very competitive results on large-scale visual recognition.
BibTeX
@article{zhao2017deep,
  author = {Tianyi Zhao and Qiuyu Chen and Zhenzhong Kuang and Jun Yu and Wei Zhang and Jianping Fan},
  title = {{Deep Mixture of Diverse Experts for Large-Scale Visual Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2017},
  url = {https://openreview.net/forum?id=2eiK9jyJeM0},
}
Figure