Modular Graph Transformer Networks for Multi-Label Image Classification

Hoang D. Nguyen, Xuan-Son Vu, Duc-Trong LeAAAI 2021 · DOI

TL;DR

This paper introduces Modular Graph Transformer Networks (MGTN) to improve multi-label image classification by explicitly modeling object dependencies. MGTN decomposes the complex label graph into smaller, more manageable sub-graphs, allowing for more focused learning and better representation of inter-label relationships. This modular approach leads to state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Real-world images often contain multiple objects, leading to the task of multi-label image classification. A key challenge in this area is understanding and modeling the complex dependencies between different object labels. Existing methods often struggle with this, as graph representations can become indistinguishable due to the sheer number of potential relationships. This can lead to suboptimal learning, especially for less common object labels. The paper aims to address this by proposing a more structured and efficient way to learn these inter-label dependencies.

Methods

The core innovation of this paper is the Modular Graph Transformer Network (MGTN). MGTN tackles the problem by:

  • Graph Modularity: It first identifies "communities" or sub-graphs within the overall object label graph. This is achieved using a hierarchical agglomeration algorithm based on modularity, effectively partitioning labels that are more interconnected.
  • Modular Backbones: Instead of a single large network, MGTN employs multiple CNN backbones, each responsible for learning representations within a specific sub-graph. This "divide and conquer" strategy allows for more specialized learning.
  • Graph Transformers: Graph Transformer Networks are used to propagate information and learn representations across these sub-graphs, effectively integrating semantic and topological label information.
  • Eigenvector-based Embedding Transformation: To further enhance label representation, the paper proposes using eigenvector centrality to weight label embeddings, giving more importance to labels with stronger inter-relational dependencies.

Datasets

The proposed MGTN framework was evaluated on two widely-used multi-label image classification datasets:

  • MS-COCO: A popular dataset for object detection and segmentation, containing a large number of object instances and captions.
  • Fashion550K: A multi-label fashion dataset known for its weakly-annotated tags, presenting a challenge due to noisy labels.

Key Insights and Ideas

  • Modularity is Key: The paper strongly suggests that breaking down the complex label graph into smaller, more manageable sub-graphs is crucial for effective multi-label learning. This modularity helps in learning more distinct representations.
  • Specialized Learning: Using multiple CNN backbones, each dedicated to a specific sub-graph, allows for more focused and efficient learning of object representations within those groups.
  • Eigenvector Centrality for Importance: Incorporating eigenvector centrality to weight label embeddings is an effective way to highlight the importance of certain label relationships, leading to better performance.
  • EV-Enhancement Speeds Learning: The eigenvector-based enhancement not only improves performance but also significantly speeds up the learning process for MGTN.

Key Results

MGTN achieved state-of-the-art results on both MS-COCO and Fashion550K datasets.

  • MS-COCO: MGTN showed significant mAP improvements, outperforming previous state-of-the-art models by up to 9.7% compared to the ResNet101 baseline and 3.3% compared to KSSNet.
  • Fashion550K: The model achieved a 6.4% improvement in mAP over the ResNet50 baseline and 3.7% over KSSNet, demonstrating its effectiveness even on noisy datasets.
  • Ablation Studies: The paper also showed that different label embeddings impact learning speed, with BERT-based embeddings generally performing well. EV-enhancement consistently improved learning speed across various embeddings.

Limitations and Future Work

  • Computational Cost: While MGTN improves efficiency compared to some previous methods, the paper notes that earlier approaches using CNNs with RNNs were prone to high computational costs. The modular approach likely mitigates this, but it's an area of ongoing consideration.
  • Label Embeddings: The paper explores various label embeddings and suggests that using information from the last few layers of pre-trained models (like BERTavg-last) might be more beneficial than averaging across many layers (like RoBERTaavg_12). This indicates that further research into optimal label embedding strategies could be fruitful.
  • Future Directions: The authors suggest that their framework's ability to integrate semantic and topological information, combined with modularity, opens up new avenues for improving network niche learning and reducing overfitting, particularly for less common labels.

Summary updated

BibTeX
@inproceedings{nguyen2021modular,
  author = {Hoang D. Nguyen and Xuan-Son Vu and Duc-Trong Le},
  title = {{Modular Graph Transformer Networks for Multi-Label Image Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i10.17098},
}
Figure