Capsule Graph Neural Network for Multi-Label Image Recognition (Student Abstract)

Xiangping Zheng, Xun Liang, Bo WuAAAI 2022 · DOI

Introduction

Multi-label image recognition aims to identify all objects within a single image, such as a "keyboard," "mouse," and "monitor." Unlike single-label classification, this task requires understanding complex topological relationships where certain objects frequently co-occur while others do not. Current approaches face two primary hurdles: standard Convolutional Neural Networks (CNNs) often fail to capture the spatial hierarchy and part-whole relationships of objects, and existing Graph Convolutional Network (GCN) methods typically rely on rigid, pre-calculated label correlation matrices that lack flexibility.

To address these issues, this paper introduces the Capsule Graph Neural Network (CGML). CGML combines the structural feature extraction capabilities of Capsule Networks with the dynamic relational reasoning of an adaptive GCN.

The CGML Framework

The CGML architecture is divided into two main components designed to handle image features and label dependencies respectively.

The CGML architecture, featuring the Image Representation Learning block and the Labels Adaptive Graph Convolutional Network.

Image Representation Learning (IRL)

The IRL module utilizes Capsule Networks (CapsNets) instead of traditional CNN layers. Capsules represent object properties—such as position, size, and orientation—as vectors, which are better suited for capturing part-whole relationships. To maintain computational efficiency, the authors implemented a capsule selection layer. This layer filters the capsules, passing only the $k$ most "active" ones to subsequent levels, thereby reducing noise and focusing the model on the most relevant visual signals.

Label Adaptive Graph Convolutional Network (LAGCN)

The LAGCN module models labels as nodes in a graph. While previous methods used fixed co-occurrence statistics, LAGCN employs an adaptive attention mechanism. This allows the network to dynamically update label features based on both immediate neighbors and long-range contextual information. By learning these dependencies during training, the model can adapt to the specific nuances of the dataset rather than relying on static counts.

The final classification is achieved by fusing the image features from the IRL and the label embeddings from the LAGCN via a dot product.

Experimental Results

The performance of CGML was evaluated on two major multi-label benchmarks: MS-COCO (80 categories) and VOC 2007 (20 categories).

Comparison of CGML performance against state-of-the-art baselines on MS-COCO and VOC2007 datasets.

As shown in the results table, CGML consistently outperformed strong baselines, including ResNet-101 and recent GCN-based models like ML-GCN, A-GCN, and F-GCN. On the MS-COCO dataset, CGML achieved a mean Average Precision (mAP) of 86.7%, a significant improvement over the 83.2% achieved by F-GCN. Similarly, on VOC 2007, the model reached 96.2% mAP, surpassing the previous state-of-the-art.

Conclusion and Future Work

The success of CGML highlights the advantages of using Capsule Networks for robust feature extraction and adaptive graphs for modeling label dependencies. By moving beyond simple co-occurrence statistics and raw pixel intensities, CGML captures the underlying structure of multi-label scenes more effectively.

While the capsule selection module improves efficiency, the inherent complexity of CapsNets and adaptive GCNs remains a consideration for scaling to datasets with thousands of labels. Future research will likely focus on further refining the interaction between visual features and label embeddings to push the boundaries of multi-label recognition performance.

Summary updated

BibTeX
@inproceedings{zheng2022capsule,
  author = {Xiangping Zheng and Xun Liang and Bo Wu},
  title = {{Capsule Graph Neural Network for Multi-Label Image Recognition (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21695},
}
Figure