Visual Attention in Multi-Label Image Classification.

Yan Luo, Ming Jiang, Qi ZhaoWorkshops @ CVPR 2019 · OpenReview

Abstract

One of the most significant challenges in multi-label image classification is the learning of representative features that capture the rich semantic information in a cluttered scene. As an information bottleneck, the visual attention mechanism allows humans to selectively process the most important visual input, enabling rapid and accurate scene understanding. In this work, we study the correlation between visual attention and multi-label image classification, and exploit an extra attention pathway for improving multi-label image classification performance. Specifically, we propose a dual-stream neural network that consists of two sub-networks: one is a conventional classification model and the other is a saliency prediction model trained with human fixations. Features computed with the two sub-networks are trained separately and then fine-tuned jointly using a multiple cross entropy loss. Experimental results show that the additional saliency sub-network improves multi-label image classification performance on the MS COCO dataset. The improvement is consistent across various levels of scene clutterness.

Introduction

Multi-label image classification involves identifying all relevant objects within a single image. Unlike single-label classification, real-world images often contain multiple objects of varying scales, often occluded or poorly lit within cluttered backgrounds. Standard Convolutional Neural Networks (CNNs) frequently struggle to isolate these semantic features from the surrounding noise.

Humans navigate this complexity through a selective attention mechanism, focusing on salient visual inputs. This research investigates whether integrating a computational model of human-like visual attention (saliency) can enhance the performance of deep learning models in multi-label tasks.

Methodology

The core of the proposed approach is a dual-stream neural network architecture. This design incorporates two distinct pathways to process visual information:

  1. The Trunk (Classification Stream): A standard ResNet-50 architecture tasked with learning discriminative features for object category identification.
  2. The Branch (Saliency Stream): A modified ResNet-50, based on the SALICON model, pre-trained on human fixation data to predict gaze patterns.
The proposed dual-stream neural network architecture featuring a saliency branch and a classification trunk.

Workflow and Training

Features from both streams are extracted and processed via spatial pooling before being concatenated into a single feature vector. The system is trained in three distinct phases: independent training of each stream, training of the final fusion layers, and finally joint fine-tuning of the entire network using a Multiple Cross Entropy (MCE) loss function, which is optimized for images containing multiple independent labels.

Key Insights

Initial analysis demonstrated a strong correlation between classification success and visual saliency. Objects correctly identified by baseline models tended to be larger, more centrally located, and possessed higher human saliency scores. By explicitly modeling this saliency, the dual-stream network learns to "attend" to regions that naturally draw human interest, which is particularly beneficial for identifying small or easily overlooked objects like handbags or wine glasses.

Experimental Results

The dual-stream model was evaluated on the MS COCO dataset, achieving a mean Average Precision (mAP) of 72.5%, a notable improvement over the 71.5% achieved by the ResNet-50 baseline.

mAP performance of the dual-stream model compared to the baseline across different numbers of objects per scene.

As shown in the performance analysis, the improvement is consistent across varying levels of scene complexity. The gain is particularly pronounced in scenes with fewer objects, where the saliency signal provides a clear guide for precise localization. Furthermore, Class Activation Maps (CAMs) revealed that the dual-stream model focuses its activations much more accurately on the target objects compared to the more diffused focus of the baseline model.

Conclusion

Integrating a dedicated saliency pathway into a classification network significantly improves multi-label image recognition. By mimicking human visual attention, the model achieves better feature representation and localization. Future work may involve exploring self-supervised attention mechanisms to reduce dependency on human fixation datasets and optimizing the architecture for real-time applications.

Summary updated

Abstract

One of the most significant challenges in multi-label image classification is the learning of representative features that capture the rich semantic information in a cluttered scene. As an information bottleneck, the visual attention mechanism allows humans to selectively process the most important visual input, enabling rapid and accurate scene understanding. In this work, we study the correlation between visual attention and multi-label image classification, and exploit an extra attention pathway for improving multi-label image classification performance. Specifically, we propose a dual-stream neural network that consists of two sub-networks: one is a conventional classification model and the other is a saliency prediction model trained with human fixations. Features computed with the two sub-networks are trained separately and then fine-tuned jointly using a multiple cross entropy loss. Experimental results show that the additional saliency sub-network improves multi-label image classification performance on the MS COCO dataset. The improvement is consistent across various levels of scene clutterness.
BibTeX
@inproceedings{luo2019visual,
  author = {Yan Luo and Ming Jiang and Qi Zhao},
  title = {{Visual Attention in Multi-Label Image Classification.}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=riivBeQeuTH},
}
Figure