Rethinking 360deg Image Visual Attention Modelling With Unsupervised Learning.

Yasser Abdelaziz Dahou Djilali, Tarun Krishna, Kevin McGuinness, Noel E. O’ConnorICCV 2021

TL;DR

This paper introduces a novel unsupervised learning framework for predicting visual attention in 360° images. By adapting contrastive learning techniques, the model learns robust representations that perform on par with fully supervised methods, while significantly reducing computational costs and the need for large labeled datasets.

What Problem Does This Paper Address?

Predicting where humans will look in 360° images (visual saliency) is crucial for applications like VR content creation, efficient data compression, and user experience design. Traditional methods often rely on supervised learning, which is hampered by the scarcity of large, accurately annotated 360° datasets due to the complex annotation process. Furthermore, existing approaches often struggle with the unique geometric distortions of 360° images, particularly near the poles, and can be computationally expensive. This paper aims to bridge this gap by leveraging the vast amount of unlabeled 360° imagery through unsupervised learning.

Methods

The core of the proposed method is an unsupervised contrastive learning framework designed to learn representations from 360° images. The approach leverages the inherent geometric properties of spherical data by maximizing mutual information between different views of the same scene.

Key components include:

  • Base Encoder (fθ): A convolutional neural network (e.g., VGG16 or ResNet50) that learns initial latent representations from input 360° images.
  • Global Module (Σσ): This module generates a compact, global representation of the input image.
  • Self-Attention Module (φβ): This module enhances the model's ability to capture spatial relationships between different parts of the image, inspired by how humans focus on specific regions.
  • Local Module (Ωω): This module processes local spatial patches from augmented views.
  • Contrastive Loss: The framework uses a Noise Contrastive Estimation (NCE) based loss to maximize agreement between global and local representations, effectively learning invariant features. The loss is formulated as a convex combination of global and local NCE losses, allowing for a trade-off between them.
  • Supervised Decoder (gϕ): After the unsupervised encoder is trained, a lightweight supervised decoder is trained on top of the frozen encoder to perform the downstream saliency prediction task. This significantly reduces the number of parameters needed for the supervised part.

A key innovation is the use of specific projections (equator-to-front and bottom-to-front) during training, which are argued to be more robust than traditional cube map projections for learning saliency.

Datasets

The paper utilizes several datasets for evaluation:

  • Salient360!: A small-scale dataset with head and eye movement data for training and validation.
  • VR-EyeTracking: A large-scale dataset derived from videos, providing a substantial amount of data for training.
  • Sitzmann: A dataset containing gaze and head orientation data.

The authors also mention gathering a new dataset of 90K 360° images from multiple sources for training the unsupervised encoder, highlighting the need for large unlabeled datasets.

Key Insights and Ideas

  • Unsupervised Learning for Saliency: The paper demonstrates that unsupervised contrastive learning, typically used for recognition tasks, can be effectively adapted for regression tasks like saliency prediction, opening new avenues for research.
  • Geometric Properties of 360° Data: The authors argue that the spherical nature of 360° images is not a hindrance but an advantage for contrastive learning, providing unique opportunities for MI maximization, especially by leveraging pole regions.
  • Efficiency Gains: By decoupling the representation learning (unsupervised) from the prediction task (supervised), the model achieves significant computational speedups (over 8x faster than the next best competitor) and requires far fewer parameters in the supervised decoder.
  • Projection Strategy: The choice of projections (equator-to-front, bottom-to-front) over traditional cube maps is shown to be crucial for learning robust representations, particularly for avoiding distortions near the poles.

Key Results

  • Competitive Performance: The proposed unsupervised model achieves performance on par with, and in some cases exceeding, fully supervised state-of-the-art methods on the Salient360! and VR-EyeTracking+Sitzman datasets across various metrics (AUC-J, NSS, CC, SIM, KLD).
  • Top Performance on VR-EyeTracking+Sitzman: The model achieved the top score for all metrics on the VR-EyeTracking+Sitzman dataset.
  • Computational Efficiency: The model is over 8x faster than the fastest competing 360° saliency approach (ATSAL) in terms of GPU inference time.
  • Reduced Parameter Count: The supervised decoder requires significantly fewer parameters (3.8x fewer for ResNet50) compared to traditional supervised approaches.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the unsupervised encoder is not fine-tuned on the downstream task, the decoder is. This means the final prediction still relies on some supervised learning.
  • The choice of hyperparameters, such as the weighting factor λ in the loss function, was determined empirically.
  • Future Work:
    • The authors suggest that the contrastive encoder could be generalized to other 360° downstream tasks like visual quality assessment (VQA) and compression.
  • Exploring the "gaze policy" embedded in the brain through unsupervised learning is highlighted as a promising future research direction.
  • Further investigation into the effect of different projection strategies and augmentation techniques for contrastive learning in 360° domains.

Summary updated

BibTeX
@inproceedings{djilali2021rethinking,
  author = {Yasser Abdelaziz Dahou Djilali and Tarun Krishna and Kevin McGuinness and Noel E. O’Connor},
  title = {{Rethinking 360deg Image Visual Attention Modelling With Unsupervised Learning.}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
}
Figure