Scanpath Prediction for Visual Attention using IOR-ROI LSTM

Zhenzhong Chen, Wanjie SunIJCAI 2018 · DOI

TL;DR

This paper introduces a novel deep learning architecture, IOR-ROI LSTM, that effectively predicts human visual scanpaths by simultaneously modeling "inhibition of return" (IOR) and region-of-interest (ROI) selection. By integrating these two crucial aspects of visual attention, the model achieves superior performance in predicting where and in what order humans will look at an image.

What Problem Does This Paper Address?

Understanding and predicting human visual attention, specifically how our eyes move across a scene (visual scanpaths), is a fundamental challenge in computer vision. Scanpaths are sequences of fixations (where we look) and saccades (rapid eye movements between fixations). While many models predict where attention will be focused (saliency maps), predicting the order of these fixations is more complex. This is crucial for applications like human-computer interaction, robotics, and understanding visual perception. A key challenge is accurately modeling two core mechanisms: inhibition of return (IOR), which discourages re-examining recently fixated locations, and the selection of regions of interest (ROIs) that capture our attention. Existing models often struggle to integrate these dynamics effectively, particularly when using deep learning approaches that might overlook spatial relationships or the temporal nature of IOR.

Methods

The proposed model, IOR-ROI LSTM, is built upon a convolutional LSTM (ConvLSTM) framework. It comprises three main components:

  1. Image Feature Extractor: A modified VGG-19 network that extracts visual features from the input image. The modification involves adjusting pooling layers and using dilated convolutions to preserve higher spatial resolution in the feature maps, which is crucial for detailed attention prediction.
  2. IOR-ROI LSTM Module: This is the core innovation. It's a dual-LSTM unit designed to work in tandem:
  3. IOR-LSTM: This component models the dynamics of inhibition of return. It simulates the visual working memory (VWM) by adaptively integrating and forgetting scene information. Unlike linear models, it uses a novel update mechanism (element-wise maximum) to better capture VWM behavior. The output of this LSTM generates an "inhibited feature map" that guides the ROI selection by highlighting areas that should not be focused on.
  4. ROI-LSTM: This component predicts the next region of interest. It takes the inhibited image features (processed by IOR-LSTM) and uses a soft attention mechanism to determine the most likely next fixation location. It's also based on ConvLSTM, allowing it to process spatial features effectively.
  5. Saliency Guidance Network: During training, this network takes image features and predicts a saliency map. This is used as an auxiliary loss to guide the feature extractor to better encode global saliency information, improving the overall feature representation.

The model is trained using a combined loss function that includes binary cross-entropy for IOR, ROI, and saliency maps, with scalar factors to balance their contributions.

Datasets

The paper uses two publicly available eye-tracking datasets:

  • OSIE Dataset: Contains 700 images with eye-tracking data from 15 participants per image. Images are 800x600 pixels.
  • MIT Dataset: Comprises 1003 images with resolutions ranging from 405 to 1024 pixels, also with data from 15 subjects per image.

The authors also describe a pre-processing step where consecutive fixations within a certain radius (representing the foveal receptive field) are merged to create "refined scanpaths," acknowledging that most human scanpaths exhibit this behavior.

Key Insights and Ideas

  • IOR as a Predictive Signal: The paper strongly advocates for explicitly modeling Inhibition of Return (IOR) as a key factor in scanpath prediction, not just as a post-hoc explanation. They show that integrating IOR dynamics significantly boosts prediction accuracy.
  • Dual LSTM for Complementary Dynamics: The separation of IOR modeling (IOR-LSTM) and ROI selection (ROI-LSTM) within a single architecture allows each component to specialize, leading to a more comprehensive model of visual attention.
  • VWM Simulation for IOR: The IOR-LSTM's novel update mechanism, simulating visual working memory with an element-wise maximum operation, is a key departure from standard LSTM and better reflects how recently attended information is retained and older information decays.
  • Spatial Context Preservation: Modifying the VGG-19 feature extractor to maintain higher spatial resolution is critical for accurately predicting fine-grained fixation locations and understanding the spatial relationships that influence attention.
  • Saliency as a Training Aid: Using a saliency guidance network during training helps the feature extractor learn more robust global representations, which in turn benefits the IOR-ROI LSTM.

Key Results

The IOR-ROI LSTM model, especially when combined with saliency guidance, consistently outperforms several state-of-the-art scanpath prediction models on both the OSIE and MIT datasets across various evaluation metrics (DTW, MultiMatch, ScanMatch).

  • OSIE Dataset: The IOR-ROI LSTM + Saliency guidance model achieved the best results in ScanMatch (0.413) and was competitive in other metrics, outperforming models like Single ConvLSTM and Dual ConvLSTM.
  • MIT Dataset: Similar trends were observed, with the IOR-ROI LSTM + Saliency guidance model showing strong performance, particularly in ScanMatch (0.39) and being competitive in MultiMatch metrics.
  • Qualitative Results: Visualizations show that the generated scanpaths closely match the shape and locations of ground truth human scanpaths, and the IOR dynamics visualization (heatmaps) effectively illustrate how the model predicts inhibition and attention shifts.

Limitations and Future Work

  • Subject-Specific Modeling: While the paper mentions training subject-specific models and averaging results for evaluation, the primary model presented is not inherently subject-specific. The authors note that scanpath prediction can be confused by inter-subject variability in IOR behavior. Future work could focus on developing more robust subject-specific models or methods to handle this variability.
  • MIT Dataset Challenges: The authors acknowledge that the MIT dataset's varied image content and scales might pose challenges, leading to slightly less optimal performance in some metrics compared to the OSIE dataset. This suggests that domain adaptation or more robust feature extraction might be beneficial for diverse datasets.
  • Computational Cost: While not explicitly detailed as a limitation, deep learning models, especially those involving ConvLSTMs and modified VGG networks, can be computationally intensive. Optimizing the architecture for speed could be a future direction.
  • Generalizability: While tested on two datasets, further evaluation on a wider range of image types and tasks would be valuable to confirm the model's generalizability.

Summary updated

BibTeX
@inproceedings{chen2018scanpath,
  author = {Zhenzhong Chen and Wanjie Sun},
  title = {{Scanpath Prediction for Visual Attention using IOR-ROI LSTM}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/89},
}

Loading related content...

Figure