TL;DR
This paper introduces SalTR, a novel transformer-based approach for predicting visual saliency by directly learning from discrete human fixation points, rather than relying on continuous saliency maps. By treating saliency prediction as a set prediction problem and leveraging parallel decoding in transformers, SalTR mimics the data collection process and achieves state-of-the-art performance on established benchmarks.
What Problem Does This Paper Address?
Predicting where humans will look on an image (visual saliency) is a fundamental problem in computer vision with applications ranging from image compression to robotics. Traditional approaches often rely on continuous saliency maps, where each pixel represents a probability of attention. However, these continuous maps are derived from discrete human eye fixations, and optimizing directly for these discrete points presents a challenge. Existing deep learning models, while successful, often don't directly replicate the way saliency datasets are collected (i.e., by aggregating individual fixations). This paper aims to bridge this gap by developing a model that learns saliency directly from these discrete fixation points, mimicking the experimental setup of saliency dataset creation.
Methods
The core of SalTR is a transformer encoder-decoder architecture. The process begins with a CNN backbone (ResNet50) to extract image features. These features are then processed by a transformer encoder to enrich them with contextual information. The key innovation lies in the transformer decoder. Instead of generating a continuous saliency map, the decoder uses a fixed set of learned "fixation queries." These queries, through a cross-attention mechanism, attend to the image features and directly output a set of spatial fixation points. This approach frames saliency prediction as a direct set prediction problem.
To handle the discrete nature of fixations, SalTR employs a bipartite matching strategy (using the Hungarian algorithm) to find the optimal correspondence between predicted and ground truth fixation points. The loss function combines an L1 loss on the predicted fixation points with a normalized scanpath saliency (NSS) loss on the generated saliency maps. The authors also explore a "Deformable SalTR" variant that incorporates a deformable attention mechanism to improve optimization stability.
Datasets
- Salicon: Used for training and evaluation. The paper mentions using the 10k/5k train/validation splits.
- MIT300: Used for out-of-distribution evaluation.
Key Insights and Ideas
- Learning from Discrete Fixations: The most significant insight is that directly learning from discrete fixation points, rather than post-processed continuous maps, is feasible and effective. This approach more closely mirrors how saliency datasets are created.
- Fixation Queries as Viewers: The learned "fixation queries" in the transformer decoder act as simulated viewers, directing attention to specific image regions to predict fixation locations. This is a novel way to leverage the transformer's attention mechanism for this task.
- Set Prediction for Saliency: Framing saliency prediction as a direct set prediction problem, similar to object detection with DETR, allows for a more principled approach to handling multiple fixations.
- Bipartite Matching for Uniqueness: The use of bipartite matching is crucial for ensuring that each predicted fixation point corresponds to a unique ground truth fixation, preventing "fixation query collapse."
- Deformable Attention for Stability: The deformable attention mechanism in Deformable SalTR helps stabilize training and improve performance, likely by allowing queries to focus on more relevant spatial locations.
Key Results
- Salicon Performance: SalTR-Base achieves competitive results on the Salicon benchmark, with scores on par with state-of-the-art methods like UNISAL and EML-NET across various metrics (SIM, s-AUC, CC, NSS, KLD).
- MIT300 Performance: As an out-of-distribution test, SalTR shows reasonable improvement compared to models not trained on this dataset, though it doesn't surpass models specifically trained on MIT300.
- Scanpath Prediction: The paper demonstrates that SalTR can be adapted for scanpath prediction by using a causal mask in the decoder, achieving a competitive Multi-Match (MM) score of 0.93 on Salicon.
- Impact of Fixation Queries: The study found that 100 fixation queries yielded optimal performance on Salicon, with diminishing returns for higher numbers.
- Gaussian Smoothing Parameter: A standard deviation of 19.0 for Gaussian smoothing generally provided the best results on Salicon.
Limitations and Future Work
- Synthetic Image Failure: The paper acknowledges that SalTR struggles with synthetic images from the O3 dataset, failing to respond well to low-level features and producing near-random fixations. This suggests a limitation in its ability to generalize to highly artificial stimuli.
- Single Subject vs. Batch Metrics: The authors note that when sampling a single subject's fixations for prediction, the resulting saliency maps don't match baseline scores. This is attributed to metrics that are designed for aggregated fixations over a batch of subjects, potentially requiring an "over-estimated" saliency map.
- Optimization Challenges: The paper mentions that SalTR can be difficult to optimize, requiring many epochs for convergence, and suggests that attention weights initialization and the discrete nature of bipartite matching might contribute to this.
- Future Work: The authors suggest that their approach opens up possibilities for "application-guided saliency prediction" due to the flexibility of their design. Further exploration into improving performance on synthetic data and understanding the nuances of metric evaluation for single-subject predictions would be valuable.
Summary updated