TL;DR
PiCANet introduces a novel "pixel-wise" attention mechanism that allows every individual pixel in an image to selectively "attend" to the most informative parts of its context. By integrating global and local attention into a U-Net architecture, the model achieves state-of-the-art results in salient object detection and demonstrates strong generalization to semantic segmentation and object detection.
What Problem Does This Paper Address?
Saliency detection aims to identify the most "important" or eye-catching regions in an image. Traditionally, this relies on contextual contrast: a pixel is salient if it stands out from its surroundings.
While Convolutional Neural Networks (CNNs) are great at extracting features, they typically process context holistically. This means that when a model looks at a pixel, it considers everything within its fixed receptive field equally. The authors argue this is sub-optimal because much of that context is "noise" or distraction. For example, to decide if a pixel on a dog's nose is salient, the model should prioritize looking at the background (for contrast) and other parts of the dog (for consistency), rather than irrelevant objects in the distance.
Methods
The core contribution is the PiCANet (Pixel-wise Contextual Attention Network). Unlike standard attention models that might produce one attention map for an entire image, PiCANet generates a unique attention map for every single pixel.
The authors propose three specific flavors of this mechanism:
- Global Attention Pooling (GAP): This allows a pixel to attend to the entire image. To make this computationally feasible, they use ReNet (a combination of horizontal and vertical LSTMs) to sweep the image and capture global dependencies before generating attention weights via a softmax layer.
- Local Attention Pooling (LAP): This focuses on a local neighborhood. It uses standard convolutions to generate attention weights that help the pixel "see" visually coherent neighboring regions.
- Attention Convolution (AC): Instead of just pooling features, this version uses sigmoid-gated attention weights to modulate the information flow during the convolution operation itself.
These modules are embedded into a U-Net architecture. GAP is used in the early (top) decoder stages where feature maps are small but semantically rich, while LAP/AC are used in later stages to refine details.
Datasets
The model was trained on the DUTS-TR dataset and evaluated on six major saliency benchmarks:
- SOD (300 images, complex backgrounds)
- ECSSD (1,000 semantically meaningful images)
- PASCAL-S (850 images from PASCAL VOC)
- DUT-O (5,168 challenging images)
- HKU-IS (4,447 images, often with multiple objects)
- DUTS-TE (5,019 test images)
The authors also tested generalization on PASCAL VOC 2012 (semantic segmentation) and PASCAL VOC 2007 (object detection).
Key Insights and Ideas
- Global Contrast via Attention: One of the most striking findings is that the global attention mechanism (GAP) naturally learns to perform contrast. When looking at a foreground pixel, the attention map highlights the background, and vice versa.
- Local Smoothness: Local attention (LAP) tends to highlight regions with similar appearances to the reference pixel. This helps the model "fill in" the interior of salient objects uniformly, preventing the "patchy" look common in older models.
- Explicit Supervision: The authors found that implicitly learning attention is hard. They introduced a novel explicit global attention loss ($L_{GA}$) that forces the model to attend to the background when looking at the foreground (and vice versa), significantly boosting performance.
- Beyond Saliency: The "pixel-wise" attention concept is a general-purpose neural module. The fact that it improved mIOU in semantic segmentation and mAP in object detection suggests it helps any task requiring fine-grained contextual reasoning.
Key Results
- Saliency Detection: PiCANet outperformed 13 state-of-the-art methods across all six datasets. For instance, on the DUTS-TE dataset, it achieved an F-measure of 0.859 and a Mean Absolute Error (MAE) of 0.051, significantly better than previous leaders like DGRL or BMP.
- Semantic Segmentation: Adding PiCANet to a DeepLab baseline improved mIOU on PASCAL VOC 2012 from 68.96% to 70.12%.
- Object Detection: Integrating it into an SSD300 framework improved mAP on PASCAL VOC 2007 from 77.2% to 78.0%.
Limitations and Future Work
- Computational Overhead: While ReNet makes global attention "efficient," generating a unique attention map for every pixel is still more computationally expensive than standard convolutions. The prediction process takes about 0.127s per image on a GTX 1080 Ti.
- Fixed Attention Size: For local attention, the authors used a fixed window (e.g., 13x13). Future work could explore multi-scale or truly adaptive window sizes.
- Noisy Supervision: The authors noted that explicitly supervising local attention (forcing pixels to look at others with the same label) actually degraded performance slightly, likely because regions with the same label don't always look similar in cluttered scenes. Refining how local attention is supervised remains an open challenge.
Summary updated