Salient Object Detection (SOD) is the task of identifying the most visually distinctive objects in a scene—essentially, what "pops out" to a human viewer. While deep learning has pushed the boundaries of accuracy, top-performing models are often massive and computationally expensive, making them difficult to deploy on mobile phones, IoT devices, or embedded systems like the Nvidia Jetson. This paper, recently accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), presents a clever way to make SOD both incredibly fast and highly accurate for resource-constrained edge devices.
Bridging the Gap: Contrast Cues and CNNs
Classical SOD methods relied on simple "contrast cues"—looking for differences in color or texture—which made them fast but less robust. Modern CNNs are powerful but often ignore these explicit contrast heuristics. The authors propose a way to build a model that has the representational power of a CNN but the efficiency and "common sense" of classical contrast-based methods.
Core Innovations: PDC and DCR
The authors introduce two main innovations to bridge this gap:
- Pixel Difference Convolutions (PDC): Instead of a standard convolution that just sums weighted pixel values, PDC explicitly computes the differences between a central pixel and its neighbors. This mimics the "center-surround" mechanism of the human visual system, making it naturally better at detecting edges and contrasts.
- Difference Convolution Reparameterization (DCR): This is the "secret sauce" for efficiency. During training, the model uses multiple PDC branches to learn rich contrast features. However, at inference time, these branches are mathematically collapsed into a single, standard convolutional layer. This means the model gains the benefit of complex contrast-aware features for "free"—no extra parameters or computation at runtime.
For video salient object detection (VSOD), the authors extend this to SpatioTemporal Difference Convolutions (STDC). Rather than using heavy 3D convolutions, they decompose the video volume into orthogonal planes (Width-Time and Height-Time) to capture motion contrast efficiently.
Experimental Results
The models, SDNet (for images) and STDNet (for video), were rigorously tested on several standard benchmarks, including ECSSD for images and DAVSOD for video. The results are particularly impressive when looking at the accuracy-runtime trade-off:
- Speed: On an Nvidia Jetson AGX Orin, SDNet runs at 46 FPS (images) and STDNet runs at 150 FPS (video). This is 2× to 14× faster than previous state-of-the-art lightweight models.
- Efficiency: Both models maintain less than 1 million parameters, making them ideal for memory-limited devices.
- Accuracy: Despite their small size, they achieve higher F-measure scores than much larger competitors.
Conclusion
SDNet and STDNet demonstrate that explicit contrast encoding through difference convolutions provides a significant performance boost when combined with reparameterization. By fusing complex training-time branches into simple inference-time layers, the authors created architectures that perform like much heavier models while remaining ultra-lightweight. This work sets a new standard for real-time saliency detection on edge devices and suggests that the "Difference Convolution" spirit could be applied to other tasks like object tracking or medical image analysis.
Summary updated