Rapid Salient Object Detection with Difference Convolutional Neural Networks

Zhuo Su, Li Liu, Matthias Müller, Jiehua Zhang, Diana Wofk, Ming-Ming Cheng, Matti Pietikäinen

Salient Object Detection (SOD) is the task of identifying the most visually distinctive objects in a scene—essentially, what "pops out" to a human viewer. While deep learning has pushed the boundaries of accuracy, top-performing models are often massive and computationally expensive, making them difficult to deploy on mobile phones, IoT devices, or embedded systems like the Nvidia Jetson. This paper, recently accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), presents a clever way to make SOD both incredibly fast and highly accurate for resource-constrained edge devices.

Bridging the Gap: Contrast Cues and CNNs

Classical SOD methods relied on simple "contrast cues"—looking for differences in color or texture—which made them fast but less robust. Modern CNNs are powerful but often ignore these explicit contrast heuristics. The authors propose a way to build a model that has the representational power of a CNN but the efficiency and "common sense" of classical contrast-based methods.

Core Innovations: PDC and DCR

The authors introduce two main innovations to bridge this gap:

  1. Pixel Difference Convolutions (PDC): Instead of a standard convolution that just sums weighted pixel values, PDC explicitly computes the differences between a central pixel and its neighbors. This mimics the "center-surround" mechanism of the human visual system, making it naturally better at detecting edges and contrasts.
  2. Difference Convolution Reparameterization (DCR): This is the "secret sauce" for efficiency. During training, the model uses multiple PDC branches to learn rich contrast features. However, at inference time, these branches are mathematically collapsed into a single, standard convolutional layer. This means the model gains the benefit of complex contrast-aware features for "free"—no extra parameters or computation at runtime.
Visualizing different Pixel Difference Convolution (PDC) and SpatioTemporal Difference Convolution (STDC) operators used to capture local and motion contrast.

For video salient object detection (VSOD), the authors extend this to SpatioTemporal Difference Convolutions (STDC). Rather than using heavy 3D convolutions, they decompose the video volume into orthogonal planes (Width-Time and Height-Time) to capture motion contrast efficiently.

Experimental Results

The models, SDNet (for images) and STDNet (for video), were rigorously tested on several standard benchmarks, including ECSSD for images and DAVSOD for video. The results are particularly impressive when looking at the accuracy-runtime trade-off:

  • Speed: On an Nvidia Jetson AGX Orin, SDNet runs at 46 FPS (images) and STDNet runs at 150 FPS (video). This is 2× to 14× faster than previous state-of-the-art lightweight models.
  • Efficiency: Both models maintain less than 1 million parameters, making them ideal for memory-limited devices.
  • Accuracy: Despite their small size, they achieve higher F-measure scores than much larger competitors.
Accuracy vs. runtime (FPS) trade-offs for image (left) and video (right) SOD benchmarks. SDNet and STDNet (red) significantly outperform competitors in both speed and accuracy.

Conclusion

SDNet and STDNet demonstrate that explicit contrast encoding through difference convolutions provides a significant performance boost when combined with reparameterization. By fusing complex training-time branches into simple inference-time layers, the authors created architectures that perform like much heavier models while remaining ultra-lightweight. This work sets a new standard for real-time saliency detection on edge devices and suggests that the "Difference Convolution" spirit could be applied to other tasks like object tracking or medical image analysis.

Summary updated

BibTeX
@article{su2025rapid,
  author = {Zhuo Su and Li Liu and Matthias Müller and Jiehua Zhang and Diana Wofk and Ming-Ming Cheng and Matti Pietikäinen},
  title = {{Rapid Salient Object Detection with Difference Convolutional Neural Networks}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3583968},
}
Figure