iNAS: Integral NAS for Device-Aware Salient Object Detection

Yuchao Gu, Shang-Hua Gao, Xu-Sheng Cao, Peng Du, Shao-Ping Lu, Ming-Ming Cheng

Salient Object Detection (SOD)—the task of identifying the most "interesting" parts of an image—is a cornerstone of modern computer vision, powering features like portrait mode and smart video cropping. However, most state-of-the-art (SOTA) models are "GPU-heavy," designed for massive desktop cards and performing poorly when ported to mobile CPUs or embedded devices.

In the paper "iNAS: Integral NAS for Device-Aware Salient Object Detection," researchers from Nankai University and Huawei Technologies introduce a framework that doesn't just design a better model, but designs the right model for specific hardware targets.

The Challenge of Device-Specific SOD

The field of SOD currently suffers from a "silo" problem. Researchers typically focus on either designing a powerful backbone (the feature extractor) or a complex saliency head (the part that fuses features into a mask), often ignoring the synergy between the two. Furthermore, a model optimized for an NVIDIA GPU might be sluggish on a mobile phone because different hardware architectures have different bottlenecks, such as memory bandwidth versus raw computation.

Currently, achieving low-latency SOD for a specific device requires labor-intensive hand-design. iNAS automates this by making the entire architecture searchable and "device-aware."

The authors propose three key innovations to make automated, hardware-aware design feasible:

1. Integral Search Space

Unlike previous Neural Architecture Search (NAS) methods that only optimize the backbone, iNAS searches the Backbone, the Transport layers (which connect features), and the Decoder (which produces the final output) simultaneously. This holistic approach allows the saliency head to compensate for backbone choices, leading to more efficient overall designs.

Comparison between handcrafted saliency heads and the proposed iNAS integral search space.

2. Searchable Multi-Scale Unit (SMSU)

To capture objects of different sizes, models usually require complex multi-branch structures. iNAS utilizes SMSUs, which employ multiple branches during training but use structural reparameterization to collapse those branches into a single, high-speed convolution for deployment.

3. Latency-Group Sampling (LGS)

Standard NAS sampling tends to find "average" models. LGS divides the search space into specific latency "buckets." This ensures the search explores the entire spectrum, from ultra-fast mobile models to high-performance desktop models, creating a diverse Pareto frontier of options.

Key Results and Insights

The models discovered by iNAS consistently outperformed hand-crafted SOTA models in speed-to-accuracy trade-offs across various hardware platforms:

  • Significant Speedups: Compared to previous SOTA methods, iNAS reduced latency by 3.8× on the Huawei Nova6 SE, 3.3× on Intel CPUs, and 2.6× on the NVIDIA Jetson Nano.
  • Efficiency: The iNAS(GPU)-L model matches top-tier accuracy (Max F-measure of 0.947 on ECSSD) while being significantly leaner in terms of FLOPs and wall-clock time.
  • Zero-Retraining Deployment: Using a "one-shot" supernet approach, the resulting models can be deployed immediately after the search without needing to be trained from scratch.
Performance comparison on the Huawei Nova6 SE, showing iNAS (stars) achieving significantly higher speeds with competitive accuracy.

Conclusion

iNAS demonstrates that searching the backbone and head together while incorporating device-specific latency leads to superior real-world performance. While the search phase still requires significant compute (0.8 GPU-days), the resulting models bridge the gap between high-accuracy research and practical mobile deployment. Future work may extend this "integral" approach to video salient object detection and memory-constrained environments.

Summary updated

Abstract

Existing salient object detection (SOD) models usually focus on either backbone feature extractors or saliency heads, ignoring their relations. A powerful backbone could still achieve sub-optimal performance with a weak saliency head and vice versa. Moreover, the balance between model performance and inference latency poses a great challenge to model design, especially when considering different deployment scenarios. Considering all components in an integral neural architecture search (iNAS) space, we propose a flexible device-aware search scheme that only trains the SOD model once and quickly finds high-performance but low-latency models on multiple devices. An evolution search with latency-group sampling (LGS) is proposed to explore the entire latency area of our enlarged search space. Models searched by iNAS achieve similar performance with SOTA methods but reduce the 3.8×, 3.3×, 2.6×, 1.9× latency on Huawei Nova6 SE, Intel Core CPU, the Jetson Nano, and Nvidia Titan Xp. The code is released at https://mmcheng.net/inas/.
BibTeX
@inproceedings{gu2021inas,
  author = {Yuchao Gu and Shang-Hua Gao and Xu-Sheng Cao and Peng Du and Shao-Ping Lu and Ming-Ming Cheng},
  title = {{iNAS: Integral NAS for Device-Aware Salient Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00489},
}
Figure