TL;DR
GEARnn enables resource-constrained edge devices to grow and train compact, corruption-resistant neural networks entirely in-situ (on-device), bypassing the cloud. By utilizing a novel two-phase "One-Shot Growth" strategy on clean data followed by efficient robust training, it slashes edge training time and energy consumption by up to 3.5× while maintaining high robust accuracy.
What Problem Does This Paper Address?
Deploying deep learning models to edge devices (like autonomous drones, robots, or smart cameras) faces a double whammy: these models are computationally massive, and they are notoriously fragile when encountering real-world "out-of-distribution" corruptions like blur, rain, or snow.
To make models robust against these corruptions, researchers typically use heavy data augmentation during training. However, doing this directly on an edge device is painfully slow—robustly training a standard VGG-19 network on a simple dataset can take over two days on an edge accelerator.
The traditional workaround is to train and compress models in the cloud and then deploy them to the edge. But this introduces severe bottlenecks: transmitting raw field data to the cloud drains battery power, introduces latency, and raises serious data privacy concerns. This paper tackles the challenge of training compact, highly robust networks completely from scratch on the edge device itself.
Methods
The authors introduce GEARnn (Growing Efficient, Accurate, and Robust neural networks), which constructs efficient networks using two core components:
- One-Shot Growth (OSG): Instead of starting with a massive, over-parameterized network and pruning it down, OSG starts with a tiny, easy-to-fit "backbone" network. It trains this small network briefly, expands its width exactly once (using a neuron-splitting algorithm), and then continues training. This single growth step avoids the massive computational overhead of traditional multi-step architecture searches.
- Efficient Robust Augmentation (ERA): Standard state-of-the-art robust training frameworks (like AugMix) are computationally expensive. ERA optimizes the stochastic transformation chains (tuning width, depth, and consistency parameters) specifically to maximize robustness while minimizing edge hardware overhead.
The authors explore two training pipelines: GEARnn-1 (a single phase where growth and robust training happen simultaneously on augmented data) and GEARnn-2 (a two-phase approach where the network is grown on clean data first, and then robustified on augmented data).
Datasets
The method was evaluated using standard image classification datasets alongside their corrupted counterparts to benchmark out-of-distribution robustness:
- Clean Data ($D_{in}$): CIFAR-10, CIFAR-100, and Tiny ImageNet.
- Corrupted Data ($D_{out}$): CIFAR-10-C, CIFAR-100-C, and Tiny ImageNet-C (which introduce 15 types of natural corruptions like weather, blur, and digital noise).
Key Insights and Ideas
- The Power of Clean Initialization (The 2-Phase Advantage): Counterintuitively, growing the network on clean data first (GEARnn-2) yields significantly better final robustness than growing it directly on augmented data (GEARnn-1).
- Why Clean Data Helps Robustness: By analyzing the Fourier spectrum of the images, the authors discovered that both clean images and common natural corruptions primarily occupy the low-frequency domain. Because their underlying frequency distributions align, initializing a network on clean data places it in an ideal loss landscape to quickly learn corruption robustness during the second phase. This contrasts with adversarial perturbations, which typically occupy high-frequency domains.
- One Growth Step is All You Need: Restricting the architecture search to a single growth step (OSG) provides comparable or superior accuracy to complex multi-step growth methods while drastically reducing training overhead.
Key Results
The authors extensively profiled GEARnn on an NVIDIA Jetson Xavier NX edge device and a server-grade Quadro GPU, comparing it against fully trained baselines and static small baselines.
- Massive Training Speedups: On the Jetson edge device, GEARnn-2 achieved a 2.3× to 3.5× reduction in training time compared to standard robust training baselines.
- Energy Efficiency: GEARnn-2 delivered up to a 2.8× reduction in training energy consumption on the edge, which is critical for battery-powered devices.
- Maintained Performance: Despite being a fraction of the size and taking far less time to train, GEARnn-2 matched or exceeded the robust accuracy of standard small baselines and closely approached the performance of full-sized cloud-trained models.
Limitations and Future Work
While the empirical results are compelling, the authors note that a theoretical convergence analysis for why the 2-phase growth approach works so efficiently is currently lacking. Providing a mathematical proof for these favorable initial conditions remains an open problem.
Additionally, the current work focuses on image classification. The authors point out that promising future directions include extending GEARnn to leverage unlabeled data (semi-supervised learning) and adapting the framework for more complex edge workloads like real-time object detection.
Summary updated