Growing Efficient Accurate and Robust Neural Networks on the Edge

Vignesh Sundaresha, Naresh Shanbhag

TL;DR

GEARnn enables resource-constrained edge devices to grow and train compact, corruption-resistant neural networks entirely in-situ (on-device), bypassing the cloud. By utilizing a novel two-phase "One-Shot Growth" strategy on clean data followed by efficient robust training, it slashes edge training time and energy consumption by up to 3.5× while maintaining high robust accuracy.

What Problem Does This Paper Address?

Deploying deep learning models to edge devices (like autonomous drones, robots, or smart cameras) faces a double whammy: these models are computationally massive, and they are notoriously fragile when encountering real-world "out-of-distribution" corruptions like blur, rain, or snow.

To make models robust against these corruptions, researchers typically use heavy data augmentation during training. However, doing this directly on an edge device is painfully slow—robustly training a standard VGG-19 network on a simple dataset can take over two days on an edge accelerator.

The traditional workaround is to train and compress models in the cloud and then deploy them to the edge. But this introduces severe bottlenecks: transmitting raw field data to the cloud drains battery power, introduces latency, and raises serious data privacy concerns. This paper tackles the challenge of training compact, highly robust networks completely from scratch on the edge device itself.

Methods

The authors introduce GEARnn (Growing Efficient, Accurate, and Robust neural networks), which constructs efficient networks using two core components:

  1. One-Shot Growth (OSG): Instead of starting with a massive, over-parameterized network and pruning it down, OSG starts with a tiny, easy-to-fit "backbone" network. It trains this small network briefly, expands its width exactly once (using a neuron-splitting algorithm), and then continues training. This single growth step avoids the massive computational overhead of traditional multi-step architecture searches.
  2. Efficient Robust Augmentation (ERA): Standard state-of-the-art robust training frameworks (like AugMix) are computationally expensive. ERA optimizes the stochastic transformation chains (tuning width, depth, and consistency parameters) specifically to maximize robustness while minimizing edge hardware overhead.

The authors explore two training pipelines: GEARnn-1 (a single phase where growth and robust training happen simultaneously on augmented data) and GEARnn-2 (a two-phase approach where the network is grown on clean data first, and then robustified on augmented data).

Figure 2: Proposed approach: GEARnn-1 performs One-Shot Growth (OSG) on augmented data (Daug) generated by Efficient Robust Aug- mentation (ERA) (using clean data (Din)) in a single phase (1-Phase). GEARnn-2 performs OSG using Din first followed by parametric train- ing on Daug in two consecutive phases (2-phase). Here LCE and Laug denote the cross-entropy loss and augmented loss, respectively.
Figure 3: OSG takes in labeled data (D) and backbone network f0, and performs a training step, a growth step, and a training step in sequence to generate network f2. The 2-tuple (L, E) = (loss function, number of epochs) employed in each step.

Datasets

The method was evaluated using standard image classification datasets alongside their corrupted counterparts to benchmark out-of-distribution robustness:

  • Clean Data ($D_{in}$): CIFAR-10, CIFAR-100, and Tiny ImageNet.
  • Corrupted Data ($D_{out}$): CIFAR-10-C, CIFAR-100-C, and Tiny ImageNet-C (which introduce 15 types of natural corruptions like weather, blur, and digital noise).

Key Insights and Ideas

  • The Power of Clean Initialization (The 2-Phase Advantage): Counterintuitively, growing the network on clean data first (GEARnn-2) yields significantly better final robustness than growing it directly on augmented data (GEARnn-1).
  • Why Clean Data Helps Robustness: By analyzing the Fourier spectrum of the images, the authors discovered that both clean images and common natural corruptions primarily occupy the low-frequency domain. Because their underlying frequency distributions align, initializing a network on clean data places it in an ideal loss landscape to quickly learn corruption robustness during the second phase. This contrasts with adversarial perturbations, which typically occupy high-frequency domains.
  • One Growth Step is All You Need: Restricting the architecture search to a single growth step (OSG) provides comparable or superior accuracy to complex multi-step growth methods while drastically reducing training overhead.

Key Results

The authors extensively profiled GEARnn on an NVIDIA Jetson Xavier NX edge device and a server-grade Quadro GPU, comparing it against fully trained baselines and static small baselines.

  • Massive Training Speedups: On the Jetson edge device, GEARnn-2 achieved a 2.3× to 3.5× reduction in training time compared to standard robust training baselines.
  • Energy Efficiency: GEARnn-2 delivered up to a 2.8× reduction in training energy consumption on the edge, which is critical for battery-powered devices.
  • Maintained Performance: Despite being a fraction of the size and taking far less time to train, GEARnn-2 matched or exceeded the robust accuracy of standard small baselines and closely approached the performance of full-sized cloud-trained models.
Figure 1: Improvements in robust accuracy, train- ing time, and model size (area of circles) of our proposed GEARnn method measured on NVIDIA Jetson Xavier NX Edge device NVIDIA (a). Ro- bust accuracy is evaluated on CIFAR-10-C for GEARnn, full network baselines (VGG-19), and small network baselines (5% VGG-19 networks with same topology as GEARnn-2). For robust training, we employ AugMix Hendrycks et al. (2019). GEARnn demonstrates significant reduc- tion in training complexity over robust baselines at similar robust and clean accuracies (shown in Sec- tion 6.2).

Limitations and Future Work

While the empirical results are compelling, the authors note that a theoretical convergence analysis for why the 2-phase growth approach works so efficiently is currently lacking. Providing a mathematical proof for these favorable initial conditions remains an open problem.

Additionally, the current work focuses on image classification. The authors point out that promising future directions include extending GEARnn to leverage unlabeled data (semi-supervised learning) and adapting the framework for more complex edge workloads like real-time object detection.

Summary updated

Abstract

The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.
BibTeX
@inproceedings{sundaresha2024growing,
  author = {Vignesh Sundaresha and Naresh Shanbhag},
  title = {{Growing Efficient Accurate and Robust Neural Networks on the Edge}},
  booktitle = {Compression @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=8rko2UqXdB},
}
Figure