Growing Efficient Accurate and Robust Neural Networks on the Edge

Vignesh Sundaresha, Naresh ShanbhagSubmission to ICLR 2025 · Rejected

TL;DR

Training neural networks to be robust against real-world corruptions typically requires heavy data augmentation and immense computational resources on the Cloud. This paper introduces GEARnn (Growing Efficient, Accurate, and Robust neural networks), a framework that trains compact, corruption-robust models entirely on-device (in-situ) on resource-constrained Edge hardware by combining single-step network growth on clean data with efficient robust augmentation.

What Problem Does This Paper Address?

Deploying deep learning on Edge devices (like autonomous vehicles, drones, and robotics) faces a double bind:

  1. Fragility to Corruptions: Real-world visual data is frequently degraded by weather, motion blur, and sensor noise. Standard defenses use robust data augmentations (e.g., AugMix), but these multiply training time and memory footprint several-fold.
  2. Edge Resource Constraints: Cloud-based training followed by pruning or quantization requires uploading sensitive field data, incurring privacy risks, high latency, and bandwidth costs. However, training robust models directly on Edge hardware is prohibitive—robustly training even a compact 5% VGG-19 model on CIFAR-10 can take more than a full day on an NVIDIA Jetson device.

Prior network growth techniques build models from tiny initial backbones, but they only optimize for clean data and require hundreds of expensive training epochs across multiple growth steps.

Methods

The authors propose GEARnn, built on two main mechanisms:

  • One-Shot Growth (OSG): Instead of multi-step growth over thousands of epochs, OSG performs a single growth step sandwiched between two training stages: train a tiny backbone $f_0$ for $\mathcal{E}_1$ epochs, grow width via splitting/adding neurons for a single growth step ($\mathcal{E}_g=1$), and train the grown network for $\mathcal{E}_2$ epochs.
  • Efficient Robust Augmentation (ERA): SOTA augmentation frameworks (e.g., AugMix) run parallel transform chains ($W > 1$), adding significant computational overhead. ERA optimizes this setup by setting the transform width $W=1$, chain depth $D=3$, and consistency samples $J=4$, reducing runtime while preserving corruption diversity.
  • 2-Phase Training (GEARnn-2): Rather than growing models directly on heavily augmented data (1-Phase / GEARnn-1), GEARnn-2 first executes OSG on clean data ($\mathcal{D}_\text{in}$ with cross-entropy loss) and then fine-tunes the resulting grown network with ERA on augmented data ($\mathcal{D}_\text{aug}$ with Jensen-Shannon divergence consistency loss).
Phase 1 (OSG on Clean Data):
  Tiny Backbone f_0 -> Train (Clean) -> Single Growth Step -> Train (Clean) -> f_2
                                                                                 |
Phase 2 (Robust Fine-tuning):                                                    v
  f_2 -> Train with ERA (Augmented Loss) -----------------------------------> Final Robust Model

Datasets

  • Standard (Clean / In-Distribution): CIFAR-10, CIFAR-100, and Tiny ImageNet.
  • Corruption Benchmarks (Out-of-Distribution): CIFAR-10-C, CIFAR-100-C, and Tiny ImageNet-C (evaluating 15 common corruption types across severity levels 1–5).
  • No new benchmarks were introduced; standard RobustBench suites were used.

Key Insights and Ideas

  • Clean Initialization Pre-Conditions Robustness: A 2-Phase approach (growing on clean data, then robustifying) significantly outperforms 1-Phase joint growth and augmentation. Frequency domain analysis reveals that both clean natural images and AugMix/common corruptions predominantly live in the low-frequency spectrum (unlike high-frequency adversarial noise). Pre-training on clean features provides an optimal initialization for low-frequency corruption defense, avoiding the chaotic optimization of growing on augmented samples.
  • One Growth Step is All You Need: Empirical comparisons against $m$-shot growth ($m \in \{2, 3, 4\}$) revealed that multi-step growth increases training overhead without improving accuracy or robustness. A single, well-placed growth step strikes the optimal trade-off between architectural flexibility and training cost.
  • Asymmetric Topology Evolution: OSG naturally assigns more channels to early layers in standard CNNs (VGG-19), matching precision patterns seen in quantization literature, whereas residual networks (ResNet-18) distribute channel growth uniformly across residual blocks.

Key Results

Experiments were benchmarked on a server GPU (NVIDIA Quadro RTX 6000) and embedded Edge devices (NVIDIA Jetson Xavier NX and Jetson Orin Nano):

  • Edge Efficiency Gains: On the Jetson Xavier NX across CIFAR-10 and CIFAR-100 (MobileNet-V1 and VGG-19), GEARnn-2 achieves comparable clean and robust accuracy to fully trained small baselines (Small trained with AugMix for 160 epochs) with an average $2.3\times$ reduction in training time and $2.8\times$ reduction in training energy.
  • Accuracies Maintained: On CIFAR-10 with VGG-19, GEARnn-2 achieves $83.45\%$ robust accuracy and $92.07\%$ clean accuracy in 596 minutes (155 kJ), compared to the standard augmented baseline's $85.73\%$ robust / $93.36\%$ clean accuracy requiring 1543 minutes (522 kJ).
  • Outperforming Prior Growth Methods: GEARnn-2 delivers $10\text{–}12\%$ higher robust accuracy at less than half the training time compared to standard network growth algorithms (e.g., Firefly and Splitting Steepest Descent).

Limitations and Future Work

  • Lack of Theoretical Convergence Bounds: The authors note that a theoretical convergence framework explaining why clean-data initialization accelerates robust training convergence is still lacking.
  • CNN-Centric Focus: Evaluations focus entirely on standard Edge CNN architectures (VGG-19, MobileNet-V1, ResNet-18); extending robust on-device growth to Vision Transformers (ViTs) remains an open challenge.
  • Supervised Data Dependency: The method currently relies on labeled data available locally on the device; adapting GEARnn to utilize self-supervised or active learning on streaming, unlabeled sensor data is highlighted as future work.

Summary updated

Abstract

The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.
BibTeX
@inproceedings{sundaresha2024growing,
  author = {Vignesh Sundaresha and Naresh Shanbhag},
  title = {{Growing Efficient Accurate and Robust Neural Networks on the Edge}},
  booktitle = {Compression @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=8rko2UqXdB},
}
Figure