Introduction
Deploying Convolutional Neural Networks (CNNs) in safety-critical edge applications faces two major hurdles: high computational costs and vulnerability to adversarial attacks. Traditionally, researchers tackle these challenges by combining model compression (like pruning or quantization) with adversarial training.
However, this combined approach has severe drawbacks. First, many compression techniques—especially unstructured pruning—achieve high theoretical compression rates but fail to improve actual throughput (Frames Per Second, or FPS) on off-the-shelf hardware. Second, methods that do improve FPS often suffer significant drops in adversarial robustness. Finally, relying on adversarial training to recover lost robustness is incredibly slow and computationally expensive, making it hard to scale to complex datasets or advanced threat models.
To bridge the gap between theoretical efficiency and real-hardware performance, this paper introduces Generalized Depthwise-Separable (GDWS) convolutions, a novel, post-training approximation technique for standard 2D convolutions. By replacing standard convolutions with GDWS, it is possible to achieve massive real-world throughput speedups on pre-trained neural networks without sacrificing their adversarial robustness or requiring expensive retraining.
The GDWS Approach
Standard Depthwise-Separable (DWS) convolutions (popularized by architectures like MobileNet) apply exactly one depthwise filter per input channel. GDWS relaxes this constraint, allowing a variable number of depthwise filters per channel. This flexibility allows GDWS to accurately approximate complex, pre-trained 2D convolutions without destroying the network's learned behavior.
The authors frame the standard 2D convolution as a matrix multiplication and prove that any 2D convolution can be mathematically decomposed into a GDWS format using Singular Value Decomposition (SVD). To perform this approximation optimally, they introduce two greedy, polynomial-time algorithms:
- MEGO: Minimizes error under a strict computational complexity constraint.
- LEGO: Minimizes complexity under a strict error constraint.
Furthermore, to decide how aggressively to approximate each layer of a deep network, the authors use a metric based on the expected "noise gain." Layers whose approximation errors would heavily impact the final classification logits are approximated more carefully.
Key Insights
- Hardware-Friendly by Design: Because GDWS relies on standard depthwise and pointwise convolution operations, it maps seamlessly onto existing off-the-shelf hardware (like the NVIDIA Jetson Xavier). It delivers actual FPS improvements, unlike unstructured pruning which often requires custom hardware to realize speedups.
- Bypassing the Retraining Bottleneck: Because GDWS is an accurate mathematical approximation applied post-training, it completely eliminates the need for adversarial retraining. This is a massive advantage when dealing with complex defenses where retraining takes exorbitant amounts of time.
- Synergy with Unstructured Pruning: One of the most surprising findings is how well GDWS pairs with highly sparse, pruned models (like those generated by HYDRA). Sparse weight matrices naturally have low-rank sub-matrices, meaning GDWS can approximate them with almost zero error. Applying GDWS to a 99% pruned model magically translates its theoretical sparsity into actual hardware speedups.
Experimental Results
The paper evaluates the proposed method on standard image classification benchmarks including CIFAR-10, SVHN, and ImageNet, testing against advanced adversarial attack models (such as the union of $L_\infty$, $L_2$, and $L_1$ norm-bounded perturbations).
- Massive FPS Boosts: On an NVIDIA Jetson Xavier, applying GDWS to a robust pre-trained ResNet-18 on CIFAR-10 improved throughput from 28 FPS to 104 FPS (>250% improvement), while keeping both natural and robust accuracy within ~1% of the baseline.
- Beating NAS Architectures: Compared to RobNet (a network specifically found via Neural Architecture Search for robustness), a GDWS-approximated WideResNet-28-4 achieved 65 FPS compared to RobNet's 5 FPS at similar robustness and model size.
- Unlocking Pruning Speedups: When applied to a 99% pruned WideResNet (which only ran at 28 FPS natively due to unstructured sparsity), GDWS boosted the throughput to 68 FPS without degrading the pruned model's accuracy.
- Union of Perturbations: Applied to a ResNet-18 trained to defend against a union of $L_\infty, L_2,$ and $L_1$ attacks, GDWS improved FPS from 28 to 101 with a negligible (<1%) drop in robust accuracy.
Limitations and Future Work
While GDWS drastically improves computational speed (FPS), it only offers modest reductions in actual model size (memory footprint) compared to aggressive pruning techniques. Additionally, GDWS cannot be applied to networks that already utilize standard Depthwise-Separable convolutions (like MobileNet or RobNet).
Future work may explore training GDWS-structured networks from scratch, as well as investigating brief fine-tuning post-approximation to recover the tiny fractions of robustness lost during the SVD decomposition.
Summary updated