Sign-Preserving Activations Before the Depthwise Filter

Mobile vision networks are built from depthwise separable convolutions: a convolution split into a cheap per-channel spatial filter (the "depthwise" step, which blurs, sharpens, or detects edges within each channel independently) followed by a 1×1 layer that mixes channels. In the standard MobileNetV2 block, a ReLU6 clips the wide expanded features to zero from below just before the depthwise filter. We argue this is a poor place to rectify. Rectifying zero-mean noise turns it into a positive offset, and a spatial filter whose weights sum to a positive value then adds up that offset over its whole window. Because no other channel is mixed in at that step, nothing can cancel it. We propose one change at no added cost: replace the pre-depthwise ReLU6 with a symmetric clip, $\mathrm{clip}(x,-c,c)$, and keep ReLU6 after the depthwise filter. The symmetric clip is odd, meaning $f(-x)=-f(x)$, so it preserves the sign of noise and lets positive and negative fluctuations cancel inside the filter. It is also bounded, so large impulses cannot pass through. It is as cheap as ReLU6 on mobile hardware.

The prediction is specific and can be falsified. Robustness should improve most on the noise corruptions of ImageNet-C (Gaussian, shot, impulse) and far less on blur or weather. The noise-induced shift at each depthwise output should scale with the filter's weight sum under ReLU6 and should largely vanish under the symmetric clip. Clean accuracy should stay within a small margin of MobileNetV2's reported 72.0% top-1 on ImageNet (MobileNetV2). If the gains are spread evenly across all corruption types, or if simply adding noise to the training data matches them, the mechanism is wrong.

Core Claim and Grounding

The mechanism comes from How deep convolutional neural networks lose spatial information with training. That paper shows that as convolutional networks learn to ignore smooth deformations, they become more sensitive to white noise. The reason is that ReLU converts zero-mean noise into a positive-mean signal, which spatial pooling then accumulates in proportion to the receptive field, whereas odd activations let the noise cancel. That analysis motivates our proposal, but it studies standard convolutions and simplified scale-detection tasks. It never looks at the one place where rectification and channel-isolated spatial summation sit back to back, which is the expansion activation feeding a depthwise filter.

To first order, if $\xi$ is zero-mean noise entering a channel and $w$ is that channel's $3\times3$ depthwise filter, the expected shift at the filter output is $\mathbb{E}[f(\xi)]\sum_{ij} w_{ij}$. For ReLU6 the first factor is strictly positive. For any odd $f$ it is zero. Batch normalization does not remove this shift at test time, because its statistics are frozen from clean training data.

MobileNetV2 supplies the architecture and one constraint. Its "linear bottleneck" result says to avoid nonlinearities in the narrow projection layers because they destroy information. We leave the narrow layers untouched and change only the activation in the wide expanded space, where MobileNetV2 itself argues ReLU is tolerable. Our claim is therefore not in conflict with the linear bottleneck finding; it concerns where, inside the wide part of the block, the rectification should sit. The saturation half of the design follows the reasoning in ELU, which holds that a bounded, flat regime stops small input fluctuations from propagating. The symmetric clip saturates on both sides rather than only on the negative side.

Relation to Existing Work at the Intersection

Three papers already combine depthwise convolutions with robustness or activation design, and each leaves this question open.

Generalized Depthwise-Separable Convolutions takes pretrained, adversarially trained standard convolutions and approximates them after training with depthwise-plus-pointwise factorizations. It keeps whatever ReLUs the original model had, and targets adversarial rather than natural-noise robustness.

Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization bounds the linear operators of depthwise layers cheaply. It treats the activation as a passive 1-Lipschitz map, so it controls worst-case gain but not the systematic noise offset we target.

Activate or Not (Meta-ACON) learns input-conditioned activations on MobileNetV1 and reports large clean-accuracy gains. It reports no corruption or noise evaluation, and its Swish-like family is not odd.

Our proposal differs from all three in the same way. It fixes a structural property, oddness, at one specific position in the block and tests it against a predicted pattern of corruption errors.

One paper points the other way. Beyond the Universal Law of Robustness argues, in the lazy-training kernel regime, that even activations reduce worst-case adversarial sensitivity. SPLASH finds that its learned activations drift toward symmetric V or W shapes and resist adversarial attacks better. We treat these results as a genuine tension. Oddness should help against random noise, which cancels on average, while evenness is argued to help against worst-case perturbations, which do not cancel. We therefore report adversarial robustness too, and we do not claim a win there.

Research Program

The work has three parts.

The first is a measurement study in trained networks. For each inverted residual block of a trained MobileNetV2, we inject Gaussian noise at the input and record the per-channel mean shift and variance at the output of the expansion activation, the depthwise filter, and the projection. We regress the depthwise output shift on each filter's weight sum. The mechanism predicts a strong positive slope under ReLU6 and a near-zero slope under the symmetric clip, with only a small second-order remainder coming from clipping of large inputs. We also check whether the downsampling (stride-2) blocks, whose learned filters tend to be low-pass with positive weight sums, carry most of the effect.

The second part trains the architecture variants from scratch under identical recipes. The main variant uses the symmetric clip before the depthwise filter and ReLU6 after it. The clip level $c$ is a fixed hyperparameter, set by default to 6 to match ReLU6's range and swept over a small set.

The third part is a check on the learned-activation side. We place a SPLASH unit at the pre-depthwise position only, and separately at the post-depthwise position only, and inspect the shapes it converges to. If SPLASH trained with corruption-free data drifts toward odd shapes before the depthwise filter but not after it, that is independent evidence that position matters.

Baselines

The naive reference is unmodified MobileNetV2 with ReLU6 in both positions, trained with our recipe. Its reported clean ImageNet figure is 72.0% top-1 at 300M multiply-adds and 3.4M parameters (MobileNetV2). We found no reported ImageNet-C result for MobileNetV2 on Lacuna, so we measure its corruption error ourselves. For scale, the only reference point in the file is ResNet-50 at 76.7 mean corruption error (Benchmarking Neural Network Robustness to Common Corruptions and Perturbations). That figure comes from a different, larger model and is not a target to beat.

The strongest competing method is Meta-ACON applied to every activation in the same MobileNetV2. The file reports only its clean gains: +6.7% top-1 on MobileNetV1 at 0.25 width, and 1.8% on ResNet-152 (Activate or Not). No corruption result for it was found on Lacuna. It is the strongest competitor because it is the best-documented activation change on depthwise backbones. If its gain in clean accuracy carries over to corruption error, our structural argument is unnecessary.

The two other competitors are SPLASH at all positions, which has learnable, possibly symmetric shapes, and MobileNetV3-style hard-swish, the standard smooth activation in mobile networks.

Two ablations isolate the key ingredient. The first is a linear (identity) pre-depthwise activation. This is also odd, so it tests whether saturation adds anything beyond sign preservation; impulse noise is where saturation should matter. The second places the symmetric clip after the depthwise filter while keeping ReLU6 before it. This tests placement: our mechanism predicts it helps little.

Fairness rules apply to every variant. All share the same data, training epochs, augmentation, optimizer schedule, and a matched hyperparameter sweep of six configurations each. The learnable activations get their own learning-rate sweep within the same budget. Latency is measured on the same phone CPU.

Beating the baselines means a lower mean corruption error than both unmodified MobileNetV2 and Meta-ACON, with the reduction concentrated in the noise category. It also requires clean top-1 within 0.5 points of the ReLU6 baseline and no measurable latency increase.

Why not just…?

Why not just train with Gaussian noise augmentation? This is the most serious threat. Noise augmentation directly teaches filters to ignore noise, and it may well recover most of the noise-category gain. We treat it as a crossed factor: every variant is trained both with and without noise augmentation. Our claim survives only if the gains from the symmetric clip add on top of augmentation's gains, or if the clip matches augmentation's noise gains without augmentation's usual cost on clean accuracy and on untrained-for corruptions. If augmentation alone closes the gap, we will say so. The contribution would then shrink to a diagnostic account of where in a mobile network the noise enters.

Why not just re-estimate batch-norm statistics on corrupted test data? Our first-order argument says the damage is a mean offset that frozen batch-norm statistics fail to subtract. Adapting those statistics at test time might therefore remove it. This is a real alternative, and we include it as a baseline evaluated on ImageNet-C. It differs from our method in requiring unlabeled corrupted batches at deployment, which a single-image mobile pipeline often lacks. The per-block measurement study also tells us how much of the damage is mean shift, which normalization can fix, versus added variance, which it cannot.

Why not just swap in hard-swish, GELU, or tanh everywhere? Hard-swish and GELU are smooth approximations of a rectifier: they send negative inputs to roughly zero instead of preserving their sign. That places them in the family Activate or Not calls smooth ReLUs, so by the mechanism in How deep convolutional neural networks lose spatial information with training they should still convert noise into a positive offset. Using tanh everywhere would also make the post-depthwise activation odd and remove the block's only rectification. MobileNetV2's own analysis suggests this costs capacity. The all-hard-swish baseline and the post-depthwise-clip ablation settle both questions directly.

Proposed Experiments

The primary run is MobileNetV2 at widths 1.0 and 0.5 on ImageNet. We report clean top-1, mean corruption error and its per-category breakdown on ImageNet-C, and mean flip probability on ImageNet-P. The ImageNet-P reference point is ResNet-18's reported 15.6% flip probability between adjacent frames (Benchmarking Neural Network Robustness to Common Corruptions and Perturbations). For fast sweeps over clip level and placement, we use a CIFAR-sized MobileNetV2 on CIFAR-10 and CIFAR-100, corrupted with the same corruption code. Detection transfer is checked with SSDLite on MS COCO, where MobileNetV2's reported figure is 22.1 mAP. We add Gaussian noise to the COCO images to test whether the noise gain survives in a dense-prediction head.

As a secondary check tied to the even-symmetry tension, we report FGSM and one-pixel attack accuracy on CIFAR-10 against SPLASH. SPLASH reports up to 28.5% and 31% robustness improvements over ReLU under these attacks (SPLASH). We expect no win here and would read a large loss as a real cost of the design.

Three results decide the question. Does the noise-category share of the corruption-error reduction clearly exceed its share of the benchmark? Does the per-channel shift-versus-weight-sum slope collapse under the symmetric clip? Does the benefit survive crossing with noise augmentation? A positive answer to all three establishes that the placement of rectification relative to a channel-isolated spatial filter is a design parameter mobile networks have been getting wrong. A negative answer on the third tells practitioners they can safely keep ReLU6 and just augment.