Where Weight Averaging Fails: Recovering Snapshot-Ensemble Gains on Tiny Objects at Single-Model Cost

A neural network's training run passes through many good solutions. Saving several of them ("snapshots") and averaging their predictions gives a cheap ensemble that is more accurate than any single snapshot. That ensemble still costs one forward pass per snapshot at test time. The standard fix is to average the snapshots' weights instead of their outputs, which yields a single model. On image classification this works almost as well as the full ensemble. We propose to test whether it keeps working when the task depends on objects only a few pixels wide.

There are reasons to doubt it. In those objects, the evidence survives each downsampling step in a few weak, spatially sparse feature channels, and their bounding boxes swing from correct to missed under a three-pixel shift. Our hypothesis is that the gap between averaging outputs and averaging weights is close to zero for large objects and grows as objects shrink. If it does, the ensemble's benefit can be recovered selectively. We would weight-average most of the detector, and then distill the snapshot ensemble's dense predictions into the model only at the high-resolution feature levels where tiny objects live. The deployed result is one detector with no added inference cost.

Core Claim

The claim has two parts, and the second depends on the first. The diagnostic part: for a detector trained on tiny objects, the averaging gap is a monotonically decreasing function of object size. The averaging gap is defined as

$$\Delta(s) = \mathrm{AP}_{\text{ens}}(s) - \mathrm{AP}_{\text{avg}}(s),$$

where $\mathrm{AP}_{\text{ens}}(s)$ is the average precision of the output-averaged snapshot ensemble on objects in size bucket $s$, and $\mathrm{AP}_{\text{avg}}(s)$ is that of the single weight-averaged model built from the same snapshots. Relatedly, the loss barrier along straight lines between snapshots should be higher when measured only on tiny objects than when measured on all objects. The method part: if $\Delta(s)$ is concentrated at small $s$, then the following deployed model closes most of the small-object gap more cheaply than distilling the whole ensemble. Start from the weight-averaged detector and fine-tune it with a distillation loss applied only at the finest feature-pyramid levels, with the snapshot ensemble's predictions cached offline as targets. If the diagnostic finds $\Delta(s)\approx 0$ everywhere, the method collapses to plain weight averaging, and we report that as the answer.

Why This Is Open

The snapshot literature supplies the ingredients but has only tested them on whole-image classification.

Several papers establish that one training run can supply a usable ensemble, and they supply the snapshot-harvesting technique we reuse. Stochastic Gradient Descent with Warm Restarts shows that periodically resetting the learning rate kicks the optimizer into new low-loss regions. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs shows that good solutions are joined by low-loss paths. Its Fast Geometric Ensembling (FGE) uses short learning-rate cycles to collect 12 to 22 checkpoints, cutting ResNet-164 CIFAR-100 error from 21.5% to 18.21%, but at 12 to 22 times the inference cost. Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling extends those paths to whole low-loss volumes.

Three lines of work then collapse such ensembles into one model, and they define our baselines. Averaging Weights Leads to Wider Optima and Better Generalization averages trajectory weights (Stochastic Weight Averaging, SWA) and matches FGE on CIFAR-100 at single-model cost. Hierarchical Weight Averaging for Deep Neural Networks adds synchronization across parallel runs. Learning Neural Network Subspaces trains a line of models and deploys its midpoint. Learn From the Past: Experience Ensemble Knowledge Distillation distills attention-weighted snapshots into a student, but only on class logits, which carry no spatial information. Collaborative Learning for Deep Neural Networks trains several heads on a shared trunk and discards the extra heads at deployment. Every one of these is evaluated on CIFAR or ImageNet classification. None of them asks whether averaging preserves localized, sparse spatial evidence.

Two papers motivate the hypothesis that it might not. LIP: Local Importance-Based Pooling replaces fixed pooling with a learned importance map computed by a small gating subnetwork with sigmoid outputs. This raises Faster R-CNN small-object AP on COCO from 21.9% to 24.0%. Because the gate is nonlinear, averaging its weights need not average its maps. Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark shows that a three-pixel shift drops the box-overlap score (IoU) of a 5×8 vehicle from 0.54 to 0.14. This suggests that the loss surface for tiny objects is narrow in exactly the directions where snapshots differ. That paper's Normalized Wasserstein Distance with Ranking-based Assigning (NWD-RKA), which treats boxes as Gaussians when matching them to targets, also constrains our design. Any improvement we claim must hold on top of NWD-RKA and LIP, not instead of them.

Research Program

The base detector is Faster R-CNN with a ResNet-50 feature pyramid, a LIP backbone, and NWD-RKA assignment. We freeze batch normalization or use group normalization so that weight averaging does not require the statistics-refresh pass that SWA needs. We train it to convergence, then run $K$ short cyclic learning-rate cycles in the FGE style. We save one snapshot per cycle with $K \in \{4, 8, 12\}$. The same snapshot set feeds every method, so differences come only from how the snapshots are combined.

The diagnostic phase measures $\Delta(s)$ across the size buckets of AI-TOD-v2 (very tiny, tiny, small, medium) and the COCO size buckets. It traces AP by size along the straight line between pairs of snapshots to get size-resolved loss barriers. It also measures per-location disagreement between snapshots at each pyramid level. We then repeat the gap measurement under two weight-averaging variants. In one, every weight is averaged. In the other, the LIP gating subnetwork comes from the best single snapshot and the rest of the network is averaged. This tells us whether any gap sits in the gates specifically.

The method phase caches the snapshot ensemble's outputs on the training set at the finest one or two pyramid levels (P2/P3): classification scores and Gaussian box parameters. We store them sparsely, only at locations with ensemble score above a low threshold, so each snapshot costs one inference pass over the data and no teacher runs during fine-tuning. We then fine-tune the weight-averaged detector briefly with

$$\mathcal{L} = \mathcal{L}_{\text{det}} + \lambda \sum_{\ell \in \mathcal{F}} \sum_{x} D\big(p^{\text{ens}}_{\ell}(x),\, p_{\ell}(x)\big).$$

Here $\mathcal{L}_{\text{det}}$ is the usual NWD-RKA detection loss. $\mathcal{F}$ is the set of fine pyramid levels. $x$ ranges over the cached locations. $p^{\text{ens}}_\ell(x)$ and $p_\ell(x)$ are the ensemble's and the student's predictions at level $\ell$ and location $x$. $D$ is a KL divergence on scores plus a Wasserstein distance on box Gaussians. $\lambda$ trades distillation against the detection loss.

Baselines

The naive reference point is the converged detector trained for the same total number of epochs with an ordinary decaying schedule. It costs one forward pass and fixes the zero point for every gain.

The strongest existing single-model method on this benchmark is NWD-RKA. Detecting tiny objects in aerial images reports 24.7 AP on AI-TOD-v2 with DetectoRS. For the COCO small-object regime, the reference is Faster R-CNN with LIP at 24.0% small-object AP, from LIP. We reproduce both in our codebase and use our own reproduced numbers as the comparison. Since our base detector already includes both components, beating them means improving on our reproduction at equal total training epochs.

The most direct competitor is SWA applied to the same snapshots. We found no reported result on Lacuna for SWA or FGE on any detection benchmark, so we establish these numbers ourselves. SWA uses the same snapshots, has no extra fine-tuning, and has identical inference cost. Our method beats it if its very-tiny and tiny AP exceed SWA's by more than the seed-to-seed standard deviation across three seeds, with no loss on medium objects. We also give SWA the same number of additional epochs our fine-tuning uses, continued at low learning rate, so extra training alone cannot explain a win.

The output-averaged snapshot ensemble is the accuracy ceiling. We report it with its K-fold latency.

Two ablations isolate the key ingredient, which is distillation restricted to fine levels. The first distills the ensemble at all pyramid levels, an EEKD-style dense distillation with cached targets, at matched fine-tuning steps. The second keeps the fine-level distillation but starts the fine-tuning from the best single snapshot instead of the weight average. A Hierarchical Weight Averaging run with $K$ parallel models is included as a training-compute-heavier weight-averaging competitor. It is matched in total GPU-hours, not epochs.

Why not just…?

Why not just apply SWA to the detector? This is the real threat. Nothing in the file shows SWA failing on detection, and if $\Delta(s)$ is near zero for tiny objects it is the correct answer. The diagnostic phase is designed to settle this before any method work. If the very-tiny-bucket gap is within noise, we stop and report that weight averaging is sufficient for tiny objects, which is itself a useful and unreported result. The proposed method only matters if the gap is real and size-dependent.

Why not just use LIP and NWD-RKA alone, which already target tiny objects? They change how features are pooled and how boxes are matched. They do not touch the run-to-run variance that ensembles reduce. Because both sit inside our base detector, every gain we report is measured on top of them. If snapshots of a LIP+NWD detector turn out nearly identical (low disagreement at P2/P3), the ensemble has nothing to offer and the diagnostic will show it.

Why not process images at higher resolution or with a high-resolution branch? Dual-branch designs like A Multi-Branch Feature Fusion Network for Building Detection in Remote Sensing Images preserve detail by adding test-time compute. That is the cost this proposal refuses to pay. We report a 1.5× input-resolution baseline with its latency so readers can see the accuracy-per-millisecond trade-off directly, rather than assume ours is better.

Why not distill the ensemble's full output, as EEKD does? This is our first ablation. If all-level distillation matches fine-level distillation at equal cost, then restricting to fine levels is not the active ingredient. We would then claim only the diagnostic finding plus the cheap cached-target recipe.

Proposed Experiments

The primary benchmark is AI-TOD-v2, where objects average 12.7 pixels, reported by its size buckets. COCO 2017 provides small, medium, and large AP to check that large-object accuracy is not traded away. VisDrone2019 tests transfer without retuning. Metrics are AP per size bucket, calibration error of detection scores binned by object size, measured latency and peak memory at inference, and total training GPU-hours including caching.

Three outcomes decide the proposal. First, $\Delta(s)$ must be positive and larger for very-tiny than medium objects, consistently across three seeds and $K \in \{4, 8, 12\}$. Size-resolved interpolation barriers should show the same ordering. Second, the fine-level distilled model must close at least half of the very-tiny gap between SWA and the ensemble at 1× inference cost. Third, the all-level distillation ablation must close less of the gap per unit of fine-tuning compute. If the first check fails, the paper is a negative result about tiny-object ensembling. If the first passes and the second fails, the gap is real but not distillable at this cost, which points toward keeping the gating subnetwork unaveraged, a variant the diagnostic already measures.