Does Activation Compression Survive Residual Restoration Networks?

Training memory for image and video restoration networks is dominated by activations, the intermediate layer outputs that must be kept from the forward pass so the backward pass can compute gradients. One of the most effective ways to shrink them, dual activation precision (DIVISION), rests on one observation. Gradients mostly depend on the smooth, low-frequency part of each activation map. So the method keeps a blurred copy in high precision and squeezes the remaining sharp detail into 2 bits. That observation was checked only on classifiers and language models, whose outputs are abstract labels.

Restoration networks work differently. Super-resolution, deblurring and video restoration models are usually built with global residual learning: the network adds its prediction to a cheap coarse estimate of the output, such as a bicubic upsample, so it only has to predict the sharp detail the coarse estimate is missing. We hypothesize that this design pushes the network's intermediate activations toward high frequencies. If so, the 2-bit detail band would carry exactly the signal the loss rewards, and blind frequency-split compression would quietly damage the textures and edges that define restoration quality.

This note proposes to measure that effect first and then repair it. The repair is a precision budget for each layer, set from that layer's measured high-frequency energy. The saved memory is then spent on what restoration training actually lacks: larger crops and larger batches. Today those are capped by the 128×128 patches typical of multi-frame video models (MultiBoot VSR).

Core Claim

The claim has two parts. First, how compressible a network's activations are depends on how its output is set up, not only on its architecture. Two otherwise identical super-resolution networks, one predicting the full image and one predicting the residual $R = Y - X_{\text{base}}$, will put different fractions of activation energy in the detail band. The residual network will put more there, especially in late layers. As a result, DIVISION applied unchanged will lose more fidelity on the residual network, and most of that loss will appear on detail-heavy test sets.

Second, a compression policy that measures each layer's detail-band share and assigns precision accordingly can recover the uncompressed quality at roughly the same memory. That memory, spent on larger spatial context rather than on doing nothing, should improve restoration quality at equal compute.

To make the per-layer policy concrete, write $A_\ell$ for the activation stored at layer $\ell$, $P$ for average pooling with block size $k$, and $U$ for nearest upsampling. The detail-band share of layer $\ell$ is then

$$\rho_\ell(k) = \frac{\|A_\ell - U(P_k(A_\ell))\|^2}{\|A_\ell\|^2}.$$

DIVISION implicitly assumes $\rho_\ell$ is small everywhere. Our policy chooses $k_\ell$ and the bit width of the detail band for each layer, under a total memory budget, so that $\rho_\ell$ multiplied by the quantization error of that bit width stays below a common tolerance. This allocates bits where detail energy lives.

Relation to Prior Work

DIVISION supplies the method we build on. It splits each activation into a pooled low-frequency part (8×8 blocks, bfloat16) and a 2-bit residual. It proves that gradient error depends mainly on the low-frequency part and reports 75.9% ImageNet top-1 with ResNet-50 against 76.15% uncompressed, at over 10× activation compression. It never tests a task whose loss lives in high frequencies, and it uses one fixed block size and bit width for all layers. We test that untested regime and make the split adaptive.

Three restoration papers motivate the hypothesis that residual objectives concentrate learning on rare high-frequency content:

None of these examines what the residual objective does to the internal activations that memory-saving methods compress.

Other memory methods constrain our design choices because they pay in compute or target transformer-specific operations:

  • Reversible transformers remove depth-dependent activation storage at a 33–38% per-epoch time cost.
  • Tempo and softmax output approximation target GELU, LayerNorm and attention matrices, which are not the dominant activations in convolutional restoration backbones.
  • WTA-CRS samples matrix products for language-model fine-tuning.

Two restoration systems define the memory pain point:

  • EDVR stores multi-frame alignment offsets and attention maps.
  • MultiBoot VSR keeps its backbone in low resolution and trains on cropped patches. Its authors acknowledge that patch processing limits global context.

Research Program

The first stage is diagnostic. We train matched pairs of networks on DIV2K plus Flickr2K at 4× super-resolution, using SRResNet and an EDSR-style backbone. Each pair contains one network with a global residual skip over bicubic upsampling and one with direct mapping. We record $\rho_\ell(k)$ for several block sizes across layers and across training. We also record the per-layer cosine similarity between gradients computed from compressed and uncompressed activations, split by whether the output pixels lie in smooth or textured regions (textured regions identified by local gradient magnitude of the target). A ResNet-50 ImageNet run gives the reference profile under which DIVISION is known to work.

The second stage builds the adaptive policy. A short calibration pass on a few hundred batches estimates $\rho_\ell$. A small integer program then assigns each layer a block size from 2 to 8 and a detail-band width of 2, 4 or 8 bits under a fixed total memory. The assignment is refreshed a few times during training, because the spectra may drift.

The third stage spends the memory. At a fixed GPU memory cap and fixed wall-clock budget, we compare training on standard 128×128 crops against larger crops or more crops per batch made affordable by compression. We do this on super-resolution and on REDS video super-resolution with an EDVR-style model.

Baselines

The naive reference point is uncompressed training at the largest batch and crop that fit the memory cap. It sets the quality ceiling at that crop size and the memory floor that every method must beat.

The strongest existing competitor is DIVISION run unchanged (8×8 pooling, bfloat16 low band, 2-bit detail band). It is the only method in the file built on a frequency split. Its ImageNet result of 75.9% against 76.15% shows it is nearly lossless where its assumption holds, and it is reported to run about 1.3× faster than ActNN. No reported result was found on Lacuna for DIVISION on super-resolution or video restoration, so we establish that number ourselves. We give it the same data, iterations, optimizer and learning-rate search as our method.

Uniform 2-bit activation quantization tests whether the frequency split matters at all. Standard gradient checkpointing is a lossless memory baseline whose cost is measured in wall-clock time rather than quality.

For the video setting, the reference quality anchor is EDVR's reported 31.09 dB PSNR on REDS4 at 4× (EDVR). We will reproduce a smaller EDVR variant under our compute and report gaps relative to that reproduction rather than to the full published model.

Two ablations isolate the key ingredient:

  • Random allocation: the same total bit budget distributed randomly across layers, which tests whether measured spectra matter or merely having more bits somewhere.
  • Direct mapping: the same policy on the direct-mapping network, which tests whether the residual objective is what creates the problem.

Beating DIVISION means matching uncompressed PSNR within 0.05 dB on Urban100 and Manga109 at a memory footprint no more than 1.3× DIVISION's. DIVISION must also show a measurably larger loss on those detail-heavy sets than on Set5.

Why not just…?

Why not just use gradient checkpointing? Checkpointing is exact, so it is the honest lossless alternative. Its cost is recomputation, which the file reports at up to about 30% slowdown (Tempo). DIVISION is described as significantly faster than checkpointing. Our wall-clock-matched comparison settles the question. If checkpointing at equal time reaches equal quality, adaptive compression has no case on throughput grounds, and we will report that.

Why not just accumulate gradients over small batches? Accumulation simulates a larger batch, but it does nothing for the memory of a single large crop or a single multi-frame clip. The binding limit in restoration is often how much spatial and temporal context fits into one example, not the number of examples. Accumulation remains a baseline for the batch-size axis of the third stage. If larger batches, rather than larger crops, turn out to drive the gains, accumulation becomes a strong competitor and the case for compression narrows to speed.

Why not just keep cropping patches? This is the real threat. Larger crops may simply not help convolutional super-resolution, whose receptive field is local. The third stage tests exactly this at equal compute. If larger context gives no gain on single-image super-resolution, the claim retreats to video, where EDVR-style alignment and attention benefit from wider context. If it gives no gain there either, the contribution reduces to the diagnostic finding and to memory savings for the same training recipe. That finding would still be useful to anyone applying activation compression outside classification.

Why not just apply DIVISION as is? It may work. If $\rho_\ell$ stays small in residual networks and DIVISION loses under 0.05 dB on Urban100, the hypothesis is false. The adaptive policy would then be unnecessary, and we would report DIVISION's transfer to restoration as a positive result.

Proposed Experiments

The decisive check comes first and is cheap. Train SRResNet with a global residual skip and with direct mapping, each uncompressed, under DIVISION, and under the adaptive policy. Evaluate PSNR, SSIM and LPIPS on Set5, Set14, B100, Urban100 and Manga109, reporting peak activation memory and step time for every run.

The prediction is a specific interaction. DIVISION's PSNR loss should be larger for the residual network than for the direct-mapping network, larger on Urban100 and Manga109 than on Set5, and concentrated in textured-region gradient error in late layers. The adaptive policy should remove that loss at near-equal memory, and random allocation at the same budget should not.

The second experiment repeats the comparison with Focal Pixel Learning added. That loss deliberately amplifies tail pixels and should worsen any detail-band distortion.

The third experiment moves to REDS, training an EDVR-style model under a fixed memory cap and time budget with standard crops versus compression-enabled larger crops. We report REDS4 PSNR and SSIM against our uncompressed reproduction. TDD document deblurring from DocWaveDiff is an optional further check: there the residual stage targets thin text strokes, and its reported 26.28 dB PSNR serves as an external reference point.