Does the Loss Decide How Far a Restoration Network Looks?

Image restoration networks for super-resolution, demoiréing and destriping are almost always trained with the Charbonnier loss, a smoothed absolute-error penalty, $\sqrt{x^2+\epsilon^2}$, where $x$ is the per-pixel error and $\epsilon$ is a small constant. It is used because squared error blurs edges. Separately, a large body of work adds attention, oversized kernels or Fourier-domain filters to these networks. The goal is to escape the local receptive field limitation: a stack of small convolutions lets each output pixel depend on only a small neighbourhood of the input. Both lines of work treat the receptive field as a property of the architecture. We propose that it is also a property of the training loss.

The loss decides which pixels drive learning. Under squared error, the gradient at a pixel grows with that pixel's error, so the largest errors dominate. These sit at edges, repeated structures and other heavy-tailed regions, which are often fixable only with distant context. Charbonnier caps every pixel's gradient near unit magnitude. That shifts learning pressure toward the many small-error smooth pixels, which need almost no context. Our hypothesis is therefore that robust losses train networks, including networks with global modules, to use less long-range context than their architecture allows. If so, part of the reported value of global modules depends on the loss, and a loss that keeps its robust shape while restoring pressure on context-dependent pixels should recover it.

Core Claim

The amount of distant context a trained restoration network actually uses is jointly set by architecture and loss. Charbonnier and $L_1$ objectives shrink this used context relative to squared error. The cause is how they redistribute gradient across pixels, not their overall gradient scale. The shrinkage is largest for architectures with a global receptive field, so pairing them with point-wise robust losses wastes much of their reach.

The claim is falsifiable in both directions. If Charbonnier-trained networks show equal or larger used context than squared-error ones, the hypothesis is wrong and the field's default pairing is fine. That outcome would still be the first measurement of its kind.

Relation to Prior Work

The motivating tool is Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. It measures the effective receptive field as the input gradient of a central output unit, shows that this region is roughly Gaussian and grows only as the square root of depth, and finds that it expands during training. That analysis, as summarized in our file, is carried out under standard squared-error backpropagation. It does not ask whether the training objective changes how fast or how far the region expands. We reuse its measurement and make the loss the experimental variable.

The constraining evidence comes from the robust-loss side. Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks and its conference version Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution train a multi-level pyramid of $3\times3$ convolutions with Charbonnier loss and report sharper edges than squared error. Two restoration systems pair Charbonnier with explicit context mechanisms: Column-Spatial Correction Network for Remote Sensing Image Destriping uses column-wise and multi-scale spatial correction, and Improved multi-scale dynamic feature encoding network for image demoiréing uses region-level non-local blocks. None of the three measures whether the loss affects how much of that context the network uses.

A General and Adaptive Robust Loss Function. supplies a method we compare against. It places Charbonnier inside a family indexed by a shape parameter and learns robustness per pixel or per wavelet coefficient, reporting roughly 17% lower average error on KITTI depth than $L_1$. It adapts robustness to the error distribution. It does not condition robustness on whether an error is fixable from distant context, which is the axis we study.

The architectures we use as the "reach" variable are four. CCNet: Criss-Cross Attention for Semantic Segmentation uses row-and-column attention to reach the full image at lower cost than Non-Local Neural Networks. ParCNetV2: Oversized Kernel with Enhanced Attention uses 1D kernels twice the size of the feature map. Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening uses learnable Fourier-domain filters. These papers were trained with segmentation, classification, or standard reconstruction objectives. We take their modules unchanged and only vary the loss.

Research Program

We first build a controlled grid. We use one super-resolution backbone in the style of the Laplacian pyramid network, at fixed parameter budget. In it we swap a single context block among five options: none (plain local convolutions), extra depth, dilated convolutions, recurrent criss-cross attention, an oversized-kernel block, and an anisotropic spectral-gating block. We cross these with five losses: squared error, $L_1$, Charbonnier at $\epsilon \in \{10^{-3}, 10^{-2}, 10^{-1}\}$, and the adaptive robust loss.

For each trained model we measure used context in two ways. The first is the gradient-based effective receptive field of Luo et al. We summarise it by its radius and by the share of gradient mass lying beyond a fixed distance $R$ from the output pixel. The second is an intervention measure, which does not rely on gradients and so cannot be confounded by the loss shaping gradient statistics. We replace an annulus of input at distance $d$ with content from another image and record the change in the central output. Plotting this change against $d$ gives a "context-dependence curve". We also track both measures over training, since Luo et al. report that the receptive field grows during optimisation; the loss may change the rate of growth even if final sizes converge.

Second, we separate cause from confound. Charbonnier differs from squared error in two ways: overall gradient scale and the distribution of gradient across pixels. A magnitude-matched control rescales the Charbonnier gradient each step so its total norm equals that of squared error on the same batch. If used context still shrinks under this control, the redistribution is responsible. A pixel-stratified analysis labels test pixels as context-dependent or not. A pixel counts as context-dependent if a global-module model trained with squared error reduces its error much more than the plain-convolution model does. We then report errors separately on each class.

Third, if the effect is real, we test a corrective loss: Charbonnier with a context-tail weight. It keeps the robust shape everywhere but multiplies the loss on each pixel by a weight that grows with that pixel's error relative to a running per-image error scale, up to a cap. This partly restores pressure on the heavy tail without returning to squared error's averaging behaviour:

$$\mathcal{L}(x_i) = w_i \sqrt{x_i^2+\epsilon^2}, \quad w_i = \min\!\left(1+\lambda\,\frac{|x_i|}{s},\; w_{\max}\right)$$

Here $x_i$ is the error at pixel $i$. The scale $s$ is a running median absolute error, $\lambda$ sets how strongly the tail is emphasised, and $w_{\max}$ caps the weight. The weights are treated as constants (stop-gradient). If $\lambda=0$ the loss is plain Charbonnier, and as $w_{\max}$ grows the per-pixel gradient approaches the squared-error profile.

Baselines

The naive reference point is the plain local-convolution backbone trained with squared error. It shows what both blurring and short reach cost.

The strongest existing recipe is the Charbonnier-trained Laplacian pyramid network. Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks reports MS-LapSRN-D5R8 at 37.78 dB PSNR and 0.960 SSIM on Set5 under its 4× setting, with 222k parameters. We will reproduce this number first to validate our training pipeline. All other models then use the same DIV2K training data, the same parameter budget (±10%), the same iteration count, and the same learning-rate search of equal trials per configuration.

The strongest competing loss is the adaptive robust loss of A General and Adaptive Robust Loss Function., with its shape and scale learned per pixel. This is the obvious objection that robustness should simply be learned.

The strongest competing architectures are the criss-cross, oversized-kernel and spectral-gating blocks, each trained with Charbonnier as current practice would.

Two ablations isolate the key ingredient. The magnitude-matched Charbonnier tests redistribution versus scale. An $\epsilon$-annealing schedule, starting near squared error and ending near $L_1$, tests whether a trivial curriculum gives the same benefit as the context-tail weight.

Beating the field means the following. A global-module model with the corrective loss must beat the same model with Charbonnier on Urban100 and Manga109 PSNR, and must do so mainly on context-dependent pixels. It must also match or beat the adaptive robust loss and $\epsilon$-annealing at equal tuning budget. Finally, it must show a larger measured used-context share, which links the accuracy gain to the mechanism rather than to generic regularisation.

Why not just…?

Why not just use the adaptive robust loss and let robustness be learned? It is the real threat. Learned per-pixel robustness could, in principle, discover that heavy-tailed pixels deserve less robust treatment. However, its learning signal is the likelihood of the error distribution, which has no notion of whether an error could be fixed with distant context. It is a direct baseline in our grid. If it matches the corrective loss on context-dependent pixels and on used-context share, we will report that and drop the corrective loss. The measurement study would still stand.

Why not just tune or anneal $\epsilon$? A larger $\epsilon$ makes Charbonnier behave like squared error for more pixels and so restores tail pressure, at the cost of the blurring Charbonnier was adopted to avoid. Our $\epsilon$ sweep and annealing ablation test exactly this. The corrective loss is worthwhile only if it gives a better trade-off between edge-region and context-dependent-pixel error than any point on the $\epsilon$ curve.

Why not just add a bigger global module and ignore the loss? If used context is loss-limited, more architectural reach yields diminishing returns under Charbonnier. Prior evidence for diminishing returns from reach alone exists: Luo et al. show depth grows the receptive field only as the square root of the number of layers, and the demoiréing ablation in Improved multi-scale dynamic feature encoding network for image demoiréing attributes +3.19 dB to its dynamic and non-local encoding rather than to extra convolutions. The architecture-by-loss interaction in our grid settles this directly. Under our hypothesis, the gain from adding a global block is larger under squared error or the corrective loss than under Charbonnier.

Why not just use dilated convolutions? They grow the receptive field cheaply. However, Luo et al. find that the effective region stays Gaussian, and dilation introduces gridding artifacts at edges. Dilated convolution is one of the architecture conditions, so its interaction with the loss is measured rather than assumed.

Proposed Experiments

The primary benchmark is 4× super-resolution trained on DIV2K and evaluated on Set5, Set14, BSDS100, Urban100 and Manga109, reporting PSNR and SSIM. Urban100 and Manga109 are the decisive sets because their repeated architectural structure and line art make distant context informative. As a second domain we use LCDMoire demoiréing, where moiré patterns span wide frequency bands. The reference there is MDDM at 42.49 dB PSNR and 0.994 SSIM as reported in Improved multi-scale dynamic feature encoding network for image demoiréing. We will compare loss variants on its architecture rather than chase that number. For destriping, no reported numerical result was found on Lacuna for CSCNet, so it is only a qualitative transfer check on simulated data.

The first decisive check is the sign of the main effect. Within each global-module architecture, compare used-context share and the intervention curve between squared error and Charbonnier, with three seeds each. A consistent reduction under Charbonnier that survives the magnitude-matched control supports the claim. No reduction refutes it.

The second check is whether the accuracy cost concentrates where predicted. The Charbonnier-versus-corrective PSNR gap should appear on context-dependent pixels of Urban100 and Manga109 and be near zero on the remaining pixels.

The third check is head-to-head. At equal tuning budget, the corrective loss must beat both the adaptive robust loss and $\epsilon$-annealing on context-dependent-pixel error while not losing PSNR on edge-rich regions overall. If it fails this, the contribution reduces to the measurement study and the recommendation that the robust loss and the global module be chosen together. That recommendation is useful but more modest.