Does a Convergence Guarantee Make Learned Discretizations Transfer?

When a variational image model (an image restored by minimizing an energy that balances fitting the noisy data against a smoothness penalty) is computed on a pixel grid, the discrete derivative filters are usually hand-designed. Recent work learns those filters from data instead. It imposes linear constraints on the filters so that, as pixels shrink, the learned discrete energy still converges in a precise sense to the same continuous energy. This proposal asks whether that mathematical constraint has a practical payoff that nobody has measured: robustness when the filters are trained on synthetic data and deployed on real images, or trained at one resolution and deployed at another. The intuition is simple. If the continuum limit is fixed by the constraints, the training data can only shape the small, resolution-dependent correction to it. A constrained model therefore has little room to absorb the quirks of a simulator.

The opportunity is twofold. Convergence proofs are usually treated as a certificate of mathematical hygiene, not as a predictor of generalization. The synthetic-data literature has meanwhile measured exchange rates between synthetic and real samples only for deep trackers, never for energy-based solvers. A second practical question comes along with the first. Learned wide filters cost up to twelve times more than compact ones. If the constraints are what buy transfer, we can ask how much of the filter support can be pruned before that benefit disappears.

Core Claim

The key tool is Γ-convergence, a notion of convergence for energy functionals rather than for their values at fixed points. It requires two things. First, no sequence of discrete states can achieve asymptotically lower energy than its continuum limit. Second, every continuum state can be approximated by discrete states whose energy does not exceed it in the limit. Together with compactness, this guarantees that discrete minimizers converge to continuum minimizers.

Learned Discretization Schemes for the Second-Order Total Generalized Variation learns interpolation and derivative filters for second-order Total Generalized Variation, written TGV², a smoothness penalty that allows piecewise-linear ramps without the staircase artifacts of plain Total Variation. The filters are fit by bilevel training, an outer loop that tunes filters so the inner energy minimization reproduces clean images. The paper proves Γ-convergence to continuous TGV² under sum-to-one-type consistency constraints on the filters.

Our claim is that these consistency constraints act as an inductive bias that measurably shrinks two gaps. The first is the synthetic-to-real gap: filters trained on procedurally generated ramp images and tested on natural photographs. The second is the resolution gap: filters trained at one grid spacing and tested at another. On both, we predict that constrained learned filters lose substantially less than unconstrained learned filters of the same support, even if unconstrained filters win in-distribution. We further claim that this benefit survives aggressive pruning of the filter support, provided the constraints are enforced on the pruned support.

Research Program

The discrete model follows Bogensperger et al. A restored image $u_h$ on a grid of spacing $h$ minimizes

$$E_h(u) = \min_w \; \alpha_1 \|L u - w\|_1 + \alpha_0 \|K w\|_1 + \tfrac{1}{2}\|u - g\|_2^2 ,$$

where $g$ is the noisy image, $w$ is an auxiliary vector field, $L$ and $K$ are learned first- and second-order filter banks, and $\alpha_0, \alpha_1$ are weights. We train three families of filters with the same memory-efficient primal-dual bilevel solver. The first family obeys the paper's consistency constraints, which are what the Γ-convergence proof needs. The second is identical but unconstrained. The third keeps the constraints but penalizes the number of active filter taps with a group-sparsity term, so that the solver discovers which of the 7×7 positions carry weight. For the pruned family we must check that the Γ-convergence argument still applies. The expected route is that the proof needs only the consistency identities and finite support, both of which pruning preserves. If a gap appears, we will report it rather than assume it.

To measure synthetic reliance, we adapt the equivalency ratio of On the Equivalency, Substitutability, and Flexibility of Synthetic Data. That paper measured roughly 30 synthetic tracking frames per real frame and found 60 to 80 percent of real data could be replaced after synthetic pre-training. Its evidence comes entirely from neural trackers and says nothing about energy-based solvers. In our setting, let $P_{\text{real}}(n)$ be the test PSNR on held-out natural images of filters trained on $n$ real images. Let $P_{\text{syn}}(m)$ be the same quantity for filters trained on $m$ synthetic ramp images drawn from an extended version of the procedural generator behind the 32-image ramp set. The ratio at which synthetic training matches $n$ real images is

$$\rho(n) = \frac{\min\{m : P_{\text{syn}}(m) \ge P_{\text{real}}(n)\}}{n}.$$

If no $m$ reaches parity, $\rho$ is infinite and we report the residual gap instead. We also trace substitutability curves: synthetic pre-training followed by fine-tuning on a fraction of the real images.

Gaussian denoising is a regime where real clean images are cheap. That is precisely why we use it: the real-data oracle is available, so the gaps can be measured exactly. The resulting ratios are a controlled measurement of how much the constraints protect against simulator bias. They are not an argument that practitioners should prefer synthetic data for denoising.

The resolution test is the most direct consequence of Γ-convergence. We train on images at one downsampling factor of DIV2K and evaluate at coarser and finer factors, with noise level matched per pixel. The theory says a constrained scheme should approach the same continuum problem at every spacing. An unconstrained scheme has no such anchor.

Baselines

The do-nothing reference is handcrafted TGV with standard compact finite differences, tuned only in its two weights on the same training set. On BSDS500 with 10% noise, Bogensperger et al. report 26.56 dB PSNR for it and 26.52 dB for plain Total Variation. On the synthetic ramps they report 40.36 dB for handcrafted TGV.

The strongest existing method is their constrained learned 7×7 scheme, with a reported 27.18 dB on BSDS500 at 10% noise and 43.09 dB on the ramps. No reported result was found on Lacuna for BSDS500 at 5% noise, and none for PSNR on DIV2K from any cited paper, so on DIV2K we establish every number ourselves. We first reproduce the 27.18 dB figure within 0.1 dB on the same 64-image protocol before running any transfer experiment. Otherwise differences could come from our implementation.

The central ablation is the unconstrained learned scheme with identical support, solver, iteration count, initialization, and outer-loop budget. It isolates the constraints as the only difference. A second ablation replaces the hard constraints with a soft quadratic penalty toward the constraint set, swept over strength. It asks whether any shrinkage toward consistency helps, or whether exact consistency matters.

All learned variants receive the same number of outer iterations and the same hyperparameter search. They are evaluated on identical noise realizations. Beating the baseline has a specific meaning. The constrained scheme's synthetic-to-real drop, and its cross-resolution drop, must be smaller than the unconstrained scheme's by more than the seed-to-seed spread over at least five seeds. For pruning, beating means recovering at least most of the 0.62 dB gain over handcrafted TGV at a small fraction of the 7×7 cost, measured in wall-clock time with the same solver.

Why not just…?

Why not just train on real images, since they are free for denoising? For this task, one should. Our claim is about what the constraints protect against, and the denoising testbed is chosen because it permits an exact real-data reference. The practical relevance lies in settings with scarce ground truth, such as the non-trivial inverse problems Bogensperger et al. leave open. Showing the effect there is a follow-up that this study would justify or rule out.

Why not just use unconstrained filters, which have more freedom and should fit better? This is a real threat. Unconstrained filters may win on in-distribution real data, and if they also transfer as well, our claim is false. The synthetic-to-real and cross-resolution comparisons are designed to settle exactly this, and we will report in-distribution wins for unconstrained filters honestly.

Why not just use smaller handcrafted stencils to cut cost? The 3×3 or wider handcrafted isotropic stencils in the spirit of Finite-differences discretizations of the mumford-shah functional are cost-matched alternatives. That work shows wider handcrafted interactions already remove grid bias. We include cost-matched handcrafted stencils in the cost-quality curve, and pruning counts as useful only if it lies strictly above them.

Why not just prune compute with adaptive coordinate updates instead of pruning filters? Multiresolution Adaptive Block-Coordinate Forward-Backward for Image Reconstruction reaches the lowest objective within one second on over 65% of 100 DIV2K instances by choosing wavelet blocks with large proposed updates. That paper is purely discrete, uses only wavelet sparsity, and is validated only on synthetic Gaussian degradations. It reduces iterations, whereas we reduce per-iteration cost, so the two are complementary. We do not claim to replace it.

Proposed Experiments

The first experiment trains on ramps and tests on BSDS500 at 5% and 10% noise, and the reverse. It compares constrained, unconstrained, soft-penalty, and pruned filters. The decisive check is the synthetic-to-real PSNR drop per variant. The claim fails if the constrained drop is not smaller than the unconstrained drop beyond seed variance.

The second experiment computes the equivalency ratio and substitutability curves above for constrained and unconstrained filters. The prediction is a lower, finite ratio for the constrained family. An infinite ratio for both would show that ramp content simply lacks natural-image information, whatever the constraints.

The third experiment is the DIV2K resolution sweep. The prediction is that constrained filters' PSNR curve stays close to one retrained at each resolution, while unconstrained filters degrade away from the training spacing. It also tests whether the minimizers visibly converge as spacing shrinks, for example through distance between upsampled solutions at successive resolutions.

The fourth experiment traces cost against PSNR for pruned constrained filters, from 2×2 to 7×7 support. It includes handcrafted cost-matched stencils and reports wall-clock time and tap count alongside PSNR. It also reports whether pruned filters keep their transfer advantage, which tells us whether constraints alone or constraints plus width drive robustness.

If the constraints do shrink the transfer gaps, this becomes the first evidence that a Γ-convergence guarantee buys something measurable in deployment. If they do not, we learn that convergence guarantees and robustness to distribution shift are separate properties. That would matter to anyone citing such proofs as a reason to trust learned discretizations on real data.