A super-resolution model trained to sharpen JPEG images tends to fail on video frames compressed with HEVC, and it fails worse on images from a neural codec. Super-resolution means recovering a high-resolution image from a small, degraded one. One plausible reason for these failures is what the model learns about frequencies. Every codec damages an image's frequency content in its own recognizable way. It adds energy at some frequencies, such as the 8×8 block-grid lines of JPEG or ringing near edges, and removes energy at others, such as quantized fine texture. A model trained with ordinary pixel losses has no reason to learn this distinction as a general rule. It can instead memorize one codec's signature. Frequency-domain losses do not help by default either. Losses that penalize spectral mismatch without regard to where the error came from treat a leftover blocking line and a missing texture the same way. Both are high-frequency errors, and in compressed images the artifacts are themselves sharp, high-frequency structure.
We propose a signed spectral loss. At training time the ground truth is available, so for every training pair we can compute, frequency by frequency and window by window, where the degradation added energy and where it erased energy. We then supervise the two cases differently. In added bands, the model is penalized for leaving excess energy behind. In erased bands, it is penalized for failing to restore energy, with a controllable, gentler weight so that a deterministic model is not pushed to hallucinate detail that no longer exists. The loss needs no codec label, no bitstream metadata and no change to the architecture at inference. The description "added versus erased energy" applies equally to rain streaks (added), blur (erased) and block artifacts (both). Our central bet is therefore that this sign structure, rather than frequency supervision as such, is what lets one restorer transfer across codecs and to non-compression tasks.
Core Claim
Our claim is that cross-codec failure is partly a failure of supervision, and that an unsigned frequency loss cannot fix it. Concretely, a restoration network trained with pixel loss plus a signed, input-referenced spectral loss will lose less accuracy on held-out codecs than the same network trained with pixel loss alone or with pixel loss plus an unsigned frequency loss. We also predict that the size of this gap tracks a measurable quantity: how much of the added-by-the-codec spectral energy survives in the output on unseen codecs.
Let $A_x$, $A_y$ and $A_p$ be the local Fourier magnitudes, computed with a windowed short-time transform, of the bicubically upsampled degraded input, the ground truth and the prediction, each indexed by window and frequency $(u,v)$. Let $\bar{\cdot}$ denote per-band normalization. The normalization follows Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution, which showed that low-frequency coefficients otherwise dominate the gradient. We define the added mask $m^{+} = \max(0, \bar A_x - \bar A_y)$ and the erased mask $m^{-} = \max(0, \bar A_y - \bar A_x)$. The loss is
$$ \mathcal{L}_{\pm} = \sum_{u,v} m^{+}\,\max(0, \bar A_p - \bar A_y) \;+\; \lambda \sum_{u,v} m^{-}\,\max(0, \bar A_y - \bar A_p) \;+\; \mu\,\mathcal{L}_{\text{phase}}, $$
where $\lambda \le 1$ sets how hard the model is pushed to restore erased energy and $\mathcal{L}_{\text{phase}}$ is a plain phase-alignment term weighted by $\mu$. The two one-sided terms are what distinguish this loss from its predecessors. The first penalizes only excess energy, and only where the codec injected energy. The second penalizes only missing energy, and only where the codec removed it.
Relation to Prior Work
Three kinds of prior work shape this proposal.
The first kind motivates the gap. SR+Codec and Sensitivity Decouple Learning for Image Compression Artifacts Reduction document that restorers are tightly coupled to their training codec. Both study that coupling but propose no supervision-level fix. The frequency-aware compressed video methods are each tied to a single standard. FCVSR evaluates only HEVC, FTVSR only H.264, and the reference-frame-proposal enhancer only HEVC. None of them asks whether their frequency machinery transfers.
The second kind constrains the design, or competes with it most directly. Focal Frequency Loss weights frequencies by the model's current error. That weighting is symmetric and knows nothing about the input, so it cannot tell a surviving artifact from missing texture; our weights are fixed by the input's degradation and carry a sign. The FFT amplitude-and-phase loss of the reference-frame-proposal work is likewise unsigned. FCVSR's wavelet contrastive loss is the closest relative. It pushes outputs away from the upsampled compressed input, which implicitly uses the input as a reference. However, it pushes away from the input in every band, including bands where the input was already correct. We push only where the codec added energy and pull only where it erased energy.
MCLIR builds task-agnostic negative samples from an average of the model's own past weights and compares them in VGG feature space. Its authors explicitly leave JPEG artifact removal unexplored, and our loss is designed to fill exactly that hole. Adversarial Sobolev Alignment shows that spectrally shaped objectives separate real detail from generated artifacts in camera super-resolution. It is expensive, however, and depends on a tuned smoothness index, which is why we choose a closed-form, training-time-only loss.
The third kind supplies the methods and benchmarks. UCIP provides the cross-codec benchmark (UCSR: six codecs, 23 configurations, with WebP and ELIC held out) and a strong spatial-only universal backbone that uses dynamic prompts. SFNet provides a frequency-selective architecture validated on five non-compression tasks but trained with spatial losses, so it serves as a natural host network for testing whether the loss adds anything beyond frequency-aware architecture. Finally, MC-TTDG shows that coarse feature-statistic matching erases high-frequency texture. That result warns us against enforcing codec invariance through feature statistics, and it is why we do not.
Research Program
The work has three parts.
The first part is diagnostic. We train a pixel-loss restorer on one codec family and apply it to others. For each output we measure the residual added energy: the portion of $m^{+}$ still present in $A_p$. We also measure the residual erased deficit: the portion of $m^{-}$ still missing. If our explanation is right, cross-codec PSNR loss should correlate more strongly with residual added energy than with the erased deficit. That would mean models fail to transfer mainly because they do not recognize a new codec's artifacts as artifacts. If the correlation runs the other way, the added-band term is not the useful part, and we would reweight the loss accordingly or drop that term.
The second part is the loss itself. We implement $\mathcal{L}_{\pm}$ with windowed transforms at two window sizes. Blocking is local and grid-aligned, while ringing and blur are broader, so one window size is unlikely to capture both. We then add the loss, unchanged, to three host networks: a UCIP-style prompted backbone, SFNet, and, for video, a BasicVSR++-class recurrent model. The added cost is one short-time transform per image per step, applied to three images: the input, the ground truth and the prediction.
The third part tests cross-task transfer. We add the same loss to MCLIR's task suite (super-resolution, deraining, dehazing and deblurring). There the sign structure is meaningful in a different way: rain is almost purely added energy, while blur is almost purely erased energy. If the sign decomposition is a general description of degradation, the gains should appear on those tasks too, and the ratio of added to erased energy should predict which term matters more for each task.
Baselines
The naive reference point is each host network trained with pixel L1 alone, on identical data, crops, schedule and iteration count. Every other number is reported as a difference from this model. Under leave-one-codec-out evaluation it establishes how large the transfer gap actually is.
The strongest existing cross-codec method is UCIP. On UCSR it is reported to improve PSNR by 0.45 dB over PromptIR with 11.4M parameters against 35.7M. The file gives only this relative number, and no absolute UCSR PSNR for UCIP was found on Lacuna. We will retrain UCIP from the released configuration on our codec splits, and add our loss to the identical model with the same data and step budget. Beating it means a consistent gain on held-out codecs (HiFiC, ELIC, WebP) larger than the variation across three random seeds, without losing more than 0.05 dB on codecs seen in training.
For video, the strongest frequency-aware competitor is FCVSR, which reports 25.20 dB PSNR on REDS4 under HEVC at QP 37. Its wavelet contrastive loss is the prior loss closest to ours. We will retrain FCVSR twice on H.264-only data: once with its own contrastive loss and once with that loss replaced by ours. Both versions will then be tested on REDS4 under HEVC at QP 37. Beating FCVSR means our variant closes more of the gap to FCVSR's HEVC-trained 25.20 dB than the contrastive variant does.
The ablation that isolates our key ingredient is an unsigned version of the same loss. It uses the same windows, normalization and input-referenced weight $|\bar A_x - \bar A_y|$, but applies symmetric absolute error. Alongside it we run Focal Frequency Loss and the FFT amplitude-phase loss, each tuned over the same grid of loss weights as ours, typically five values. If the unsigned ablation matches the signed loss, the sign is not what matters and our central claim fails.
Why not just…?
Why not just add Focal Frequency Loss to UCIP? This is the most serious threat. FFL already moves gradient toward the frequencies the model gets wrong, and on a mixed-codec training set that might be enough. The difference is only visible on held-out codecs. On an unseen codec, the frequencies where error is currently largest can be exactly the artifact bands, and a symmetric loss is equally content to reduce that error by adding or by removing energy there. Nothing in the FFL paper evaluates on compressed inputs, so the file offers no evidence either way. The head-to-head leave-one-codec-out comparison, run with matched tuning, settles the question directly. If FFL is within seed noise of our loss, we will report that and treat the signed formulation as unnecessary.
Why not just train on more codecs and randomized degradation pipelines? Shuffled degradation pipelines help models tell texture from compression boundaries (Expanding Synthetic Real-World Degradations for Blind Video Super Resolution). UCIP's six-codec training is the strongest form of this idea in the file. Data diversity and our loss are complementary rather than competing. The fair test is whether the loss still helps when the training mixture is already broad. We will therefore run the leave-one-out study at two diversity levels, three training codecs and five training codecs. If the benefit disappears at five, the honest conclusion is that our loss substitutes for data diversity, which still has value when codecs are expensive to simulate, rather than adding to it.
Why not just condition the network on codec or quality information? CAVSR's quality-aware feature modulation (Compression-Aware Video Super-Resolution) and UCIP's prompts both tell the network what kind of degradation it faces. Explicit conditioning has two weaknesses here: it depends on labels or metadata that are missing when a stream has been transcoded or stripped, and it does not help on a codec the conditioner has never seen. Our loss works only at training time and leaves inference blind. Because UCIP is one of our host networks, the experiment measures whether the loss adds to conditioning, not whether it replaces it.
Why not just use FCVSR's wavelet contrastive loss? It already pushes outputs away from the compressed input. However, it does so in every band, which also penalizes staying close to the input where the input was correct, usually in the low frequencies. This matters more when the codec used at test time differs from the training codec. The direct swap experiment described under Baselines answers the question.
Proposed Experiments
The primary experiment is a leave-one-codec-out study on UCIP's UCSR benchmark. We train on JPEG, HEVC and VVC, and test on ChengPSNR, ChengSSIM and HiFiC, plus WebP and ELIC. A second split trains on the learned codecs and tests on the traditional ones. We report PSNR and SSIM, LPIPS for HiFiC, and the two diagnostic quantities, residual added energy and residual erased deficit.
The video experiment trains on Vimeo-90k and REDS compressed with H.264 at CRF 15, 25 and 35, the protocol of FTVSR, which reports 28.05 dB on REDS4 at CRF 25. We check this in-distribution result for no regression, then evaluate zero-shot on REDS4 under HEVC at QP 37 against FCVSR's reported 25.20 dB. Because extreme compression erases detail that cannot be recovered, as COMISR documents, we sweep $\lambda$ at QP 37. We expect a smaller $\lambda$ to trade a little PSNR for less invented texture, measured by LPIPS and by the log-spectral distance restricted to the erased bands.
The cross-task experiment adds the loss to MCLIR's hosts on its benchmarks: EDSR and SwinIR for ×4 super-resolution, where MCLIR reports gains of 0.07 and 0.09 dB; IDT for deraining, where MCLIR reports 0.26 dB on average and 0.7 dB on SPA-Data; and FFANet on RESIDE indoor, where MCLIR reports 3.41 dB. The loss is also combined with MCLIR, to test whether the two stack. It is also added to SFNet on SOTS-Outdoor and GoPro.
Three results would support the claim together. First, the signed loss must beat the unsigned ablation and FFL on held-out codecs. Second, residual added energy on unseen codecs must fall more than the erased deficit. Third, the per-task gain must follow the task's added-to-erased energy ratio. The claim fails if the unsigned ablation matches the signed loss, or if the gain disappears once the training mixture spans five codecs.
References
- Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution
- SR+Codec: a Benchmark of Super-Resolution for Video Compression Bitrate Reduction
- Sensitivity Decouple Learning for Image Compression Artifacts Reduction
- FCVSR: A Frequency-Aware Method for Compressed Video Super-Resolution
- Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution
- Boosting the Performance of Video Compression Artifact Reduction With Reference Frame Proposals and Frequency Domain Information
- Focal Frequency Loss for Generative Models
- Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration
- Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution
- UCIP: A Universal Framework for Compressed Image Super-Resolution Using Dynamic Prompt
- Selective Frequency Network for Image Restoration
- Test-time Domain Generalization for Image Super-resolution
- Expanding Synthetic Real-World Degradations for Blind Video Super Resolution
- Compression-Aware Video Super-Resolution
- COMISR: Compression-Informed Video Super-Resolution