Traditional super-resolution (SR) algorithms operate under a clean mathematical assumption: a high-resolution image is downsampled via a clean, predictable operation (like bicubic downsampling), and the model's job is simply to invert this process. However, as Saem Park et al. point out in Dual-Stage Super-Resolution for Edge Devices, real-world streaming video almost never conforms to this ideal. Instead, video transmitted over networks undergoes lossy compression via codecs like HEVC or AV1, which introduce severe spatial and temporal distortions.
When standard SR models encounter these compressed inputs, their performance degrades drastically. Benchmarks established by Dmitriy S. Vatolin et al. in SR+Codec: a Benchmark of Super-Resolution for Video Compression Bitrate Reduction demonstrate that models optimized for clean imagery fail to mitigate compression artifacts. Even worse, as Yinxiao Li et al. observe in COMISR: Compression-Informed Video Super-Resolution, naive upscaling architectures often amplify these distortions, mistaking blocky compression boundaries for genuine high-frequency details. As Liangbin Xie et al. demonstrate in VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution, even when models are trained on high-fidelity datasets, they encounter severe performance barriers and fail to recover structural details once the source video falls below a critical compression threshold.
To solve this, a new paradigm has emerged: codec-aware reconstruction, which explicitly integrates the mathematical and structural mechanics of video compression into deep learning architectures.
Why Compression Artifacts Break Standard Models
To understand why codec awareness is necessary, we must look at how video compression operates. Unlike stochastic sensor noise, compression noise is highly structured, geometric, and deterministic. As explained by Jian Yue et al. in A Global Appearance and Local Coding Distortion Based Fusion Framework for CNN Based Filtering in Video Coding, video codecs process frames using block-based partitioning, creating grid-like artifacts, blocking, and ringing.
Furthermore, video compression introduces an asymmetrical complexity between static and moving scenes. Tie Liu et al. note in Enhancing Quality for HEVC Compressed Videos that modern standards generate far more intricate spatial distortions than older formats like JPEG, meaning that standard image denoising models lack the representational capacity to reconstruct details lost during intra-coding. When models treat these complex distortions as generic noise—a flaw highlighted early on by Michael K. Ng et al. in A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video—they fail to preserve sharp edges. Boyang Wang et al. reinforce this in APISR: Anime Production Inspired Real-World Anime Super-Resolution, showing that simple JPEG-based noise simulation completely fails to capture the motion-vector and block-based distortions inherent to actual video streams.
In-Loop and Codec-Aware Training
The most direct way to build intuition about compression artifacts into a model is to integrate the codec directly into the training loop. By forcing the super-resolution network to learn from data that has been explicitly encoded and decoded at varying bitrates, the network learns the statistical patterns of lossy compression.
In Expanding Synthetic Real-World Degradations for Blind Video Super Resolution, Mehran Jeelani et al. demonstrate that shuffling a randomized pipeline of actual video codecs allows a model to cleanly differentiate between genuine high-frequency textures and blocky artifacts. Similarly, Yuxuan Jiang et al. utilize this strategy in RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content by exposing the network to AV1-encoded streams at multiple quantization levels.
This codec awareness allows the model to perform joint super-resolution and artifact reduction. As shown by Fan Zhang et al. in Video Compression Based on Spatio-Temporal Resolution Adaptation, training on codec-processed data enables deep networks to effectively "hide" encoding flaws like blocking and blurring when standard codecs hit a quality wall at low bitrates.
Exploiting Compressed Domain Priors
Beyond just training on compressed pixels, the most powerful realization in this field is that the video bitstream itself contains rich structural metadata that can guide reconstruction. When a video is compressed, the encoder makes explicit decisions about block sizes, motion vectors, and residual errors.
In Compressed Domain Prior-Guided Video Super-Resolution for Cloud Gaming Content, Qizhe Wang et al. exploit these "coding priors"—such as partition maps and residuals—to help the super-resolution network accurately distinguish between legitimate object boundaries and artificial block edges. Because the partition map explicitly tells the network where the codec drew its block boundaries, the model knows exactly where grid-like artifacts are likely to occur, preventing the accidental amplification of ringing errors.
[Compressed Bitstream] ──> Extract Metadata (Partition Maps, Residuals) ──┐
▼
[Degraded Video Frames] ──────────────────────────────────────────> [Codec-Aware SR Model] ──> [Clean HR Video]
This deep architectural awareness is also crucial for security. Jingcai Guo et al. demonstrate in On the Robustness of Neural-Enhanced Video Streaming against Adversarial Attacks that models lacking codec awareness are highly vulnerable to "Codec Hijacking" attacks, where adversarial perturbations are hidden inside quantization matrices. Only by incorporating codec-specific artifacts into the training regime can a model learn to ignore these malicious manipulations.
Architectural Specialization for Information Skewness
When dealing with compressed video, the fundamental nature of the data changes, requiring specialized neural architectures. For example, compression introduces an "information skewness" between spatial and temporal domains. In Hierarchical Separable Video Transformer for Snapshot Compressive Imaging, Ping Wang et al. observe that in compressed frames, spatial clues are often far more reliable than temporal ones due to motion-compensation artifacts. By separating spatial and temporal operations and avoiding early-stage temporal mixing, their architecture achieves superior reconstruction of fine textures while slashing computational overhead.
Other approaches shift the problem entirely to alternative representations. Qiang Zhu et al. introduce FCVSR: A Frequency-Aware Method for Compressed Video Super-Resolution, which models motion and refines features across multiple frequency subbands. Because compression artifacts like blocking manifest as specific high-frequency patterns, analyzing the video in the frequency domain makes it significantly easier to isolate and eliminate codec noise while preserving genuine spatial details.
Interestingly, this heavy conditioning can sometimes be turned into an advantage. The authors of TurboVSR: Fantastic Video Upscalers and Where to Find Them note that because a low-resolution compressed video provides such strong structural conditioning, generative models (like diffusion networks) become inherently more robust to compression artifacts than pure text-to-image or unconditioned generation tasks, allowing for highly aggressive spatial and temporal compression trade-offs.
Key Foundational Work
Enhancing Quality for HEVC Compressed Videos (2018) — contributed 2 concept elements to this direction.
A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video (2007) — contributed 1 concept elements to this direction.