Overcoming Computational Overhead in Neural Video Codecs

455 concepts289 papers1 proposal

Traditional video codecs like H.264, HEVC, and VVC rely on hand-crafted, block-based algorithms optimized over decades to run on dedicated hardware blocks (ASICs). Because these operations map directly to silicon, consumer devices can decode high-resolution video instantly with minimal power consumption.

In contrast, Neural Video Codecs (NVCs) replace these rigid pipelines with deep neural networks, framing compression as an end-to-end optimization problem. While NVCs achieve superior rate-distortion performance—often outperforming traditional standards at lower bitrates—they introduce a massive computational burden. Running millions of floating-point operations (FLOPs) per frame creates a severe latency bottleneck, making real-time deployment on consumer and mobile hardware exceptionally difficult.

To make neural video compression practical, researchers must dismantle the specific architectural bottlenecks that slow down encoding and decoding.


The Asymmetry of Encoding and Decoding Latency

In video streaming, the computational budget is highly asymmetric. A video is encoded once (often offline) but decoded millions of times by end-users in real time. However, current NVC architectures push this asymmetry to extremes, introducing bottlenecks at both ends of the pipeline.

The Encoding Bottleneck: Online Optimization

To maximize compression efficiency, some modern codecs use "content adaptation," where the model dynamically adjusts its parameters to fit the unique spatial and temporal patterns of a specific video segment. While highly effective, this introduces massive computational overhead during encoding.

As Mehrdad Khani Shirkoohi et al. demonstrate in their work on Efficient Video Compression via Content-Adaptive Super-Resolution, a lightweight super-resolution network can decode video at a smooth 90 frames per second on consumer hardware. However, the encoding phase requires intensive fine-tuning that takes significantly longer than the actual duration of the video segment, rendering it unusable for live streaming.

Similarly, online feature optimization introduces severe delays. Chuanbo Tang et al. show in Offline and Online Optical Flow Enhancement for Deep Video Compression that performing iterative gradient descent on latent features during the encoding process yields substantial bitrate savings. However, this iterative feed-forward and backpropagation loop makes the encoder far too slow for real-time video processing.

The Decoding Bottleneck: Monolithic Generative Layers

Even when a model is fully trained and deployed for inference-only decoding, deep generative components remain a primary source of latency. In early end-to-end deep codecs, such as the framework proposed by Chao-Yuan Wu et al. in Video Compression Through Image Interpolation, the high cost of deep network inference and autoregressive entropy models made real-time playback impossible without specialized hardware acceleration.

Even when generative models are optimized—such as using distilled diffusion models to generate or restore frames in fewer steps—underlying structural bottlenecks persist. For example, Bojia Zi et al. observe in MiniMax-Remover: Taming Bad Noise Helps Video Object Removal that even when sampling steps are drastically reduced, the Variational Autoencoder (VAE) itself becomes the primary throughput bottleneck, limiting the overall speed of the video processing pipeline.


Strategies for Real-Time Viability

To bridge the gap between theoretical compression gains and real-time deployment, researchers employ three primary strategies: hardware-aware quantization, hybrid post-processing, and parameter-efficient adaptation.

Traditional Codec Pipeline:
[Bitstream] ──> [Hardware ASIC Decoder] ──> [Display] (Ultra-fast, low power)

Pure Neural Codec Pipeline:
[Bitstream] ──> [Heavy FP32 Neural Network] ──> [Display] (High latency, GPU-bound)

Hybrid / Optimized Pipeline:
[Bitstream] ──> [Hardware ASIC Decoder] ──> [Lightweight INT8 CNN] ──> [Display]

1. Hardware-Aware Quantization

Most deep learning models are designed using 32-bit floating-point precision (FP32), which is poorly suited for the power and hardware constraints of mobile processors. Translating these models into hardware-friendly formats is essential for mobile deployment.

In MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device, Amir Habibian et al. achieved a breakthrough by developing the first neural video codec capable of decoding 1080p video at over 30 frames per second on a standard mobile device. They accomplished this by pairing a sophisticated 8-bit quantization (INT8) strategy with hardware-aware, block-based motion compensation, reducing computational complexity tenfold while maintaining high compression efficiency.

2. Hybrid Post-Processing Loops

Rather than replacing the entire video codec with a neural network, a highly effective alternative is to keep the mature, hardware-accelerated traditional codec intact and use a lightweight neural network purely to clean up compression artifacts.

  • In-Loop Filtering: Feng Wu et al. designed VRCNN in A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding to operate directly within the HEVC decoding loop. By using variable filter sizes and residue learning, the authors significantly reduced visual artifacts and bitrates while maintaining a low memory footprint.
  • Segment-Specific Arrays: To avoid the massive storage and memory footprint of a single monolithic network, Everett Fall proposed an alternative approach in Dynamically Expanded CNN Array for Video Coding. Instead of a large model, the system uses an array of ultra-small, specialized CNNs that are dynamically swapped and fine-tuned for short segments of a video, surgically removing artifacts with minimal computational overhead.

3. Parameter-Efficient Adaptation

For content-adaptive codecs that must update their weights to match new video distributions, updating the entire model during encoding is computationally prohibitive.

To solve this, Zhenghao Chen et al. introduced a method called GPU (Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression). Instead of retraining the entire codec, the system uses lightweight "adapters" to update less than 8% of the model's parameters across an entire Group of Pictures (GoP). This targeted update strategy allows the codec to adapt to entirely new content types on the fly, capturing significant bitrate savings without crippling the encoder's throughput.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 293 papers
  2. Version 1 · 2026-10-04 · 498 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_real_time_inference_constraints_computat_b8a49390,
  title = {real-time inference constraints (computational decoding overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-inference-constraints-computational-decoding-overhead/rdr_b8a4939093dd406bad88b53d17f64a59},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_real_time_inference_constraints_computat_b8a49390_v2,
  title = {real-time inference constraints (computational decoding overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-inference-constraints-computational-decoding-overhead/rdr_b8a4939093dd406bad88b53d17f64a59/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake