CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery

Tung-I Chen, Lingdong Wang, Subhransu Maji, Ramesh K. SitaramanCVPR 2026 Findings 2026 · OpenReview

TL;DR

CATRF introduces a practical, "standard-codec-in-the-loop" training framework for plane-factorized radiance fields (like TriPlanes). By using a straight-through estimator (STE) to backpropagate gradients through real, non-differentiable codecs (JPEG, HEVC, AV1), it allows neural features to adapt directly to real-world compression distortions. This achieves up to a 63.3% BD-rate reduction over traditional methods without requiring any custom neural decoders on the client side.

What Problem Does This Paper Address?

Streaming volumetric 3D content (like NeRFs or 3D Gaussian Splatting) requires massive bandwidth. To make streaming viable, researchers often factorize 3D scenes into 2D feature planes that can be compressed using standard image or video codecs.

However, existing approaches fall into two flawed paradigms:

  1. Codec-Agnostic (CA) approaches: Radiance fields are trained normally and compressed afterwards. Because the features never "see" codec distortions (like quantization or block artifacts) during training, rendering quality degrades severely at low bitrates.
  2. Learned-Codec-in-the-Loop (LCL) approaches: These integrate a differentiable, neural network-based codec into the training loop. While this improves the rate-distortion trade-off, it forces the client to download custom neural decoder weights and run heavy machine learning inference just to unpack the 3D scene.
Overview of codec-integrated NeRF compression pipelines

CATRF bridges this gap by introducing Standard-Codec-in-the-Loop (SCL) optimization, allowing features to adapt to real codec artifacts while maintaining a completely neural-free, standard-compliant decoding process on the client side.

Methods

The core architecture of CATRF relies on a standard TriPlane radiance field representation, which factorizes a 3D scene into three orthogonal 2D feature planes and an explicit 3D density grid.

During each training step, CATRF executes a full codec round trip:

  1. Quantization & Packing: Multi-channel floating-point feature planes are quantized into an 8-bit normalized range $[0, 1]$ and packed into 2D canvases. The authors found that stacking channels into a single monochrome canvas (FlattenGray) yielded the best results by avoiding cross-channel artifact mixing.
  2. Codec Round Trip: The packed canvases are passed through an actual, off-the-shelf, non-differentiable codec (such as JPEG for static scenes, or VP9/HEVC/AV1 for dynamic videos).
  3. Unpacking & Rendering: The decoded canvases are unpacked, dequantized, and passed to a lightweight neural renderer to compute the final image and reconstruction loss.
Figure 2. Illustration of the encode–decode codec round trip. In a streaming setting, the client receives and renders the decoded NeRF model reconstructed from the encoded bitstream.

To overcome the non-differentiable nature of standard codecs, CATRF uses a Straight-Through Estimator (STE). During the forward pass, the actual decoded features are used to render the scene, but during the backward pass, the gradients bypass the codec entirely and flow directly into the pre-codec features.

Additionally, because running a video codec on every single training minibatch is computationally prohibitive, the authors introduce a caching mechanism that reuses decoded planes across multiple training steps, refreshing only when a set number of steps pass or the frame drifts too far.

Datasets

The authors evaluated CATRF across both static and dynamic benchmarks:

  • Static Scenes: NeRF Synthetic (8 objects) and Tanks and Temples (highly detailed real-world scenes).
  • Dynamic Scenes: Neural 3D Video (N3D) and the NHR Dataset, both representing multi-view dynamic human performances.

Key Insights and Ideas

  • Real Codecs Beat Differentiable Proxies: Instead of training against a "differentiable proxy" that merely approximates a codec, training against the actual black-box codec via an STE forces the network to find clever, compression-resilient feature representations.
  • No Rate Loss Required: Traditional learned compression requires balancing a reconstruction loss with an explicit rate loss (to keep file sizes small). CATRF completely omits a rate loss. Because standard codecs have highly optimized, mature rate-control algorithms, simply minimizing distortion at a fixed codec configuration implicitly drives the features to the most optimal point on the rate-distortion curve.
  • Isolate the Channels: Packing features into a single grayscale canvas (FlattenGray) outperforms RGB packing layouts. Standard video codecs heavily compress chroma channels; keeping features separated prevents artifacts in one channel from bleeding into and corrupting adjacent feature semantics.

Key Results

CATRF demonstrates significant improvements in both compression efficiency and runtime performance:

  • Rate-Distortion Performance: On the dynamic N3D dataset, CATRF achieved massive Bjontegaard Delta (BD)-rate reductions over the codec-agnostic baseline TeTriRF, saving 40.5% with AV1, 51.8% with VP9, and 63.3% with HEVC.
  • Outperforming 3D Gaussian Splatting (3DGS): Compared to state-of-the-art compressed 3DGS streaming methods like GIFStream, CATRF-AV1 achieved a 39% BD-rate reduction, delivering comparable or better visual quality at substantially lower bitrates.
Rate-distortion curves on dynamic datasets
  • Decoding Speed: Because CATRF relies entirely on hardware-accelerated standard video decoding rather than neural network decoders, it achieves an exceptional client-side decoding throughput of 138 FPS (using HEVC), compared to just 2.94 FPS for GIFStream and 0.35 FPS for HAC++.

Limitations and Future Work

While CATRF achieves blazing-fast decoding speeds, its end-to-end client throughput is still bottlenecked by the rendering speed of the underlying NeRF backbone (averaging around 5.86 FPS in their prototype). The authors note that integrating their standard-codec-in-the-loop framework with faster hybrid backbones or more efficient neural renderers is a promising direction for achieving true real-time, low-bitrate volumetric streaming.

Summary updated

Abstract

Volumetric media promises next-generation content delivery applications, but its bandwidth demand remains a key bottleneck. Implicit and hybrid volumetric representations reduce model sizes, yet still require careful coding to reach 2D video-like bitrates. We present CATRF, a standard-codec-in-the-loop compression framework for plane-factorized radiance fields. During training, we quantize and pack 2D feature planes into codec-friendly canvases, run a standard codec round trip (JPEG/VP9/HEVC/AV1), then unpack and dequantize the decoded features before volume rendering. We use a straight-through estimator (STE) to insert the non-differentiable, standard codec pipeline into the training loop, allowing radiance-field features to adapt directly to the real, client-side codec distortions without introducing any learned codec parameters. On both static and dynamic benchmarks, CATRF consistently achieves a better rate-distortion trade-off over codec-agnostic and learned-codec-in-the-loop baselines, and also outperforms recent compressed 3DGS methods in both compression efficiency and decoding speed. These results highlight a practical path toward low-bitrate, compression-resilient volumetric representations for free-viewpoint video streaming.
BibTeX
@inproceedings{chen2026catrf,
  author = {Tung-I Chen and Lingdong Wang and Subhransu Maji and Ramesh K. Sitaraman},
  title = {{CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=GH8zBgrcSV},
}
Figure