TL;DR
CATRF introduces a practical, "standard-codec-in-the-loop" training framework for plane-factorized radiance fields (like TriPlanes). By using a straight-through estimator (STE) to backpropagate gradients through real, non-differentiable codecs (JPEG, HEVC, AV1), it allows neural features to adapt directly to real-world compression distortions. This achieves up to a 63.3% BD-rate reduction over traditional methods without requiring any custom neural decoders on the client side.
What Problem Does This Paper Address?
Streaming volumetric 3D content (like NeRFs or 3D Gaussian Splatting) requires massive bandwidth. To make streaming viable, researchers often factorize 3D scenes into 2D feature planes that can be compressed using standard image or video codecs.
However, existing approaches fall into two flawed paradigms:
- Codec-Agnostic (CA) approaches: Radiance fields are trained normally and compressed afterwards. Because the features never "see" codec distortions (like quantization or block artifacts) during training, rendering quality degrades severely at low bitrates.
- Learned-Codec-in-the-Loop (LCL) approaches: These integrate a differentiable, neural network-based codec into the training loop. While this improves the rate-distortion trade-off, it forces the client to download custom neural decoder weights and run heavy machine learning inference just to unpack the 3D scene.
CATRF bridges this gap by introducing Standard-Codec-in-the-Loop (SCL) optimization, allowing features to adapt to real codec artifacts while maintaining a completely neural-free, standard-compliant decoding process on the client side.
Methods
The core architecture of CATRF relies on a standard TriPlane radiance field representation, which factorizes a 3D scene into three orthogonal 2D feature planes and an explicit 3D density grid.
During each training step, CATRF executes a full codec round trip:
- Quantization & Packing: Multi-channel floating-point feature planes are quantized into an 8-bit normalized range $[0, 1]$ and packed into 2D canvases. The authors found that stacking channels into a single monochrome canvas (
FlattenGray) yielded the best results by avoiding cross-channel artifact mixing. - Codec Round Trip: The packed canvases are passed through an actual, off-the-shelf, non-differentiable codec (such as JPEG for static scenes, or VP9/HEVC/AV1 for dynamic videos).
- Unpacking & Rendering: The decoded canvases are unpacked, dequantized, and passed to a lightweight neural renderer to compute the final image and reconstruction loss.
To overcome the non-differentiable nature of standard codecs, CATRF uses a Straight-Through Estimator (STE). During the forward pass, the actual decoded features are used to render the scene, but during the backward pass, the gradients bypass the codec entirely and flow directly into the pre-codec features.
Additionally, because running a video codec on every single training minibatch is computationally prohibitive, the authors introduce a caching mechanism that reuses decoded planes across multiple training steps, refreshing only when a set number of steps pass or the frame drifts too far.
Datasets
The authors evaluated CATRF across both static and dynamic benchmarks:
- Static Scenes: NeRF Synthetic (8 objects) and Tanks and Temples (highly detailed real-world scenes).
- Dynamic Scenes: Neural 3D Video (N3D) and the NHR Dataset, both representing multi-view dynamic human performances.
Key Insights and Ideas
- Real Codecs Beat Differentiable Proxies: Instead of training against a "differentiable proxy" that merely approximates a codec, training against the actual black-box codec via an STE forces the network to find clever, compression-resilient feature representations.
- No Rate Loss Required: Traditional learned compression requires balancing a reconstruction loss with an explicit rate loss (to keep file sizes small). CATRF completely omits a rate loss. Because standard codecs have highly optimized, mature rate-control algorithms, simply minimizing distortion at a fixed codec configuration implicitly drives the features to the most optimal point on the rate-distortion curve.
- Isolate the Channels: Packing features into a single grayscale canvas (
FlattenGray) outperforms RGB packing layouts. Standard video codecs heavily compress chroma channels; keeping features separated prevents artifacts in one channel from bleeding into and corrupting adjacent feature semantics.
Key Results
CATRF demonstrates significant improvements in both compression efficiency and runtime performance:
- Rate-Distortion Performance: On the dynamic N3D dataset, CATRF achieved massive Bjontegaard Delta (BD)-rate reductions over the codec-agnostic baseline TeTriRF, saving 40.5% with AV1, 51.8% with VP9, and 63.3% with HEVC.
- Outperforming 3D Gaussian Splatting (3DGS): Compared to state-of-the-art compressed 3DGS streaming methods like GIFStream, CATRF-AV1 achieved a 39% BD-rate reduction, delivering comparable or better visual quality at substantially lower bitrates.
- Decoding Speed: Because CATRF relies entirely on hardware-accelerated standard video decoding rather than neural network decoders, it achieves an exceptional client-side decoding throughput of 138 FPS (using HEVC), compared to just 2.94 FPS for GIFStream and 0.35 FPS for HAC++.
Limitations and Future Work
While CATRF achieves blazing-fast decoding speeds, its end-to-end client throughput is still bottlenecked by the rendering speed of the underlying NeRF backbone (averaging around 5.86 FPS in their prototype). The authors note that integrating their standard-codec-in-the-loop framework with faster hybrid backbones or more efficient neural renderers is a promising direction for achieving true real-time, low-bitrate volumetric streaming.
Summary updated