Immersive media applications—such as virtual reality, autonomous driving, and digital twin simulations—rely on high-fidelity 3D representations to capture the real world. Whether structured as sparse point clouds, dense volumetric videos, or continuous neural fields, these representations share a fundamental problem: they generate an astronomical volume of data. A few minutes of uncompressed volumetric video can easily consume terabytes of storage, far exceeding the transmission capacities of modern wireless networks. As noted by Sebastian Schwarz in an overview of emerging MPEG standards for point cloud compression, managing this data density is the critical gatekeeper for deploying real-time 3D experiences.
To bridge the gap between massive 3D datasets and bandwidth-constrained networks, researchers have developed a variety of compression frameworks. Rather than treating 3D data as an unorganized collection of coordinates and colors, these techniques exploit spatial redundancy, attribute correlations, temporal continuity, and human perceptual limits to compress volumetric data into streamable formats.
Exploiting Spatial Redundancy via Hierarchical Structures
The primary challenge of raw 3D data, such as LiDAR point clouds, is its unorganized nature. Storing explicit $(x, y, z)$ coordinates for millions of points is highly inefficient. To build intuition for how this can be optimized, imagine a sparse 3D space: instead of defining the exact location of every single point, we can recursively divide the space into smaller regions and only record whether a region contains data.
This is the core mechanism behind octree and K-d tree decompositions. By organizing points into a hierarchical tree structure, the spatial position of a point is implicitly defined by its path from the root of the tree. To maximize the efficiency of this structure, Lila Huang and Shenlong Wang developed OctSqueeze, an octree-structured entropy model that uses a neural network to predict the probability distribution of child nodes based on their parent context, significantly reducing the bits required for geometry coding. Similarly, the authors of ROAD (Sergey Zakharov et al.) demonstrated that traversing an implicit recursive octree auto-decoder allows thousands of complex 3D shapes to be compressed into a minimal latent footprint.
Beyond traditional point clouds, modern neural rendering techniques like 3D Gaussian Splatting (3DGS) face a similar spatial overhead because they require storing millions of unorganized geometric primitives. To compress these scenes, Yihang Chen et al. introduced HAC, a framework that anchors unorganized Gaussians to a structured hash grid. By leveraging the spatial context of the hash grid, the authors can accurately predict the attributes of neighboring Gaussians, enabling ultra-efficient entropy coding without degrading rendering quality.
The Attribute Compression Challenge
Compressing spatial coordinates (geometry) is only half the battle. In high-fidelity point clouds, the attributes associated with each point—such as color, normals, and reflectance—often consume significantly more data volume than the geometry itself. Unlike 2D images where pixels sit on a predictable grid, 3D attributes are scattered across irregular surfaces, making standard compression transforms (like the Discrete Cosine Transform) perform poorly.
To compress these irregular signals, researchers treat the point cloud as a graph, where points are nodes and spatial proximity defines the edges. Zhaobin Zhang et al. demonstrated this by optimizing a Graph Transform for Laplacian sparsity, which smooths out attribute variations across K-d trees to yield bitstreams up to four times smaller than traditional methods.
More recently, deep learning models have been introduced to capture more complex, non-linear attribute distributions. For instance, Changhao Peng proposed a Generalized Gaussian Entropy Model that dynamically adjusts likelihood intervals during arithmetic coding to better accommodate the statistical variations of 3D color data. Taking a generative approach, Hui Yuan et al. developed PCAC-GAN, which uses a sparse-tensor-based Generative Adversarial Network to synthesize high-fidelity textures from compressed representations, matching industry-standard codecs while preserving visual realism.
However, because geometry and attributes are fundamentally different data types, compression frameworks must carefully balance the bit allocation between them. As analyzed by Pan Gao et al. in their rate-distortion modeling work, altering geometry quantization directly impacts how attributes are mapped to the 3D surface, requiring unified distortion metrics to prevent visual artifacts like structural "holes."
Temporal Dynamics and 2D Hardware Codec Projection
When 3D environments move over time, they become 4D volumetric video. Capturing changes across every individual frame multiplies the data bottleneck exponentially. A powerful intuition for solving this problem is to avoid inventing entirely new 3D streaming architectures and instead leverage the highly optimized 2D hardware video codecs (like H.264 or HEVC) already present on modern consumer devices.
[4D Volumetric Scene]
│
▼ (Projection / Baking)
[2D Feature / Attribute Videos] ──► [Standard 2D Hardware Codec] ──► [Low-Bandwidth Transmission]
│
[Real-Time 3D Reconstruction] ◄── [Hardware-Accelerated Decoding] ◄────────────┘
This projection strategy underpins several state-of-the-art volumetric streaming pipelines:
- VideoRF: Liao Wang et al. designed a system that unfolds dynamic neural radiance fields into 2D feature video streams, allowing photorealistic, free-viewpoint human performances to be streamed and decoded smoothly on standard mobile hardware.
- $V^3$: To bring volumetric video to mobile platforms, Penghao Wang et al. developed a method that bakes dynamic 3D Gaussian Splatting attributes into streamable 2D dynamic Gaussians, bypassing the massive memory overhead typical of explicit 3D primitives.
- Hierarchical Progressive Coding (HPC): Addressing the volatility of network bandwidth, Zihan Zheng et al. introduced a progressive coding framework for volumetric video. By structuring the underlying representation into hierarchical residuals, the bitstream can be truncated mid-transmission to match available network speeds, delivering a lower-resolution 3D reconstruction rather than freezing the stream entirely.
Perceptual and Semantic Prioritization
A final, highly effective vector for data reduction is recognizing that not all 3D data points carry equal value. In a streaming environment, bandwidth can be conserved by allocating bits based on human visual attention or the specific requirements of downstream tasks.
From a human perception standpoint, Gustavo Sandri et al. integrated importance weights into the Region Adaptive Hierarchical Transform (RAHT) to enable Region of Interest (ROI) coding. By automatically detecting and prioritizing critical regions like human faces, their method achieves massive quality gains where it matters most, while aggressively compressing background elements. This concept is extended to spatial distance by Xi Wang et al., whose spatial perceptual quality-aware streaming framework drops fine geometric details that are physically imperceptible to a viewer standing at a distance from the volumetric object.
In machine-to-machine contexts, such as cooperative autonomous driving, the data volume can be reduced even further by discarding visual attributes entirely in favor of sparse semantic features. As demonstrated by Hao Cheng et al., sharing compressed 3D keypoint features between connected vehicles provides a 50x reduction in communication overhead compared to transmitting raw spatial data, proving that understanding the meaning of a 3D scene can be the ultimate tool for reducing its transmission footprint.