TL;DR
This paper introduces the first deep learning-based framework to remove geometry compression artifacts from dynamic point clouds encoded with the MPEG V-PCC (Video-based Point Cloud Compression) standard. By exploiting the prior knowledge that V-PCC quantization noise occurs strictly along a single spatial axis, the authors design a 3D sparse convolutional network that significantly improves reconstruction quality and saves bandwidth without altering the underlying codec.
What Problem Does This Paper Address?
Dynamic 3D point clouds—essential for AR/VR, telepresence, and volumetric video—require massive amounts of data, making efficient compression critical. The state-of-the-art MPEG standard, V-PCC, compresses these point clouds by projecting 3D patches onto 2D planes and encoding them using standard 2D video codecs (like HEVC).
While highly efficient, V-PCC's lossy quantization process introduces severe, visually jarring "blocking" artifacts in the 3D geometry, especially at low bitrates. Furthermore, because lossy compression alters the total number of points in the cloud, standard 3D denoising techniques struggle to map the noisy compressed points back to the clean ground-truth geometry. This paper aims to clean up these artifacts as a post-processing step.
Methods
The authors propose an out-of-the-loop artifact removal framework consisting of three main stages:
- Cube-based Sampling: To solve the "patch correspondence mismatch" (the differing point counts between the original and compressed clouds), the authors avoid standard k-Nearest Neighbors (k-NN) grouping. Instead, they use Farthest Point Sampling (FPS) to select center points and extract all points within a fixed 3D cube volume.
- 3D Sparse U-Net: The extracted 3D patches are fed into a 3D U-Net built with submanifold sparse convolutions. This architecture efficiently handles the variable number of points in each cube without "dilating" the sparse data, saving memory and compute.
- Projection-Aware Noise Calculation: Instead of directly predicting 3D noise, the network predicts a 3D projection vector and a 1D scalar weight. Because V-PCC projects points orthogonally onto bounding box planes, the quantization noise always falls strictly along the x, y, or z axis. The network converts the projection vector into a one-hot vector (e.g.,
[1, 0, 0]) and multiplies it by the scalar weight to determine the exact magnitude and direction of the noise. - Aggregation: Overlapping cleaned patches are averaged together to form the final high-quality point cloud.
Datasets
The framework is trained and evaluated on the 8i voxelized full bodies dataset from 8i Labs, a standard benchmark widely used by the MPEG group for point cloud compression.
- Training: longdress and loot sequences.
- Testing: redandblack, soldier, and queen sequences.
- The point clouds were encoded at three different bitrates (br1, br2, br3) using the V-PCC standard to simulate varying levels of compression artifacts.
Key Insights and Ideas
- The Directional Noise Prior: The biggest "aha moment" of the paper is the realization that V-PCC geometry distortion is highly constrained. Because points are projected onto orthogonal 2D planes, the resulting quantization noise only shifts points along a single axis. By forcing the neural network to output a one-hot directional vector, the authors drastically reduced the degrees of freedom for the learned noise, outperforming networks that try to predict raw 3D noise directly.
- k-NN Fails on Compressed Data: Traditional point cloud networks use k-NN to group points into patches. But lossy compression changes point density. A 61-point k-NN patch covers a different physical area in a sparse compressed cloud versus a dense original cloud. Fixed-volume cube extraction elegantly solves this mismatch, allowing the network to compare apples to apples.
Key Results
- Bandwidth Savings: The method achieves an impressive 11.3% BD-Rate savings, meaning it can deliver the same visual quality at an 11.3% lower bitrate compared to standard V-PCC.
- PSNR Improvements: At the lowest bitrate (br3), the method improved the point-to-point MSE PSNR by an average of 0.85 dB (from 59.62 dB to 60.47 dB) across the test sequences.
- Visual Quality: Qualitative results show a dramatic reduction in blocking artifacts, particularly on the edges and surfaces of the human models. The network even manages to reconstruct some "broken" or missing parts of the geometry lost during V-PCC encoding.
- Ablation on Sampling: The proposed overlapping cube-based sampling outperformed a non-overlapping octree-based baseline by ~0.4 dB, proving the value of the aggregation step.
Limitations and Future Work
- Geometry Only: The paper strictly focuses on removing artifacts from the geometry of the point cloud. It does not address compression artifacts in the texture (color) information, which is also heavily compressed by V-PCC and crucial for final visual quality.
- Computation Time: The overlapping patch extraction and aggregation steps scale linearly with the number of overlapping patches (controlled by a hyperparameter $C$). Higher quality requires more overlap, which increases processing time.
- Future Directions: While not explicitly detailed by the authors, natural extensions of this work would include joint geometry-texture artifact removal, and potentially integrating this learned prior directly into the V-PCC decoding loop rather than applying it strictly as a post-processing step.
Summary updated