Video-Based Point Cloud Compression Artifact Removal

Anique Akhtar, Wen Gao, Li Li 0040, Zhu Li 0001, Wei Jia, Shan Liu

TL;DR

This paper introduces the first deep learning-based framework to remove geometry compression artifacts from dynamic point clouds encoded with the MPEG V-PCC (Video-based Point Cloud Compression) standard. By exploiting the prior knowledge that V-PCC quantization noise occurs strictly along a single spatial axis, the authors design a 3D sparse convolutional network that significantly improves reconstruction quality and saves bandwidth without altering the underlying codec.

What Problem Does This Paper Address?

Dynamic 3D point clouds—essential for AR/VR, telepresence, and volumetric video—require massive amounts of data, making efficient compression critical. The state-of-the-art MPEG standard, V-PCC, compresses these point clouds by projecting 3D patches onto 2D planes and encoding them using standard 2D video codecs (like HEVC).

While highly efficient, V-PCC's lossy quantization process introduces severe, visually jarring "blocking" artifacts in the 3D geometry, especially at low bitrates. Furthermore, because lossy compression alters the total number of points in the cloud, standard 3D denoising techniques struggle to map the noisy compressed points back to the clean ground-truth geometry. This paper aims to clean up these artifacts as a post-processing step.

Methods

The authors propose an out-of-the-loop artifact removal framework consisting of three main stages:

  1. Cube-based Sampling: To solve the "patch correspondence mismatch" (the differing point counts between the original and compressed clouds), the authors avoid standard k-Nearest Neighbors (k-NN) grouping. Instead, they use Farthest Point Sampling (FPS) to select center points and extract all points within a fixed 3D cube volume.
  2. 3D Sparse U-Net: The extracted 3D patches are fed into a 3D U-Net built with submanifold sparse convolutions. This architecture efficiently handles the variable number of points in each cube without "dilating" the sparse data, saving memory and compute.
  3. Projection-Aware Noise Calculation: Instead of directly predicting 3D noise, the network predicts a 3D projection vector and a 1D scalar weight. Because V-PCC projects points orthogonally onto bounding box planes, the quantization noise always falls strictly along the x, y, or z axis. The network converts the projection vector into a one-hot vector (e.g., [1, 0, 0]) and multiplies it by the scalar weight to determine the exact magnitude and direction of the noise.
  4. Aggregation: Overlapping cleaned patches are averaged together to form the final high-quality point cloud.

Datasets

The framework is trained and evaluated on the 8i voxelized full bodies dataset from 8i Labs, a standard benchmark widely used by the MPEG group for point cloud compression.

  • Training: longdress and loot sequences.
  • Testing: redandblack, soldier, and queen sequences.
  • The point clouds were encoded at three different bitrates (br1, br2, br3) using the V-PCC standard to simulate varying levels of compression artifacts.

Key Insights and Ideas

  • The Directional Noise Prior: The biggest "aha moment" of the paper is the realization that V-PCC geometry distortion is highly constrained. Because points are projected onto orthogonal 2D planes, the resulting quantization noise only shifts points along a single axis. By forcing the neural network to output a one-hot directional vector, the authors drastically reduced the degrees of freedom for the learned noise, outperforming networks that try to predict raw 3D noise directly.
  • k-NN Fails on Compressed Data: Traditional point cloud networks use k-NN to group points into patches. But lossy compression changes point density. A 61-point k-NN patch covers a different physical area in a sparse compressed cloud versus a dense original cloud. Fixed-volume cube extraction elegantly solves this mismatch, allowing the network to compare apples to apples.

Key Results

  • Bandwidth Savings: The method achieves an impressive 11.3% BD-Rate savings, meaning it can deliver the same visual quality at an 11.3% lower bitrate compared to standard V-PCC.
  • PSNR Improvements: At the lowest bitrate (br3), the method improved the point-to-point MSE PSNR by an average of 0.85 dB (from 59.62 dB to 60.47 dB) across the test sequences.
  • Visual Quality: Qualitative results show a dramatic reduction in blocking artifacts, particularly on the edges and surfaces of the human models. The network even manages to reconstruct some "broken" or missing parts of the geometry lost during V-PCC encoding.
  • Ablation on Sampling: The proposed overlapping cube-based sampling outperformed a non-overlapping octree-based baseline by ~0.4 dB, proving the value of the aggregation step.

Limitations and Future Work

  • Geometry Only: The paper strictly focuses on removing artifacts from the geometry of the point cloud. It does not address compression artifacts in the texture (color) information, which is also heavily compressed by V-PCC and crucial for final visual quality.
  • Computation Time: The overlapping patch extraction and aggregation steps scale linearly with the number of overlapping patches (controlled by a hyperparameter $C$). Higher quality requires more overlap, which increases processing time.
  • Future Directions: While not explicitly detailed by the authors, natural extensions of this work would include joint geometry-texture artifact removal, and potentially integrating this learned prior directly into the V-PCC decoding loop rather than applying it strictly as a post-processing step.

Summary updated

Abstract

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.
BibTeX
@article{akhtar2022video,
  author = {Anique Akhtar and Wen Gao and Li Li and Zhu Li and Wei Jia and Shan Liu},
  title = {{Video-Based Point Cloud Compression Artifact Removal}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3090148},
}

Loading related content...

Figure