Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding

Yunze Liu

Abstract

Processing 4D point cloud sequences is computationally demanding due to the unstructured nature of point data and the explosive growth of memory requirements over time. This paper presents the Point Primitive Transformer (PPTr), a hierarchical backbone that leverages "primitive planes" as a mid-level representation. By grouping points into stable geometric primitives, PPTr efficiently captures long-term spatial-temporal context. Our approach achieves state-of-the-art results on action recognition and semantic segmentation benchmarks, demonstrating the ability to process sequences up to 10x longer than previous methods.

Introduction

In robotics and augmented reality, understanding dynamic 3D scenes is critical. However, 4D point cloud video understanding faces two primary challenges: effectiveness, due to inconsistent point sampling and occlusions across frames, and efficiency, as the "memory wall" of standard transformers often limits processing to very short snippets (e.g., 3 frames). The Point Primitive Transformer (PPTr) addresses these by observing that while individual points are transient, underlying geometric primitives like walls or floors remain stable over time.

Methodology

PPTr introduces a two-level hierarchical architecture centered on Primitive Fitting.

The PPTr architecture overview, showing the transition from raw input clips to primitive-based features for segmentation and classification.

Primitive Fitting and Intra-Primitive Attention

The model first groups points into "primitive planes" (e.g., using RANSAC). The Intra-Primitive Point Transformer then processes short video clips by restricting attention to points within the same primitive. This geometry-aware locality bias reduces computational complexity while ensuring extracted features are spatially descriptive.

Long-Term Primitive Transformer

To capture long-term dependencies, PPTr maintains a Memory Pool of pre-computed primitive features. The Primitive Transformer performs self-attention at the primitive level across a much longer history (up to 30+ frames). Since the number of primitives is significantly smaller than the number of points, this approach bypasses the quadratic memory cost associated with raw point processing.

Experiments and Results

The researchers evaluated PPTr on three benchmarks: MSR-Action3D (human movement), Synthia4D (autonomous driving), and HOI4D (human-object interaction).

Performance and Scaling

PPTr consistently outperformed previous state-of-the-art (SOTA) methods. On MSR-Action3D, it achieved 92.33% accuracy. On HOI4D, it improved mIoU from 61.97% to 70.13%.

Graphs showing the influence of clip length on action recognition accuracy (a) and memory consumption (b).

As shown in the figures above, increasing the clip length directly correlates with improved recognition accuracy. Crucially, PPTr's efficiency allows it to scale to 16-30 frames, whereas previous models were often limited by memory constraints at much shorter lengths.

Conclusion

The Point Primitive Transformer demonstrates that mid-level geometric abstractions are key to efficient 4D scene understanding. By decoupling point-level detail from long-term temporal context via primitives, PPTr sets a new standard for processing long-duration point cloud videos. Future work may explore organic shape representations beyond simple planes to further generalize the model's capabilities.

Summary updated

Abstract

This paper proposes a method for representation learning of multimodal data using contrastive losses. A traditional approach is to contrast different modalities to learn the information shared between them. However, that approach could fail to learn the complementary synergies between modalities that might be useful for downstream tasks. Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences. However, that approach could consider only the stronger modalities while ignoring the weaker ones. To address these issues, we propose a novel contrastive learning objective, TupleInfoNCE. It contrasts tuples based not only on positive and negative correspondences, but also by composing new negative tuples using modalities describing different scenes. Training with these additional negatives encourages the learning model to examine the correspondences among modalities in the same tuple, ensuring that weak modalities are not ignored. We provide a theoretical justification based on mutual-information for why this approach works, and we propose a sample optimization algorithm to generate positive and negative samples to maximize training efficacy. We find that TupleInfoNCE significantly outperforms previous state of the arts on three different downstream tasks.
BibTeX
@misc{liu2022point,
  author = {Yunze Liu},
  title = {{Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding}},
  year = {2022},
  eprint = {2208.00281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.00281},
}
Figure