Abstract
Processing 4D point cloud sequences is computationally demanding due to the unstructured nature of point data and the explosive growth of memory requirements over time. This paper presents the Point Primitive Transformer (PPTr), a hierarchical backbone that leverages "primitive planes" as a mid-level representation. By grouping points into stable geometric primitives, PPTr efficiently captures long-term spatial-temporal context. Our approach achieves state-of-the-art results on action recognition and semantic segmentation benchmarks, demonstrating the ability to process sequences up to 10x longer than previous methods.
Introduction
In robotics and augmented reality, understanding dynamic 3D scenes is critical. However, 4D point cloud video understanding faces two primary challenges: effectiveness, due to inconsistent point sampling and occlusions across frames, and efficiency, as the "memory wall" of standard transformers often limits processing to very short snippets (e.g., 3 frames). The Point Primitive Transformer (PPTr) addresses these by observing that while individual points are transient, underlying geometric primitives like walls or floors remain stable over time.
Methodology
PPTr introduces a two-level hierarchical architecture centered on Primitive Fitting.
Primitive Fitting and Intra-Primitive Attention
The model first groups points into "primitive planes" (e.g., using RANSAC). The Intra-Primitive Point Transformer then processes short video clips by restricting attention to points within the same primitive. This geometry-aware locality bias reduces computational complexity while ensuring extracted features are spatially descriptive.
Long-Term Primitive Transformer
To capture long-term dependencies, PPTr maintains a Memory Pool of pre-computed primitive features. The Primitive Transformer performs self-attention at the primitive level across a much longer history (up to 30+ frames). Since the number of primitives is significantly smaller than the number of points, this approach bypasses the quadratic memory cost associated with raw point processing.
Experiments and Results
The researchers evaluated PPTr on three benchmarks: MSR-Action3D (human movement), Synthia4D (autonomous driving), and HOI4D (human-object interaction).
Performance and Scaling
PPTr consistently outperformed previous state-of-the-art (SOTA) methods. On MSR-Action3D, it achieved 92.33% accuracy. On HOI4D, it improved mIoU from 61.97% to 70.13%.
As shown in the figures above, increasing the clip length directly correlates with improved recognition accuracy. Crucially, PPTr's efficiency allows it to scale to 16-30 frames, whereas previous models were often limited by memory constraints at much shorter lengths.
Conclusion
The Point Primitive Transformer demonstrates that mid-level geometric abstractions are key to efficient 4D scene understanding. By decoupling point-level detail from long-term temporal context via primitives, PPTr sets a new standard for processing long-duration point cloud videos. Future work may explore organic shape representations beyond simple planes to further generalize the model's capabilities.
Summary updated