Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding

Hao Wen, Yunze Liu, Jingwei Huang, Bo Duan, Li Yi

Introduction

Understanding 4D data—3D point clouds evolving over time—is a cornerstone for advancements in autonomous driving, robotics, and augmented reality. However, processing these sequences remains computationally expensive and technically challenging. Traditional methods, such as 4D convolutions or flat transformers, often struggle with the unstructured nature of point clouds and the massive memory requirements of long-term video sequences. To address these limitations, we introduce the Point Primitive Transformer (PPTr), a hierarchical backbone that leverages "primitive planes" as a stable mid-level representation to capture long-term spatial-temporal context efficiently.

The Challenges of 4D Point Cloud Understanding

Current 4D backbones face two primary hurdles:

  1. Efficiency and Memory Constraints: Point clouds are dense and unstructured. Loading long sequences into GPU memory for global attention mechanisms leads to a "memory explosion." For instance, a standard 24GB GPU is often limited to processing only a few frames at a time, preventing models from learning long-term temporal dependencies.
  2. Temporal Inconsistency: At the point level, data is often "flickery." Due to camera motion, occlusions, or sensor noise, specific points may disappear or shift between frames. This lack of point-to-point correspondence makes it difficult for models to track features over extended durations.

PPTr overcomes these issues by moving away from raw point-level processing toward a more stable geometric abstraction.

The Point Primitive Transformer (PPTr)

The core innovation of PPTr is the use of Primitive Planes. Instead of treating every point as an independent entity, the model groups points into geometrically coherent planes (using algorithms like RANSAC or Region Growing). These planes act as a mid-level representation that remains stable even when individual points shift.

Hierarchical Architecture

The PPTr architecture is divided into two main components designed to balance local detail with long-term context:

  • The Online Branch: This branch processes short, local video clips. It utilizes an Intra-primitive Point Transformer, where attention is restricted to points within the same primitive. This "geometry-aware" inductive bias simplifies the optimization process and reduces noise by preventing the model from attending to irrelevant global features during local feature extraction.
  • The Offline Branch and Memory Pool: To handle long-term context without exceeding memory limits, PPTr uses a decoupled memory architecture. The model pre-computes primitive-level features for the entire video and stores them in an offline Memory Pool.
  • Primitive Transformer: This module performs attention between the current clip's primitives and the long-term primitives stored in the memory pool. By attending to these high-level abstractions rather than raw points, the model can "look back" dozens of frames (e.g., 30+ frames) with minimal computational overhead.

Experimental Evaluation

PPTr was evaluated across three major benchmarks, demonstrating significant improvements over previous state-of-the-art (SOTA) models like P4Transformer.

Synthia 4D

On the Synthia 4D dataset for outdoor autonomous driving, PPTr achieved 84.48% mIoU, surpassing the previous SOTA of 83.16%. Notably, PPTr utilized 30 frames of context, whereas previous models were typically capped at 3 frames due to memory constraints.

HOI4D

In the HOI4D egocentric dataset, which focuses on complex human-object interactions, PPTr showed a massive performance leap. It reached 70.13% mIoU, representing an 8.16% absolute improvement over P4Transformer. This highlights the model's ability to maintain context in dense, interactive environments.

MSR-Action3D

For action recognition tasks using human body point clouds, PPTr achieved an accuracy of 92.33%, proving its versatility across both semantic segmentation and classification tasks.

Key Insights

  • Geometry as a Filter: Restricting attention to intra-primitive points acts as a natural filter, making the learning process more stable for dense tasks like segmentation.
  • Stability Over Time: While points are inconsistent, a primitive (like a wall or floor) is constant. This makes temporal association significantly easier for the network.
  • Scalability: PPTr scales comfortably to long sequences where other models hit a memory ceiling at approximately 15 frames.

Conclusion and Future Work

The Point Primitive Transformer demonstrates that mid-level geometric abstractions are key to unlocking long-term 4D video understanding. By decoupling memory from active computation and utilizing primitive planes, PPTr achieves superior accuracy and efficiency.

Future research will focus on expanding the types of primitives used (moving beyond simple planes) and making the primitive fitting process fully differentiable to allow for end-to-end training without a multi-stage pipeline requirement.

Summary updated

Abstract

This paper proposes a 4D backbone for long-term point cloud video understanding. A typical way to capture spatial-temporal context is using 4Dconv or transformer without hierarchy. However, those methods are neither effective nor efficient enough due to camera motion, scene changes, sampling patterns, and the complexity of 4D data. To address those issues, we leverage the primitive plane as a mid-level representation to capture the long-term spatial-temporal context in 4D point cloud videos and propose a novel hierarchical backbone named Point Primitive Transformer(PPTr), which is mainly composed of intra-primitive point transformers and primitive transformers. Extensive experiments show that PPTr outperforms the previous state of the arts on different tasks
BibTeX
@misc{wen2022point,
  author = {Hao Wen and Yunze Liu and Jingwei Huang and Bo Duan and Li Yi},
  title = {{Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding}},
  year = {2022},
  eprint = {2208.00281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.00281},
}
Figure