A single 3D scan from a LiDAR, radar, or depth sensor is fundamentally incomplete. Because these sensors rely on line-of-sight ray casting, they only capture the surfaces immediately facing the sensor. Distant objects receive very few point returns, and foreground objects cast long, occluding "shadows" over everything behind them.
Processing these frames in isolation discards a massive, naturally occurring source of information: time. As a sensor moves through an environment or as objects move past the sensor, new perspectives are revealed. Temporal point cloud aggregation is the practice of aligning and fusing these sequential, partial observations into a single, densified 3D representation. By shifting from static frame-by-frame processing to temporal accumulation, models can resolve geometric ambiguities, denoise sensor readings, and even reconstruct complete 3D shapes without manual labels.
Why It Works: Perspective and Redundancy
To understand why temporal aggregation is so effective, consider the difference between a single photograph of a sculpture and a video walking around it. The photograph provides a 2.5D snapshot from one angle, while the video contains the full 3D structure distributed across time.
In autonomous driving or robotics, this temporal distribution manifests in two ways:
- Ego-Motion: As the vehicle or robot moves, its changing viewpoint naturally fills in the occluded regions of the background.
- Object Motion: As a dynamic object (like a passing car or pedestrian) moves along a trajectory, it exposes different sides of its geometry to a stationary or moving sensor.
By accumulating points across consecutive frames, we can "densify" the point cloud. As noted by Sheng et al., incorporating multi-frame data resolves ambiguities caused by occlusion or sensor noise that are impossible to fix in a single frame.
Furthermore, temporal accumulation acts as a geometric filter. Zheng et al. demonstrate that aggregating consecutive LiDAR sweeps serves as a geometric denoiser, filling in structural gaps. This works because random sensor noise averages out over time, while true physical surfaces consistently return points, reinforcing the underlying geometry. Hu et al. show that enforcing this temporal consistency allows models to exploit the redundant information inherent in motion, recovering data gaps that are otherwise lost.
The Core Challenge: Mapping to a Shared Space
We cannot simply dump multiple point clouds into a single coordinate system; doing so without accounting for movement results in severe "motion smear" or blurring. The core engineering challenge of temporal aggregation is alignment.
Static Alignment via Localization
For permanent environmental structures (like roads and buildings), alignment relies on compensating for the sensor's ego-motion. By using vehicle pose or global localization, historical point clouds can be transformed into the current coordinate frame. For example, Phoo et al. aggregate historical point clouds from multiple previous traversals of an area to construct a dense spatial prior of the static background, giving camera-only detectors a highly accurate geometric reference.
Dynamic Alignment via Canonical Spaces
When objects are moving independently, ego-motion compensation is not enough. The system must track individual object trajectories and isolate their motion from the background.
One elegant solution is to map time-varying observations into a shared, object-centric canonical coordinate space. Lei et al. use this approach to represent deforming 3D surfaces. By mapping partial observations from various time steps into a single canonical space, their framework can reconstruct a complete, topologically consistent 3D shape from incomplete frames. Similarly, Huang et al. explicitly disentangle moving objects from the static background, accumulating points along each object's rigid trajectory to prevent motion blur and maintain visibility through temporary occlusions.
[Frame t-1: Partial View] ---\
[Frame t : Partial View] ----+---> [Canonical Space Alignment] ---> [Dense, Completed 3D Shape]
[Frame t+1: Partial View] ---/
Bootstrapping Supervision Without Manual Labels
Perhaps the most impactful consequence of temporal aggregation is its ability to generate "free" supervision for self-supervised learning. Manual 3D bounding box or occupancy annotations are incredibly expensive to produce. Temporal aggregation bypasses this bottleneck by letting the passage of time do the work.
As a platform moves around an object, the accumulated dense point cloud can be treated as a "ground truth" representation of what the object looks like, which can then be used to train a model that only sees a single, sparse frame during inference.
- Shape Priors: Takmaz et al. leverage multi-view temporal aggregation to mine completed shape priors from unlabeled sequences, distilling them into a model to enable zero-shot 3D scene completion.
- Trajectory Aggregation: Huang et al. automatically track moving objects across sequences and aggregate their points over time. This creates a densified geometric profile that provides a clean training target for estimating 3D bounding boxes from sparse, single-frame captures.
- Surface Reconstruction: Wei et al. stitch multiple temporal frames together and apply Poisson Reconstruction to generate continuous surface meshes. These meshes are then voxelized to create dense training labels for multi-camera 3D occupancy prediction without requiring human annotators to label every voxel.
By turning time into density, temporal point cloud aggregation bridges the gap between the sparse, noisy reality of real-time 3D sensors and the complete, high-fidelity geometric understanding required for autonomous perception.