CLIFT: Contrastive LiDAR Instance Features for 3D Pedestrian Tracking

German Devchich, Timur Akhtyamov, Denis Fatykhoph, Gonzalo FerrerarXiv 2026 · arXiv

TL;DR

CLIFT (Contrastive LiDAR Instance Features for Tracking) transforms standard 3D LiDAR detectors into state-of-the-art trackers by adding a contrastive learning objective to their intermediate voxel features. By learning identity-aware embeddings directly within the detector's backbone, it achieves a massive 13-point HOTA jump on the JRDB benchmark—all without using a motion model or camera fusion.

What Problem Does This Paper Address?

For a social robot navigating a crowded campus, "who is who" matters just as much as "where is everyone." This is the domain of 3D Multi-Object Tracking (MOT). While 3D detection (finding boxes in a single frame) has improved rapidly, tracking (linking those boxes over time) has lagged behind.

The traditional approach is "tracking-by-detection": you run a detector, then use a separate motion model (like a Kalman filter) or a dedicated re-identification (Re-ID) network to link boxes. On LiDAR, this is notoriously difficult because point clouds are sparse and pedestrians are small, deformable objects. Many researchers "solve" this by fusing LiDAR with cameras to get better visual features, but this introduces headaches like sensor calibration, synchronization, and failure in poor lighting.

The authors of CLIFT ask a provocative question: Can we achieve state-of-the-art tracking using LiDAR alone by better utilizing the features the detector is already calculating?

Methods

The core innovation of CLIFT is backbone-agnostic contrastive learning on voxel features.

  1. Feature Extraction: Instead of looking at raw point clouds or cropped images, CLIFT taps into the intermediate voxel-level features of a 3D detector (like DCCLA or RPEA). These features are already "semantically rich" because the detector has been trained to distinguish pedestrians from the background.
  2. Instance Embeddings: For every detected bounding box, the system gathers the voxels inside it, performs mean-pooling to get a single vector, and passes it through a small MLP to create a 128-dimensional embedding.
  3. Contrastive Objective: During training, they use a multi-positive InfoNCE loss. This pulls embeddings of the same person (from different frames) together in vector space while pushing embeddings of different people apart.
  4. Joint Fine-tuning: This is the "secret sauce." They fine-tune the entire detector backbone using both the original detection loss and the new tracking loss. This forces the backbone to learn features that are stable over time for a specific individual.
  5. Simple Association: At inference time, they don't use a Kalman filter or complex motion priors. They simply compare the embeddings of new detections to a memory of recent tracks using cosine similarity and solve the matching problem with the Hungarian algorithm.

Datasets

The primary evaluation is performed on JRDB (JackRabbits DataBase), a challenging 3D tracking benchmark recorded from a mobile robot in dense pedestrian crowds. To test generalization, they also evaluate on SiT (Socially Interactive pedestrian Trajectory), another robot-mounted dataset.

Key Insights and Ideas

  • Voxel Features > Raw Geometry: Raw point clouds change drastically as a person moves or is partially occluded. However, the intermediate features inside a neural network are more stable. By pooling these "detector-level" features, CLIFT gains a representation that is already tuned for the object of interest.
  • The Invariance vs. Discrimination Conflict: There is a fundamental tension in multi-task learning here. Detection wants invariance (all pedestrians should look like the "pedestrian" class), while tracking wants discrimination (this pedestrian must look different from that one). The authors found that despite these gradients conflicting 40-50% of the time, joint training still significantly improves both identity stability and association.
  • Identity over Motion: In dense, erratic crowds (like a university quad), motion models often fail because people change direction suddenly. CLIFT proves that if your identity features are strong enough, you can throw away the motion model entirely and still beat the state-of-the-art.
  • Consolidation: Probing the features showed that joint fine-tuning doesn't just separate different people; it "consolidates" the identity, making the features of the same person much more similar across different frames and viewpoints.

Key Results

CLIFT delivered a dominant performance on the JRDB 3D tracking leaderboard:

  • Ranked 1st in HOTA (38.58) and IDF1 (37.99).
  • Outperformed the next best published method by 13.04 HOTA points and 9.30 IDF1 points.
  • Reduced identity switches (IDSW) by approximately 60% compared to the baseline.
  • Efficiency: Despite the deep features, the system runs at 11.4 FPS on a single RTX 3090, making it viable for real-time robotics.

Limitations and Future Work

While CLIFT is a leap forward for LiDAR-only tracking, the authors note a few areas for improvement:

  • Long-range Re-ID: While it handles short-term occlusions well, matching identities after a person disappears for a long time remains a challenge.
  • OSPA Metric: The method ranks second on the OSPA metric. This is because CLIFT tends to produce more track fragments/IDs than the baseline, which the OSPA cardinality penalty punishes heavily.
  • Downstream Integration: The ultimate goal is better robot navigation. Future work involves testing how these more stable trajectories actually translate into smoother, safer robot motion planning.

For more details, check out the code at https://hermak56.github.io/CLIFT/.

Summary updated

Abstract

A social robot has to keep track of the people around it over time to plan its motion, a task handled by 3D multi-object tracking. For the LiDAR modality, trackers usually sit on top of a detector and link its detections across frames with a separate motion model or re-identification network. We instead show how a 3D LiDAR detector itself can be turned into a tracker by adding an instance-level contrastive objective on its intermediate voxel features. Each embedding is read from the voxels inside a detected box, and boxes are matched across frames by cosine similarity and the Hungarian algorithm, without a motion model. The detector backbone is fine-tuned together with this objective, which learns identity-aware features while leaving detection accuracy essen- tially intact. Because the method uses only the voxel features that Li- DAR detectors already produce, it is backbone-agnostic. On the JRDB 3D tracking leaderboard our LiDAR-only tracker ranks first on HOTA (38.58) and IDF1 (37.99), 13.04 and 9.30 points above the next pub- lished method, and second on MOTA and OSPA. Code is available at https://hermak56.github.io/CLIFT/.
BibTeX
@misc{devchich2026clift,
  author = {German Devchich and Timur Akhtyamov and Denis Fatykhoph and Gonzalo Ferrer},
  title = {{CLIFT: Contrastive LiDAR Instance Features for 3D Pedestrian Tracking}},
  year = {2026},
  eprint = {2611.20737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20737},
}
Figure