Lifting Multi Object Tracking into Three Dimensional Space

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 150 papers65 conceptsfrozen 2026-10-04

Traditional Multiple Object Tracking (MOT) has long been treated as a 2D problem: finding bounding boxes in an image and linking them across frames. However, the 2D image plane is a lossy projection of a complex 3D world. When two people walk past each other, their 2D bounding boxes overlap, causing "identity switches" where a tracker loses track of who is who.

By "lifting" tracking from 2D pixels into 3D world coordinates, researchers have found that many of the most difficult problems in computer vision—occlusions, scale changes, and camera motion—become geometrically trivial to solve.

Resolving Ambiguity Through Depth

In a 2D image, two objects can occupy the same pixel coordinates, leading to inevitable confusion for tracking algorithms. However, as Xinshuo Weng and Kris Kitani demonstrate in AB3DMOT, physical objects rarely overlap in 3D space. By using 3D bounding boxes, trackers can resolve depth ambiguities that would otherwise cause identity switches. Even when one person is directly behind another from the camera's perspective, they remain distinct along the Z-axis.

This spatial awareness is particularly powerful when dealing with complex human interactions. Shubham Goel and Angjoo Kanazawa show in Humans in 4D that reconstructing a full 3D human mesh allows for superior disambiguation. Instead of just a box, the system understands the subject's 3D pose and shape, providing a "spatial fingerprint" that is far more robust than 2D appearance alone.

Decoupling Object Motion from Camera Motion

A major failure point for 2D trackers is camera ego-motion. If a drone tilts or a sports camera pans rapidly, every pixel in the frame moves, often breaking the "intersection-over-union" (IoU) logic used to link detections.

The solution is to register the tracking to a fixed 3D coordinate system. Cheng-Yen Yang addresses this in Sea You Later by using a drone's GPS and gimbal sensors to project 2D detections into 3D world coordinates. Similarly, Hong-Qi Chen and Chao-Chi Liao introduce a "register-then-track" paradigm in FieldMOT, where players are projected onto a 3D field model. In this 3D space, a player's "velocity" is their actual physical speed on the grass, which remains constant even if the camera cuts to a different angle or zooms in.

Occlusion as a Geometric State

In 2D tracking, when an object is occluded, it simply disappears from the data stream. In 3D, however, occlusion is just a geometric state: the object still exists at a specific coordinate; it is simply hidden behind another 3D volume.

Jathushan Rajasegaran and Jitendra Malik exploit this in PHALP by predicting a person's future 3D state. Because the system understands the 3D environment, it can maintain a "persistent" track of a person even when they are invisible, essentially "hallucinating" their trajectory through 3D space until they reappear. This is echoed by Yilun Du and Jiajun Wu in POD-Net, which uses 3D primitives to ensure that objects persist over time, adhering to physical laws of motion that 2D pixels often violate.

Geometric Consistency over Visual Appearance

While deep learning has made 2D appearance matching (Re-identification) very strong, it still struggles with "distractors"—objects that look nearly identical. In SynapFlow, Pamela Osuna-Vargas and colleagues demonstrate that in 4D microscopy, where biological structures look almost identical, 3D spatial consistency is a much more reliable signal than visual appearance.

This shift toward geometry is further refined by Wenhao Chai in CCGS, which uses 3D pointmaps to associate masks across views. By establishing pixel-to-pixel links through 3D space, the system creates a "geometric bridge" that maintains identity more reliably than any 2D feature vector could. As Bin Deng shows in DepTR-MOT, even "pseudo" 3D information—like instance-level depth predicted from a monocular image—can provide the necessary spatial context to refine trajectories and prevent errors in crowded scenes.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lifting_multi_object_tracking_into_three_073b1592,
  title = {Lifting Multi Object Tracking into Three Dimensional Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lifting-multi-object-tracking-into-three-dimensional-space/rdr_073b159249884a41ae5773220a8fbdc8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_lifting_multi_object_tracking_into_three_073b1592_v1,
  title = {Lifting Multi Object Tracking into Three Dimensional Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lifting-multi-object-tracking-into-three-dimensional-space/rdr_073b159249884a41ae5773220a8fbdc8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}