Computer vision has thrived on massive, single-modality datasets. Standard RGB tracking algorithms leverage millions of annotated frames to learn rich, generalizable visual representations. However, real-world deployments—such as autonomous driving, robotics, and 24-hour surveillance—frequently encounter degraded conditions like pitch darkness, extreme glare, fast motion, and heavy occlusions where visible light alone fails.
Fusing RGB video with auxiliary modalities—such as Depth (RGB-D), Thermal Infrared (RGB-T), Near-Infrared (RGB-NIR), or bio-inspired Event streams—offers a clear physical solution to these edge cases. Yet, as noted by Jinyu Yang et al. and Tianyang Xu et al., multi-modal tracking has historically been bottlenecked by a severe scarcity of large-scale, diverse, annotated datasets. This data deficiency creates a fundamental architectural challenge: how can deep neural networks learn to exploit multi-sensor inputs without overfitting to small, specialized benchmarks?
Why Small Benchmarks Mislead
When a tracking model is trained on a small dataset, it cannot effectively learn the complex, high-level geometric or physical features unique to auxiliary sensors. Instead, networks often default to relying heavily on pre-trained RGB weights or hand-crafted heuristics, as observed by Song Yan et al..
Early multi-modal benchmarks, such as the Princeton Tracking Benchmark introduced by Shuran Song et al., successfully demonstrated that depth maps could reduce tracking drift and handle occlusions. However, these early datasets were largely restricted to small-scale, indoor environments with short sequences where targets remained continuously visible.
As the field progressed, this lack of scale exposed a surprising "performance gap." Alan Lukezic et al. revealed that when evaluated on more complex, long-term outdoor scenarios, top-tier RGB-only trackers—which benefited from massive single-modality pre-training—actually outperformed specialized depth-aware trackers. The depth-aware models simply lacked the data volume required to generalize across diverse environments.
This data scarcity spans across multiple multi-modal domains, constraining development in:
- Robotic Manipulation: Where robust grasp detection requires large-scale datasets integrating both color and depth information (Christopher Kanan et al.).
- Human and Crowd Analysis: Where small participant pools limit the generalization of gaze estimation models (Weixin Luo) and crowd localization networks (Dongze Lian).
- In-Hand Pose Estimation: Where a lack of visual-tactile datasets restricts models from learning how to combine touch and sight during occlusions (Yun Liu et al.).
- Autonomous Driving: Where small sample sizes in multimodal driving datasets limit model robustness across varied real-world environments (Wenshuo Wang et al.).
The Scale Revolution Across Modalities
To unlock true end-to-end multi-modal learning, researchers have focused on building massive, diverse benchmarks that mirror the scale of monocular datasets. By providing millions of annotated frames, these benchmarks allow deep architectures to learn robust cross-modal representations offline.
RGB + Depth (RGB-D)
To move beyond short, indoor sequences, Song Yan et al. introduced DepthTrack, a dataset designed to force networks to learn deep geometric features directly from depth maps rather than relying on RGB heuristics. This was expanded by Tianyang Xu et al., who released RGBD1K, a benchmark featuring 1,050 sequences and 2.5 million frames, proving that large-scale offline training on real depth data drastically boosts tracking robustness in complex scenarios.
RGB + Thermal (RGB-T) and Multi-Spectral
For 24-hour surveillance and all-weather tracking, models must fuse visible light with thermal signatures. The authors of Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline introduced VTUAV, providing 1.7 million high-resolution frame pairs to train complex neural architectures. For multi-object scenarios, Yabin Zhu et al. introduced VT-MOT, featuring nearly 4 million annotation boxes to prevent models from failing in low light or haze.
To address the operational reality of multi-camera networks, Huijie Fan et al. released M3Track, the first large-scale multi-modality dataset for tracking targets across different camera views in both day and night conditions. This scaling pattern is also mirrored in multi-spectral vehicle re-identification, where Chenglong Li et al. introduced spatially aligned datasets across multiple infrared bands to capture structural vehicle complexities.
RGB + Near-Infrared (RGB-NIR)
In real-world surveillance, cameras often switch from RGB to NIR mode mid-stream when light drops. Traditional datasets provided parallel streams, which do not reflect this single-camera operational reality. To bridge this gap, Chenglong Li 0002 et al. introduced CMOTB, a large-scale benchmark specifically designed for cross-modal tracking where the modality changes mid-stream.
RGB + Event Cameras
Event cameras capture asynchronous pixel-level intensity changes, offering immense potential for high-speed and low-light vision. However, algorithms have struggled to exploit them due to a lack of aligned data. Jiqing Zhang et al. introduced the FE108 dataset to combine conventional frames and event streams for degraded conditions, while Xiaoyu Shi et al. developed BlinkVision, a massive synthetic benchmark with dense annotations across 410 object categories to advance long-term point tracking and scene flow.
Traditional Paradigm (Data-Hungry Fusion):
[RGB Data] -----\
+---> [Multi-Modal Fusion Module] ---> [Overfitting / Drift]
[Auxiliary] ----/ (Trained on small datasets)
Visual Prompting Paradigm (ProTrack):
[Auxiliary] ---> [Modality "Dyeing"]
|
v
[Visual Prompt]
|
v
[RGB Data] -------> [Blended Input] ---> [Frozen RGB Tracker] ---> [Robust Target]
(No extra training needed)
Algorithmic Strategies to Bypass Scarcity
While scaling datasets provides a long-term solution, researchers have simultaneously designed architectures that inherently mitigate data scarcity by altering how modalities interact.
Visual Prompting
Instead of training a complex multi-modal fusion module from scratch on limited data, Jinyu Yang et al. proposed ProTrack, a visual prompting paradigm. ProTrack "dyes" auxiliary modalities (like depth, thermal, or event data) and blends them directly into the input image space. This allows a frozen, state-of-the-art RGB tracker to process multi-modal data without any architectural modifications or extra training, completely bypassing the data-hungry fusion bottleneck.
Challenge-Aware and Prototype Representations
When training data is limited, standard models struggle to learn specialized representations for rare environmental hurdles. Chenglong Li et al. addressed this in their Challenge-Aware RGBT Tracker (CAT) by splitting the network into specialized branches. By separating "modality-shared" challenges (like occlusion) from "modality-specific" ones (like poor lighting), the network learns to transfer information efficiently between sensors without requiring massive datasets for every specific hurdle.
Furthermore, to handle the sudden "appearance shock" when surveillance cameras switch modes, Chenglong Li et al. developed ProtoTrack. Instead of relying on extensive cross-modal training to bridge the appearance gap, ProtoTrack maintains a dynamic "multi-modal prototype"—a hybrid template combining a stable first-frame sample with dynamically updated representative samples from each modality. This architectural constraint ensures tracking stability even when specialized cross-modal training data is highly constrained.
Key Foundational Work
Tracking Revisited Using RGBD Camera: Unified Benchmark and Baselines (2013) — contributed 1 concept elements to this direction.
Robotic Grasp Detection using Deep Convolutional Neural Networks (2016) — contributed 1 concept elements to this direction.