MUOT_3M: A 3 Million Frame Multimodal Underwater Benchmark and the MUTrack Tracking Method

Ahsan Baidar Bakht, Mohamad Alansari, Muhayy ud Din, Jiale Cao, Sajid Javed, Irfan Hussain, Jiri Matas, Arif Mahmood

Abstract

Underwater Object Tracking (UOT) is crucial for efficient marine robotics, large-scale ecological monitoring, and ocean exploration; however, progress has been hindered by the scarcity of large, multimodal, and diverse datasets. Existing benchmarks remain small and RGB-only, limiting robustness under severe color distortion, turbidity, and low visibility conditions. We introduce MUOT_3M, the first pseudo-multimodal UOT benchmark comprising 3 million frames from 3,030 videos (27.8h) annotated with 32 tracking attributes, 677 fine-grained classes, and synchronized RGB, estimated enhanced RGB, estimated depth, and language modalities validated by a marine biologist. Building upon MUOT_3M, we propose MUTrack, a SAM-based multimodal-to-unimodal tracker featuring visual geometric alignment, vision-language fusion, and four-level knowledge distillation that transfers multimodal knowledge into a unimodal student model. Extensive evaluations across five UOT benchmarks demonstrate that MUTrack achieves up to 8.40% higher AUC and 7.80% higher precision than the strongest SOTA baselines while running at 24 FPS. MUOT_3M and MUTrack establish a new foundation for scalable, multimodally trained yet practically deployable underwater tracking.

Introduction

Underwater Object Tracking (UOT) is a nightmare for standard computer vision. Unlike terrestrial environments, the "submerged domain" suffers from severe color distortion (red light disappears first), extreme turbidity, and non-uniform illumination. Current AI trackers usually fail because they are trained on terrestrial data or small, RGB-only underwater datasets. There is a massive "data gap": we lack large-scale, diverse datasets that capture the geometric (depth) and semantic (language) complexities of the ocean. Furthermore, while multimodal sensors (like sonar or depth cameras) could help, they are often too expensive or bulky for the small autonomous underwater vehicles (AUVs) used in marine biology and exploration.

Comparison of raw underwater RGB frames with estimated depth maps and enhanced RGB frames, illustrating the challenges of the submerged domain.

The MUOT-3M Dataset

To address this, researchers have unveiled MUOT-3M, the largest and most diverse UOT dataset to date. It sets a new bar for the field with:

  • Scale: 3 million frames across 3,030 videos (roughly 28 hours of footage). This is three times larger than the previous state-of-the-art benchmark, WebUOT-1M.
  • Diversity: It covers 677 fine-grained species across 16 Phylum categories, all validated by an expert marine biologist.
  • Modalities: Every frame includes raw RGB, estimated enhanced RGB, estimated depth maps, and language annotations.
  • Attributes: Videos are tagged with 32 different tracking attributes (e.g., "Greenish Tint," "Bubbles," "Fast Motion") to help researchers understand exactly where models succeed or fail.
A comprehensive view of the MUOT-3M dataset, showcasing diverse species and the synchronized multimodal data provided for each frame, including expert-validated language annotations.

The MUTrack Method

Alongside the dataset, the authors introduced MUTrack, a real-time tracker that uses a "teacher-student" distillation strategy to learn from complex multimodal data while remaining efficient enough to run on standard RGB cameras. MUTrack is built on the Segment Anything Model 2 (SAM2) and uses a three-stage training pipeline:

  1. Multimodal Alignment: The model learns to align "pseudo-modalities"—estimated depth maps and enhanced RGB frames—with natural language descriptions provided by marine biologists.
  2. The Multimodal Teacher: A "Teacher" model is trained using all available modalities (RGB, enhanced RGB, depth, and language). This model becomes an expert at identifying targets despite underwater degradation.
  3. The Unimodal Student: To make the tracker practical for real-world deployment, the authors use Knowledge Distillation (KD). They train a "Student" model that only sees raw RGB video but is forced to mimic the internal representations and attention patterns of the expert Teacher.

By the end, you have a lightweight student model that "thinks" like it has depth and language sensors, even though it’s only looking at a standard video feed.

Key Results

  • Performance Boost: MUTrack achieved up to an 8.40% higher Success Rate (AUC) and 7.80% higher precision than the strongest existing baselines.
  • Real-Time Efficiency: Despite the complex training, the "Student" model runs at 24 FPS, making it viable for real-time deployment on AUVs and robotic platforms.
  • Generalization: When tested on datasets it had never seen before (cross-dataset evaluation), MUTrack consistently outperformed other trackers, suggesting it has truly learned the "language" of underwater motion.

Conclusion

MUOT-3M and MUTrack establish a new foundation for scalable, multimodally trained yet practically deployable underwater tracking. By leveraging pseudo-multimodality as supervision and the power of distillation, this work bridges the gap between high-fidelity training and low-power deployment in the challenging marine environment.

Summary updated

Abstract

Underwater Object Tracking (UOT) is crucial for efficient marine robotics, large scale ecological monitoring, and ocean exploration; however, progress has been hindered by the scarcity of large, multimodal, and diverse datasets. Existing benchmarks remain small and RGB only, limiting robustness under severe color distortion, turbidity, and low visibility conditions. We introduce MUOT_3M, the first pseudo multimodal UOT benchmark comprising 3 million frames from 3,030 videos (27.8h) annotated with 32 tracking attributes, 677 fine grained classes, and synchronized RGB, estimated enhanced RGB, estimated depth, and language modalities validated by a marine biologist. Building upon MUOT_3M, we propose MUTrack, a SAM-based multimodal to unimodal tracker featuring visual geometric alignment, vision language fusion, and four level knowledge distillation that transfers multimodal knowledge into a unimodal student model. Extensive evaluations across five UOT benchmarks demonstrate that MUTrack achieves up to 8.40% higher AUC and 7.80% higher precision than the strongest SOTA baselines while running at 24 FPS. MUOT_3M and MUTrack establish a new foundation for scalable, multimodally trained yet practically deployable underwater tracking.
BibTeX
@misc{bakht2026muot,
  author = {Ahsan Baidar Bakht and Mohamad Alansari and Muhayy ud Din and Jiale Cao and Sajid Javed and Irfan Hussain and Jiri Matas and Arif Mahmood},
  title = {{MUOT\_3M: A 3 Million Frame Multimodal Underwater Benchmark and the MUTrack Tracking Method}},
  year = {2026},
  eprint = {2602.18006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.18006},
}
Figure