Visual Tracking of Deepwater Animals Using Machine Learning-Controlled Robotic Underwater Vehicles

Kakani Katija, Paul L. D. Roberts, Joost Daniels, Alexandra Lapides, Kevin Barnard, Mike Risi, Ben Y. Ranaan, Benjamin G. Woodward, Jonathan TakahashiWACV 2021

TL;DR

This paper introduces ML-Tracking, a novel algorithm that uses machine learning to enable underwater robots to autonomously track deep-sea animals for extended periods. By combining object detection with 3D stereo tracking and a sophisticated supervisor module, ML-Tracking allows for unprecedented long-duration observations, paving the way for new discoveries in unexplored ocean environments.

What Problem Does This Paper Address?

The deep ocean remains one of the least explored frontiers on Earth, with estimates suggesting that 95% of it is unexplored and 91% of marine life is unknown. The midwaters, in particular, present a vast, three-dimensional environment with no fixed landmarks, making observation and study incredibly challenging. Traditional methods often struggle to provide the persistent, non-disruptive tracking needed to understand animal behavior. Existing underwater tracking primarily relies on acoustics, which lack the high-resolution spatiotemporal data necessary for detailed identification and behavioral analysis. While vision-based tracking has advanced significantly, applying it to the dynamic, often transparent, and feature-poor environment of the midwater, especially for long durations, requires robust algorithms that can handle variable object sizes, multiple targets, and challenging visual conditions.

Methods

The core of the ML-Tracking system is a tracking-by-detection approach. This involves:

  • Multi-Object Detector: A RetinaNet model, specifically a ResNet50 backbone, is trained to detect multiple classes of midwater animals simultaneously. This is crucial because many deep-sea creatures share similar features, necessitating the ability to distinguish between them and ignore irrelevant objects.
  • 3D Stereo Tracker: This module takes the 2D bounding boxes from the detector and uses a stereo imaging system to determine the 3D position of the detected objects. It employs an unscented Kalman filter (UKF) for state estimation and incorporates stereo intersection over union (IOU) and the Hungarian algorithm for robustly finding matching stereo pairs and associating them with existing tracks.
  • Supervisor Module: This intelligent component oversees the entire tracking process. It evaluates the quality of detected targets, manages different operational modes (search, acquire, confirm, track), and modifies the vehicle's behavior (e.g., auto-depth, heading, thruster control) based on the tracking status and target identification. This module is key to enabling autonomous, long-duration missions.

A novel aspect is the adaptation of pre-trained ResNet50 models for monochrome imagery by simply replicating the single channel to create a three-channel image, which proved sufficient for fine-tuning.

Datasets

The paper utilizes two primary sources for training data:

  • VARS Database: Expertly annotated color imagery from previous ROV dives at the Monterey Bay Aquarium Research Institute (MBARI).
  • FathomNet: A subset of this underwater image training set was used.
  • ROV MiniROV Dives: Monochrome imagery collected during midwater dives using the stereo camera system was also incorporated to augment the training data.

The training data was annotated by experts using tools like GridView, RectLabel, and Tator, resulting in 28,485 localized images across 17 different classes.

Key Insights and Ideas

  • Tracking-by-Detection for the Deep Sea: Applying the tracking-by-detection paradigm, which has seen success in other domains, to underwater robotics for animal tracking is a significant step. It moves beyond simple detection to robust, continuous observation.
  • Multi-Class Detection is Essential: The ability to simultaneously detect and differentiate between multiple types of animals is critical in the visually complex and often homogeneous midwater environment. This prevents misidentification and allows for more targeted tracking.
  • Stereo Vision for 3D Localization: For unknown-sized targets in a 3D space, stereo imaging is indispensable for accurate depth estimation, which is a fundamental requirement for effective tracking and navigation.
  • Supervisor Module for Autonomy: The supervisor module is the "brain" that translates detection and tracking data into actionable vehicle commands, enabling autonomous mission phases and adaptive behavior. This is crucial for long-duration, hands-off operations.
  • Monochrome Image Adaptation: The simple yet effective method of replicating monochrome channels for pre-trained color models significantly broadens the applicability of existing deep learning architectures to underwater imaging.

Key Results

The ML-Tracking algorithm demonstrated impressive performance across various challenging use cases:

  • High Control Percentage: For all tested scenarios, the vehicle controller received 3D position information from ML-Tracking for over 99% of the time.
  • Precise Tracking: For most use cases (1-4), the standard deviation for range and altitude was less than 5 cm, and bearing deviation was less than 6 degrees, indicating highly stable tracking.
  • Long Duration Observation: A record 5+ hour continuous observation of a siphonophore was achieved, showcasing the algorithm's capability for extended missions.
  • Robustness to Occlusion: While generally robust, the paper notes that occlusion by objects of the same class was identified as a failure point, which will be addressed in future work.
  • Qualitative Success: Framegrabs and supplementary videos (not directly viewable here but referenced) demonstrate successful tracking across various scenarios, including single-object steady/dynamic swimming, nested classes, and multi-object occlusions.

Limitations and Future Work

  • Limitations:
    • COVID-19 Delays: Field deployments of the most recent iteration were unfortunately delayed.
  • Single-Class Occlusion: The algorithm struggles when objects of the same class occlude each other, leading to catastrophic tracking failure.
  • Transparency and Reflectance: Detecting highly transparent objects and those with variable reflectance properties remains challenging, impacting detector performance for certain classes.
  • Future Work:
    • Improved Robustness: Addressing single-class occlusion and enhancing detection of challenging targets.
  • Integration with Acoustic Sensing: Combining visual tracking with acoustic sensing for more comprehensive tracking solutions, especially in scenarios where vision is limited.
  • Autonomous Acquisition: Developing fully automated target acquisition capabilities to streamline the process of finding and initiating tracking of new objects.
  • Supervised Autonomy: Implementing external communication interfaces to allow human operators to periodically review and confirm targets, enhancing reliability and enabling faster model development.
  • Seafloor Applications: Adapting the ML-Tracking framework for surveying and observing objects on the seafloor, opening up new avenues for benthic exploration.

Summary updated

BibTeX
@inproceedings{katija2021visual,
  author = {Kakani Katija and Paul L. D. Roberts and Joost Daniels and Alexandra Lapides and Kevin Barnard and Mike Risi and Ben Y. Ranaan and Benjamin G. Woodward and Jonathan Takahashi},
  title = {{Visual Tracking of Deepwater Animals Using Machine Learning-Controlled Robotic Underwater Vehicles}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}

Loading related content...

Figure