Visual Detection and Tracking System for a Spherical Amphibious Robot

Shuxiang Guo, Shaowu Pan, Liwei Shi, Ping Guo, Yan‐Lin He, Kun TangSensors 2017 · DOI

TL;DR

This paper presents a novel visual detection and tracking system specifically designed for small, spherical amphibious robots. By combining adaptive image enhancement, robust target detection, and efficient tracking with a Kalman filter, the system achieves real-time performance on a low-power System-on-Chip, making it suitable for ecological observation and surveillance in challenging underwater and terrestrial environments.

What Problem Does This Paper Address?

Amphibious robots, particularly small, spherical ones, are gaining traction for applications like ecological monitoring and surveillance in littoral regions due to their stealth and maneuverability. However, equipping these robots with effective vision systems presents significant challenges. These include:

  • Harsh Environments: Underwater conditions cause image degradation (light scattering, absorption), while amphibious scenarios introduce variable illumination and clutter (swaying plants, particles).
  • Robot Constraints: Small robots have limited payload capacity, power, and computational resources, making complex vision algorithms difficult to implement.
  • Real-time Performance: Detecting and tracking moving targets in dynamic environments requires high processing speeds to avoid missing crucial information.

Existing vision systems often struggle with these issues, being either too power-hungry, too computationally intensive, or not robust enough for the unpredictable nature of amphibious environments. This paper aims to bridge this gap by developing a vision system tailored to the unique constraints and demands of spherical amphibious robots.

Methods

The proposed system is built around a multi-stage approach integrated onto a low-power Xilinx XC7Z045 System-on-Chip (SoC) with an asymmetric and heterogeneous computing architecture.

  1. Image Enhancement: To combat underwater image degradation, the Multi-Scale Retinex with Color Restoration (MSRCR) algorithm is employed. This algorithm leverages multi-scale analysis and color constancy principles to enhance image visibility, contrast, and color fidelity, addressing issues like light scattering and absorption.
  2. Moving Target Detection: The Adaptive Gaussian Mixture Model (GMM) is used for detecting moving targets. GMM is chosen for its robustness to background jitter and its ability to handle complex environments better than simpler methods like frame differencing. It models each pixel as a mixture of Gaussian distributions to distinguish foreground (moving) objects from the background.
  3. Moving Target Tracking: A Fast Compressive Tracker (FCT) is utilized for tracking the detected target. FCT is known for its effectiveness and efficiency. To improve its performance in scenarios with potential target loss due to sampling limitations or low frame rates, a second-order Kalman filter is integrated. This filter predicts the target's position in the next frame, allowing the FCT to sample around the predicted location, thus enhancing robustness.
  4. Hardware Implementation: The entire system is implemented on a low-power SoC with a heterogeneous architecture. This involves a dual-core ARM processor running Linux for general robot functions and a bare-metal program on another core for real-time detection and tracking. Customized hardware accelerators on the programmable logic (FPGA) part of the SoC are used to offload computationally intensive tasks like image enhancement and the Naïve Bayes classifier for tracking, ensuring real-time performance within the power and resource constraints.

Datasets

The paper does not mention specific datasets used for training. However, for evaluation, they utilized:

  • Eight standard image sequences: CarDark, Trellis, David, Couple, Fish, Dog1, Sylvester, and ClifBar from the Visual Tracker Benchmark.
  • Two underwater videos of fishes: Provided by the Fish4Knowledge project.
  • Three sequences captured by the amphibious robot: One in a tank with a toy fish, and two underwater videos.
  • One sequence captured by the robot in a terrestrial environment: Featuring a small tracked robot.

Key Insights and Ideas

  • Adaptive Algorithms are Crucial: The success of the system hinges on using adaptive algorithms (MSRCR for enhancement, GMM for detection, FCT with Kalman for tracking) that can adjust to the dynamic and often challenging conditions of amphibious environments.
  • Heterogeneous Computing is Key for Resource-Constrained Robots: Integrating specialized hardware accelerators on an SoC is vital for achieving real-time performance in vision tasks on small robots with limited power and processing capabilities. This approach balances computational load effectively.
  • Kalman Prediction Enhances Robustness: Augmenting a fast tracker like FCT with a Kalman filter significantly improves its ability to handle target motion and potential frame drops, a common issue in real-world robotic applications.
  • First of its Kind: The paper claims to be the first practical visual detection and tracking system designed for generic targets on small-sized spherical amphibious robots, highlighting its novelty in this specific niche.

Key Results

  • Real-time Performance: The system achieves an average pre-processing rate of 20.8 fps, detection rate of 19.7 fps, and tracking rate of 56.3 fps.
  • Power Efficiency: The dynamic power consumption of the vision system is as low as 4.57 W, enabling over 2.5 hours of continuous operation.
  • Detection Accuracy: The proposed detection subsystem showed a 47.7% lower PWC (percentage of wrong classifications) and a 17.2% higher Pr (precision) on average compared to a GBM-based algorithm.
  • Tracking Performance: The proposed tracking subsystem achieved a high success rate (SR) and low center location error (CLE) across various test sequences, outperforming classical and some discriminative tracking algorithms, especially in underwater scenarios. For instance, it achieved 100% SR in several sequences.
  • Speedup: The customized accelerator for the Naïve Bayes classifier achieved a 4.1x speedup compared to software implementation.

Limitations and Future Work

  • MSRCR Adaptability: The MSRCR algorithm's parameters may need careful adjustment for different environments, limiting its adaptability to highly dynamic conditions.
  • Algorithm Robustness/Precision: The authors acknowledge that the detection and tracking algorithms, while improved, may not be sufficiently robust or precise for long-term robotic vision applications.
  • Future Work:
    • Focus on high-level vision applications like automatic navigation and object grasping.
  • Explore advanced visual algorithms, including convolutional neural networks (CNNs), to further enhance the system's capabilities.
  • Improve the adaptability of the image enhancement algorithm.

Summary updated

Abstract

With the goal of supporting close-range observation tasks of a spherical amphibious robot, such as ecological observations and intelligent surveillance, a moving target detection and tracking system was designed and implemented in this study. Given the restrictions presented by the amphibious environment and the small-sized spherical amphibious robot, an industrial camera and vision algorithms using adaptive appearance models were adopted to construct the proposed system. To handle the problem of light scattering and absorption in the underwater environment, the multi-scale retinex with color restoration algorithm was used for image enhancement. Given the environmental disturbances in practical amphibious scenarios, the Gaussian mixture model was used to detect moving targets entering the field of view of the robot. A fast compressive tracker with a Kalman prediction mechanism was used to track the specified target. Considering the limited load space and the unique mechanical structure of the robot, the proposed vision system was fabricated with a low power system-on-chip using an asymmetric and heterogeneous computing architecture. Experimental results confirmed the validity and high efficiency of the proposed system. The design presented in this paper is able to meet future demands of spherical amphibious robots in biological monitoring and multi-robot cooperation.
BibTeX
@inproceedings{guo2017visual,
  author = {Shuxiang Guo and Shaowu Pan and Liwei Shi and Ping Guo and Yan‐Lin He and Kun Tang},
  title = {{Visual Detection and Tracking System for a Spherical Amphibious Robot}},
  booktitle = {Sensors},
  year = {2017},
  doi = {10.3390/s17040870},
}
Figure