Distributed biologically based real time tracking in the absence of prior target information

T. Nathan Mundhenk, Jacob Everist, Chris Landauer, Laurent Itti, Kirstie L. BellmanProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2005 · DOI

TL;DR

This paper introduces a biologically inspired distributed system for real-time object tracking that doesn't require prior knowledge of target features. It leverages visual saliency and a novel H2SV color space to identify and track objects, enabling robust performance even with changing viewpoints and lighting.

What Problem Does This Paper Address?

Traditional object tracking systems often rely on pre-defined features of a target, requiring significant prior information. This makes them brittle, as they can fail if the target's appearance changes, if the environment is dynamic, or if an adversary tries to fool the system. The authors aim to overcome this limitation by developing a system that can discover and track unknown objects in real-time, mimicking how biological organisms can identify and communicate about novel objects without explicit prior training. This is particularly relevant for distributed systems where multiple agents (cameras) need to share information about dynamically appearing targets.

Methods

The core of the system is a two-tiered approach:

  • Complex Tracker (Saliency-based Discovery): This component uses a biologically inspired visual saliency model (from the iLab Neuromorphic Vision Toolkit - INVT) to identify "interesting" regions in a scene. It leverages gestalt principles like color opponency, continuity, and motion to detect salient features without prior knowledge. These salient features are then classified using a non-parametric classifier called NPclassify, which forms classes based on feature proximity and weighting, rather than a fixed number of predefined classes. This process generates "signatures" for detected objects.
  • Simple Tracker (Feature-based Tracking): Once the complex tracker identifies and characterizes an object with a signature, this signature (containing mean color values and standard deviations) is passed to a simpler, faster tracker. This simple tracker uses this information as a statistical prior to efficiently track the object. The system also incorporates adaptive color models, allowing the tracker to adjust to changes in lighting.

A key innovation is the H2SV color space, which transforms the standard HSV color space to address the "modulus" property of hue (where 0 and 360 degrees are both red). H2SV converts hue into two components (huel and hue2) that are bounded between 0 and 1, making them more amenable to statistical analysis and avoiding issues with bi-modal distributions. This transformation also mimics human red/green and blue/yellow color opponencies.

Datasets

Not discussed in the paper. The system is demonstrated using custom video sequences of a person entering/leaving a room and swinging balls.

Key Insights and Ideas

  • Saliency as a "Priorless" Discovery Mechanism: The system demonstrates that visual saliency can effectively serve as a mechanism to discover targets in a scene without any prior knowledge of their appearance. This shifts the paradigm from "what to look for" to "what is interesting to look at."
  • Meta-Priors for Flexible Learning: The paper frames the system's learning process within a spectrum of "meta-priors," from highly flexible (weak meta-priors) to highly certain (logical inductive priors). The complex tracker operates at the weak meta-prior end, allowing it to adapt and learn new object classes.
  • H2SV Color Space for Robustness: The H2SV color space is a significant contribution, providing a more stable representation of color that is less susceptible to lighting variations and avoids the statistical pitfalls of the standard HSV hue. Its resemblance to human color opponencies is also noteworthy.
  • Synergy of Complex and Simple Trackers: The combination of a discovery-oriented complex tracker and an efficient, prior-driven simple tracker creates a powerful system. The complex tracker handles novelty and classification, while the simple tracker provides speed and robustness for known (or recently discovered) targets.
  • Distributed Agent Communication: The viewpoint-invariant signatures generated by the complex tracker are designed to be shared among multiple camera agents, enabling a truly distributed tracking system.

Key Results

  • The system successfully tracked a person entering and leaving a room without prior knowledge of the person or the room's appearance.
  • It tracked two balls, even when they were touching or overlapping, demonstrating its ability to discriminate between objects.
  • The system achieved real-time performance, with saliency computation and signature extraction occurring at approximately 15 frames per second.
  • The H2SV color space was shown to improve tracking of objects with colors that would cause issues in standard HSV (e.g., purple shirts and skin tones).

Limitations and Future Work

  • Limited Identification/Memory: The current system's identification abilities are limited; it lacks long-term memory to store and re-identify old targets. Future work aims to address this by improving visual recognition speed, potentially by narrowing the inspection area.
  • Resource Allocation: Managing the allocation of simple trackers to detected objects is a challenge, especially in a system with potentially unlimited complex tracker detections but limited simple trackers.
  • System Complexity: Mixing different types of experts (trackers) can increase system complexity and the likelihood of unexpected behavior. Managing this complexity carefully is crucial.
  • Future Work: The authors suggest that the ability to narrow down the search area based on tracking information will aid in visual recognition. They also emphasize the potential for sharing information among distributed agents.

Summary updated

Abstract

We are developing a distributed system for the tracking of people and objects in complex scenes and environments using biologically based algorithms. An important component of such a system is its ability to track targets from multiple cameras at multiple viewpoints. As such, our system must be able to extract and analyze the features of targets in a manner that is sufficiently invariant of viewpoints, so that they can share information about targets, for purposes such as tracking. Since biological organisms are able to describe targets to one another from very different visual perspectives, by discovering the mechanisms by which they understand objects, it is hoped such abilities can be imparted on a system of distributed agents with many camera viewpoints. Our current methodology draws from work on saliency and center surround competition among visual components that allows for real time location of targets without the need for prior information about the targets visual features. For instance, gestalt principles of color opponencies, continuity and motion form a basis to locate targets in a logical manner. From this, targets can be located and tracked relatively reliably for short periods. Features can then be extracted from salient targets allowing for a signature to be stored which describes the basic visual features of a target. This signature can then be used to share target information with other cameras, at other viewpoints, or may be used to create the prior information needed for other types of trackers. Here we discuss such a system, which, without the need for prior target feature information, extracts salient features from a scene, binds them and uses the bound features as a set for understanding trackable objects.
BibTeX
@inproceedings{mundhenk2005distributed,
  author = {T. Nathan Mundhenk and Jacob Everist and Chris Landauer and Laurent Itti and Kirstie L. Bellman},
  title = {{Distributed biologically based real time tracking in the absence of prior target information}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2005},
  doi = {10.1117/12.631076},
}
Figure