Adaptive Visual Face Tracking for an Autonomous Robot

Herke van Hoof, Tijn van der Zant, Marco WieringUniversity 2011

Abstract

Perception is an essential ability for autonomous robots in non-standardized conditions. However, the appearance of objects can change between different conditions. A system visually tracking a target based on its appearance could lose its target in those cases. A tracker learning the appearance of the target in different conditions should perform better at this task. To learn reliably, the system needs feedback. In this study, feedback is provided by a secondary teacher system that trains the tracker. The learning tracker is compared to a baseline non-learning tracker using data from an autonomous mobile robot operating in realistic conditions. In this experiment, the learning tracker outperforms the non-learning tracker. This shows that we successfully used the teacher system to train a visual tracking system on an autonomous robot.

Introduction

For an autonomous robot, maintaining visual contact with a target is a fundamental challenge. While identifying an object once is straightforward, tracking it as the robot traverses a dynamic environment is difficult. Standard visual trackers often rely on a fixed model of the target's appearance. However, as a robot moves from a controlled laboratory into a corridor with varying natural light, the target's visual signature—such as its color histogram—changes drastically.

If a tracker attempts to adapt too aggressively without external guidance, it risks "drifting," where it mistakenly begins tracking background elements like walls or furniture. This paper presents a "teacher-student" framework designed to allow real-time adaptation to changing lighting conditions without the risk of drift.

Methodology: The Teacher-Student Framework

The proposed architecture splits the tracking task between two systems:

  1. The Student (Particle Filter Tracker): This is a real-time system that represents the target's location as a set of hypotheses or "particles." Each particle is weighted by comparing the local image data to a 3D RGB color histogram of the target.
  2. The Teacher (Face Detector): A more robust but computationally expensive detector using Haar-like features (Viola-Jones).

The Learning Loop

The core innovation lies in the feedback mechanism. When the Teacher detects a face with high confidence, it captures a pair of models: a histogram of the face (target model) and a histogram of the surrounding scene (background model). These pairs are stored in a database.

A sample frame showing the target in a corridor environment, illustrating the features used for tracking.

When the Teacher is unavailable or too slow, the Student takes over. It analyzes the current background and uses the Bhattacharyya distance to find the most similar background in its database. It then retrieves and applies the associated target model. This allows the robot to "remember" how the target should look under specific environmental conditions it has previously encountered.

Experimental Setup and Datasets

The system was tested using a Pioneer 2 mobile platform. Data was collected across two distinct environments: a robot laboratory with stable artificial lighting and a corridor with large windows, which introduced significant natural light variability.

Montage showing the target person in three different environments: a lab (left), a corridor (middle), and near a window (right).

The training set consisted of 1,263 images, while the testing phase involved 18 separate paths. This variety ensured the tracker was challenged by both sudden and gradual changes in illumination.

Results and Discussion

The learning tracker demonstrated significant improvements over a non-learning baseline, outperforming it in 16 out of 18 test sequences. Statistical analysis via a binomial test confirmed these results were highly significant ($p < 0.005$).

Key findings include:

  • Contextual Adaptation: Using the background as a proxy for appearance proved highly effective. The system correctly assumed that if the environment looked familiar, the target's appearance would match previous records from that setting.
  • Robustness to Transitions: While the baseline tracker often lost the target immediately upon entering the corridor due to lighting shifts, the learning tracker successfully switched models to maintain its lock.
  • Correlation with Training Data: Performance was positively correlated ($r = 0.536$) with the number of successful detections during the training phase, highlighting the importance of a diverse training set.

Conclusion and Future Work

This study successfully demonstrates that a teacher-student framework can enable robust visual tracking in autonomous robots. However, limitations remain. The current instance-based learning approach leads to linear database growth; future iterations could employ clustering to manage memory. Additionally, integrating the robot's own motion commands into the tracking model and incorporating shape or texture features beyond simple color histograms would likely further enhance stability in complex environments.

Summary updated

Abstract

Perception is an essential ability for autonomous robots in non-standardized conditions. However, the appearance of objects can change between different conditions. A system visually tracking a target based on its appearance could lose its target in those cases. A tracker learning the appearance of the target in different conditions should perform better at this task. To learn reliably, the system needs feedback. In this study, feedback is provided by a secondary teacher system that trains the tracker. The learning tracker is compared to a baseline non-learning tracker using data from an autonomous mobile robot operating in realistic conditions. In this experiment, the learning tracker outperforms the non-learning tracker. This shows that we successfully used the teacher system to train a visual tracking system on an autonomous robot.
BibTeX
@inproceedings{hoof2011adaptive,
  author = {Herke van Hoof and Tijn van der Zant and Marco Wiering},
  title = {{Adaptive Visual Face Tracking for an Autonomous Robot}},
  booktitle = {University},
  year = {2011},
}
Figure