Robust Autonomous Structure-based Color Learning on a Mobile Robot

Mohan Sridharan

Abstract

Vision is a critical yet computationally expensive sensor for mobile robots, often hindered by the brittleness of color-based object recognition. Traditional manual calibration is time-consuming and fails under dynamic lighting. This paper presents a framework for mobile robots to autonomously learn color distributions by leveraging the known physical structure of their environment. By using a world model to "self-supervise" color calibration, the system reduces setup time from over an hour to just five minutes while maintaining robustness against the jerky motions and shifting illumination typical of real-world robotics.

Introduction

For mobile robots, vision is a "high-risk, high-reward" sensor. While images provide rich data, processing them is computationally expensive, and color-based object recognition is notoriously brittle. Traditionally, robots require a tedious manual calibration process where humans label pixel colors (e.g., "this pixel is 'ball-orange'") for every new lighting environment.

This approach fails in two primary ways: scalability and dynamic lighting. Manual labeling is impractical for robots deployed in changing environments, and a color map created under specific conditions often fails when lighting shifts nonlinearly, such as when a cloud passes over or the robot moves into a shadow. This research addresses these challenges by allowing the robot to learn what colors look like autonomously, using the objects it already knows should be present in its environment.

Methodology

The research is implemented on the SONY Aibo—a quadruped robot widely used in the early 2000s RoboCup soccer leagues. The methodology consists of three main components: a baseline vision system, structure-based learning, and illumination adaptation.

Baseline Vision System

The robot converts raw YCbCr camera input into color labels using a "color cube" (a 3D lookup table). It utilizes run-length encoding to group pixels into contiguous strips, which are then merged into bounding boxes for object recognition.

Structure-based Learning

This is the core innovation of the framework. Instead of a human labeling pixels, the robot is provided with a map of the field and a list of objects (e.g., goals, markers). The robot moves to known positions, calculates where an object should be in its field of view based on its geometry, and then samples the pixels in that area to build a 3D Gaussian model of that color.

The Sony Aibo robot on a RoboCup soccer field, illustrating the structured environment used for autonomous color learning.

Illumination Adaptation

To handle lighting changes, the robot maintains multiple color maps for different conditions (Bright, Dark, Intermediate). It periodically samples image statistics and uses Kullback-Leibler (KL) divergence—a measure of the difference between two probability distributions—to determine which pre-trained illumination model best matches the current environment and switches its color map accordingly.

Experimental Setup and Datasets

As a robotics paper from 2006, the study does not use standard modern computer vision datasets like COCO or ImageNet. Instead, the experimental data consists of:

  • Real-time image streams from the SONY Aibo’s CMOS camera.
  • Hand-labeled images used to establish a baseline for comparison against the autonomous learning method.
  • The structured environment of the RoboCup soccer field, which provides the ground-truth geometry (shapes and locations of goals, balls, and lines).

Key Results

The proposed system demonstrated significant improvements in efficiency and robustness:

  • Speed: The autonomous calibration process takes approximately 5 minutes, a massive improvement over the hour-plus required for manual calibration.
  • Accuracy: The segmentation performance of the autonomously learned color maps was found to be comparable to maps generated by manual human calibration.
  • Robustness: The system successfully transitioned between discrete lighting conditions (Bright, Dark, Intermediate) by selecting the appropriate color map based on real-time image statistics.

Discussion and Future Work

The most powerful insight from this work is that a robot’s prior knowledge of the world (the map) can effectively replace human labeling. Modeling colors as 3D Gaussians rather than static ranges allows the robot to handle noise and minor variations in shade more gracefully than simple thresholding.

However, limitations remain. The current system requires the robot to start from a known location to begin the learning process. Future work could explore learning colors from unknown starting positions. Additionally, while 3D Gaussians work well in controlled lab settings, they may not capture the complexity of colors in completely uncontrolled, natural outdoor lighting. Future iterations aim to allow the robot to "re-learn" on the fly to handle minor, continuous drifts in illumination rather than switching between discrete pre-set maps.

Summary updated

Abstract

Motivation The scientific community is working towards the creation of fully autonomous mobile robots capable of interacting with the proverbial real world. Autonomous robots are predominantly dependent on sensory information but the ability to accurately sense the complex world is still missing. Visual input, in the form of color images from a camera, should be an excellent and rich source of such information, considering the significant amount of progress made in machine vision. But color, and images in general, have been used sparingly on mobile robots, where people have mostly focussed their attention on other sensors such as sonar and laser. There are three main reasons for this reliance on other relatively low-fidelity sensors. First, most sophisticated vision algorithms require substantial amount of computational and/or memory resources making them infeasible to use on mobile robotic systems that typically have constrained memory and computational resources but demand real-time processing. Second, most vision algorithms assume a stationary or slowly moving camera and hence cannot account for the rapid non-linear camera motion that is characteristic of mobile robot domains. Third, the variation of illumination over the operating environment causes a nonlinear shift in color distributions that is difficult to model; mobile robots, while moving around the world, often go into places with changing illumination. Even in the face of
BibTeX
@inproceedings{sridharan2006robust,
  author = {Mohan Sridharan},
  title = {{Robust Autonomous Structure-based Color Learning on a Mobile Robot}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure