Event-Based Color Segmentation With a High Dynamic Range Sensor

Alexandre Marcireau, Sio-Hoï Ieng, Camille Simon-Chane, Ryad BenosmanFrontiers in Neuroscience 2018 · DOI

TL;DR

This paper introduces a novel event-based color camera system that captures color information at microsecond temporal resolution, enabling computationally efficient color segmentation and object tracking. By leveraging the high dynamic range and asynchronous nature of event-based sensors, this approach overcomes limitations of traditional frame-based cameras for real-time applications.

What Problem Does This Paper Address?

Traditional computer vision systems often struggle with real-time color segmentation and object tracking, especially in challenging lighting conditions due to the high computational cost of processing dense, frame-based visual data. While event-based cameras offer advantages in terms of speed and efficiency by only reporting changes, existing color event-based sensors have been limited to detecting relative color changes rather than absolute luminance. This paper addresses the need for an event-based system that can capture and process absolute color information at high temporal resolutions, paving the way for more efficient and robust color-aware machine vision.

Methods

The core of this work is a custom-built three-chip event-based color camera. This system combines three Asynchronous Time-based Image Sensors (ATIS), each sensitive to a different color band (red, green, and blue). Light is split using dichroic mirrors and band-pass filters to direct specific wavelengths to each ATIS sensor.

The key innovation lies in how color events are generated:

  • Absolute Luminance Capture: Unlike previous event-based color sensors, this system captures absolute luminance values.
  • Asynchronous Event Generation: Each ATIS pixel operates independently, generating events only when luminance changes significantly. This results in sparse data, reducing computational load.
  • Color Event Merging: Since each ATIS sensor outputs events independently, a mechanism is devised to merge these asynchronous color-channel events into a single color event (pixel index, timestamp, R, G, B values). This is achieved by maintaining a memory buffer for each pixel to store the latest color component values.
  • Color Model and Segmentation: The captured RGB values are converted to the perceptually uniform CIELab color space. A bivariate normal distribution* is used to model the color signature of objects. This statistical model is robust and computationally inexpensive.
  • Object Tracking: A moving mean algorithm is employed for real-time object tracking. This algorithm updates an object's estimated position based on incoming color events, minimizing computational requirements. A spatio-temporal activity filter is used to reduce the impact of camera noise.

Datasets

The paper does not explicitly mention training datasets. Instead, it relies on:

  • Macbeth ColorChecker: Used for calibrating the sensor and evaluating the color conversion model from RGB to CIELab*.
  • Indoor Controlled Scene: Five colored wooden pieces on a rotating surface.
  • Outdoor Uncontrolled Scenes:
    • A scene with a moving camera capturing a red road sign and a green pharmacy neon light.
  • A scene with a static camera recording pedestrians wearing colored sweaters.

These scenes serve as benchmarks for evaluating the color segmentation and tracking performance of the proposed system.

Key Insights and Ideas

  • Absolute Color in Events is Feasible and Beneficial: The paper demonstrates that capturing and processing absolute color information from event-based sensors is achievable and leads to computationally efficient color segmentation.
  • Leveraging Temporal Resolution for Efficiency: The microsecond-level temporal resolution of event-based sensors, combined with sparse data, allows for simple, high-speed algorithms to achieve complex tasks like object tracking.
  • CIELab* for Robust Color Modeling: Using the CIELab* color space for statistical modeling (bivariate normal distribution) provides a perceptually uniform and robust way to represent object colors, making segmentation less sensitive to minor color variations.
  • Moving Mean for Efficient Tracking: The moving mean algorithm proves to be a surprisingly effective and computationally light method for tracking objects based on color events, especially when combined with spatio-temporal filtering.
  • High Dynamic Range is Crucial: The inherent high dynamic range of ATIS sensors is highlighted as a significant advantage, allowing the system to handle scenes with extreme variations in illumination without losing detail.

Key Results

  • The prototype event-based color camera successfully performs color segmentation and tracking in various indoor and outdoor scenes.
  • Tracking performance in outdoor scenes showed mean errors ranging from two to twenty pixels compared to ground truth.
  • For the wooden pieces experiment, mean position estimation errors were reported as: Blue sign (1.6%), Green sign (3.3%), Purple sign (10%), Orange sweater (4.7%), Yellow sign (19%).
  • The system demonstrates the ability to track objects using only color information, a significant step for event-based vision.
  • The algorithm is computationally cheap, requiring little processing power, making it suitable for embedded systems.

Limitations and Future Work

  • Sensitivity to Blue Wavelengths: The ATIS camera used in the prototype exhibits lower sensitivity to blue wavelengths, leading to longer integration times and potential timestamp differences between color channels, which can affect color reconstruction accuracy.
  • Optical Aberrations: The current hardware setup, with its large electronics boards, leads to optical aberrations (like color fringing and vignetting) near the edges of the sensor's field of view, degrading tracking performance for objects in these regions.
  • Spatial Information Ignored: The current work focuses on color and temporal information. The authors suggest that incorporating spatial information (e.g., through data fusion) could lead to more robust algorithms.
  • Hardware Improvements: Future work could involve designing dedicated electronics boards to reduce optical aberrations and exploring sensors with improved sensitivity to all wavelengths.
  • New Chip Design: The authors propose that a new chip design, potentially incorporating analog data transfer between pixels (similar to biological vision), could further improve color information acquisition efficiency.

Summary updated

Abstract

This paper introduces a color asynchronous neuromorphic event-based camera and a methodology to process color output from the device to perform color segmentation and tracking at the native temporal resolution of the sensor (down to one microsecond). Our color vision sensor prototype is a combination of three Asynchronous Time-based Image Sensors, sensitive to absolute color information. We devise a color processing algorithm leveraging this information. It is designed to be computationally cheap, thus showing how low level processing benefits from asynchronous acquisition and high temporal resolution data. The resulting color segmentation and tracking performance is assessed both with an indoor controlled scene and two outdoor uncontrolled scenes. The tracking's mean error to the ground truth for the objects of the outdoor scenes ranges from two to twenty pixels.
BibTeX
@inproceedings{marcireau2018event,
  author = {Alexandre Marcireau and Sio-Hoï Ieng and Camille Simon-Chane and Ryad Benosman},
  title = {{Event-Based Color Segmentation With a High Dynamic Range Sensor}},
  booktitle = {Frontiers in Neuroscience},
  year = {2018},
  doi = {10.3389/fnins.2018.00135},
}
Figure