Smart retina as a contour-based visual interface

Paul Nadrag, Antoine Manzanera, Nicolas Burrus2009

TL;DR

This paper introduces a vision-based input system that uses a "programmable retina" to efficiently detect moving object contours. By tightly integrating image acquisition and processing, the system adapts to varying lighting conditions and dramatically reduces data transfer, enabling a low-power, robust interface for applications like drawing or gesture recognition.

What Problem Does This Paper Address?

Creating intuitive and non-invasive human-computer interfaces is a significant challenge. Traditional input devices like mice and keyboards require physical manipulation, while other vision-based systems often demand users wear special equipment or are sensitive to lighting changes. The authors aim to develop a robust vision-based input device that can interpret user commands through natural hand movements, adapting to diverse environments and minimizing data processing overhead.

Methods

The core of the system is a programmable retina, a specialized vision chip that combines photoreceptors with processing units at each pixel. This allows for massively parallel, low-level image processing directly on the chip. Key techniques include:

  • Adaptive Acquisition: The retina dynamically adjusts its light sensitivity and exposure based on local contrast and scene dynamics. This is achieved by multiple readings of the photodiode at different time intervals, effectively creating "bit planes" that represent different gray levels.
  • Contour Detection: The system focuses on detecting contours of moving objects. This is done using morphological edge detection operators, which are well-suited for the binary processing capabilities of the retina.
  • Massive Data Reduction: By computing contours directly on the retina, the system significantly reduces the amount of data that needs to be transferred to an external processor (an ARM core in this case). This is crucial for low-power operation and real-time interaction.
  • Moving Contour Extraction: A temporal filtering approach is used to distinguish moving contours from static ones. This involves maintaining short-term and long-term history (STH and LTH) of contour presence at each pixel.
  • External Processing: A low-power ARM processor handles higher-level computations, such as analyzing the extracted moving contours to determine user commands (e.g., drawing, erasing, or doing nothing based on contour orientation and position).

Datasets

Not discussed in the paper. The system is demonstrated using real-time interaction with a user's hand movements.

Key Insights and Ideas

  • Tight Integration of Acquisition and Processing: The paper highlights the significant advantages of performing image processing directly on the sensor (the programmable retina). This allows for a much closer adaptation to the scene's properties than traditional separate camera-sensor pipelines.
  • Contour as the Primary Visual Feature: Focusing on contours, rather than full image data, is a clever way to achieve massive data reduction while retaining essential information for interaction.
  • Low-Power Vision: The system demonstrates that sophisticated visual interaction can be achieved with very low power consumption, making it suitable for embedded applications.
  • Robustness to Lighting: The adaptive acquisition mechanism allows the system to function effectively across a wide range of lighting conditions, a common challenge for vision systems.

Key Results

  • The system successfully extracts moving contours from video streams.
  • It demonstrates a functional drawing interface where user hand movements are translated into on-screen drawing actions.
  • The system can differentiate between "draw," "erase," and "do nothing" actions based on the orientation and position of the moving contour.
  • Execution times are reported: 10ms for the retina's processing, 8.6ms for the ARM processor's analysis, and 50ms for image capture (which is identified as the bottleneck).
  • Power consumption is very low, in the tens of milliwatts, orders of magnitude less than a PC.

Limitations and Future Work

  • Image Capture Bottleneck: The paper identifies image capture time as the main limitation for achieving higher frame rates.
  • False Detections: Due to the high sensitivity, some false contour detections occur, which are filtered by the external processor.
  • Future Work: The authors plan to investigate more complex interactions, including global hand posture estimation for 3D mouse control and tracking 3D models for complete arm/hand/finger posture recovery. They also aim to further develop the higher-level interpretation of moving contours.

Summary updated

Abstract

The purpose of this work is to provide a robust visionbased input device. In our system, a programmable retina is looking at the user who sends commands by moving his hand. The fusion between the acquisition and the processing functions of the retina allows a close adaptation to the lighting conditions and to the dynamic range of the scene. Thanks to its optical input and massive parallelism, the retina computes efficiently the contours of the moving objects. This feature has nice properties in terms of motion detection capabilities and allows a dramatic reduction in the volume of data to be output of the retina. An external low-power processor then performs global computations on the output data, such as extreme points or geometric moments, which are temporally filtered to generate a command. 1
BibTeX
@misc{nadrag2009smart,
  author = {Paul Nadrag and Antoine Manzanera and Nicolas Burrus},
  title = {{Smart retina as a contour-based visual interface}},
  year = {2009},
}
Figure