IRIS: Integrated Retinal Functionality in Image Sensors

Zihan Yin, Md Abdullah-Al Kaiser, Lamine Ousmane Camara, Mark Camarena, Maryam Parsa, Ajey P. Jacob, Gregory W. Schwartz, Akhilesh JaiswalbioRxiv 2022 · DOI

TL;DR

This paper introduces IRIS, a novel neuromorphic image sensor technology that integrates complex retinal processing directly into the sensor hardware. By mimicking downstream retinal circuits beyond simple light adaptation and change detection, IRIS aims to enable more efficient and robust machine vision for dynamic environments.

What Problem Does This Paper Address?

Current machine vision systems, relying on conventional image sensors, often treat pixels as raw data points. While some neuromorphic sensors have mimicked early retinal functions like light adaptation and temporal differentiation, they largely overlook the sophisticated feature extraction performed by downstream retinal circuits. This gap means that valuable information about motion and other visual cues is lost or requires significant computational overhead to reconstruct. The authors aim to bridge this gap by developing an image sensor that directly implements these advanced retinal computations, leading to more efficient, high-speed, and low-bandwidth machine vision.

Methods

The IRIS technology is a technology-circuit co-design solution. The core idea is to implement specific retinal computations, namely Object Motion Sensitivity (OMS) and Looming Detection (LD), directly within the image sensor's circuitry.

  • Algorithmic Foundation: The paper first confirms the algorithmic understanding of OMS and LD by implementing them in software and testing them on video data.
  • Hardware Implementation: The core innovation lies in the hardware implementation using standard CMOS technology.
  • Bipolar Cell Mimicry: For both OMS and LD, the initial stage involves circuits that mimic bipolar cells, which act as differentiators, signaling changes in luminance. This is achieved using either Active Pixel Sensors (APS) or Dynamic Vision Sensors (DVS) pixels.
  • Amacrine and Ganglion Cell Mimicry: The paper then details the design of circuits that mimic the functionality of amacrine and ganglion cells to perform the OMS and LD computations.
  • OMS: The OMS circuit uses a center-surround receptive field architecture. It employs transistors that integrate signals from "center" pixels and others that drain charge based on signals from "surround" pixels. This differential processing allows for the detection of object motion relative to the background.
  • LD: The LD circuit uses pairs of transistors driven by ON and OFF bipolar signals. It's designed to detect expanding dark objects (or approaching bright objects) by looking for simultaneous decreases (or increases) in light intensity across the receptive field.
  • 3D Integration: The authors leverage advances in 3D chip stacking technology to integrate these complex circuits with the pixel array, similar to the layered structure of the biological retina.

Datasets

The paper mentions using "dashboard video segments from the Berkeley DeepDrive database (30)" for software simulations of the OMS and LD algorithms. No new datasets were introduced.

Key Insights and Ideas

  • Beyond First Synapse: The most significant insight is the move beyond mimicking only the initial stages of retinal processing (phototransduction and first synapse) to integrating computations from downstream retinal circuits (inner retina, amacrine, and ganglion cells).
  • Hardware Feature Extraction: Implementing feature extraction directly in the sensor hardware, rather than relying solely on post-processing, offers substantial advantages in speed, energy efficiency, and bandwidth.
  • Center-Surround for Motion: The successful implementation of the center-surround receptive field for OMS demonstrates how biological principles can be translated into effective electronic circuits for motion detection.
  • Looming Detection for Threat Avoidance: The LD circuit highlights the potential for specialized sensors to directly address critical tasks like collision avoidance in autonomous systems.
  • Leveraging 3D Integration: The paper underscores the critical role of 3D integration in enabling the complex, multi-layered architecture required for these retina-inspired sensors.

Key Results

The paper presents simulation results demonstrating the effectiveness of the proposed IRIS circuits:

  • Software Validation: The software implementations of OMS and LD algorithms successfully identified expected features in natural scene videos, such as runners in front of a car (OMS) and approaching cars (LD).
  • Circuit Functionality: Simulations of the hardware circuits on a Globalfoundries 22nm technology node show that the OMS and LD computations can be implemented.
  • The OMS circuit correctly generates feature-spikes for object motion by differentiating between center and surround signals.
  • The LD circuit generates feature-spikes indicating approaching objects.
  • Compatibility: The proposed designs are compatible with existing image sensor fabrication technologies.

Limitations and Future Work

  • Limitations:
    • The work focuses on simulating the circuits, not on fabricating and testing physical hardware.
  • The paper acknowledges that the aim is to functionally mimic retinal computations, not to replicate the detailed electro-chemical dynamics of biological neurons.
  • The current work implements only two specific motion computations (OMS and LD).
  • Future Work:
    • Broader Retinal Functionality: The authors suggest that similar circuit-design techniques can be used to embed a wider range of retinal functionalities, including color processing, object orientation, and shape detection.
  • Advanced Integration: Further exploration of 3D integration and heterogeneous sensor designs is proposed to co-integrate different wavelength sensitivities and implement more complex computations.
  • Non-Volatile Memory: The potential use of emerging non-volatile memory technologies for programmable weights in these circuits is mentioned.
  • Impact on Computer Vision: The paper emphasizes the potential for IRIS sensors to revolutionize computer vision by providing rich, context-aware feature data, thereby simplifying machine learning algorithms and improving decision-making speed and efficiency.
  • Hybrid Systems: IRIS cameras could generate both feature-spikes and standard intensity maps, offering flexibility for different vision algorithms.

Summary updated

Abstract

Abstract Neuromorphic image sensors draw inspiration from the biological retina to implement visual computations in electronic hardware. Gain control in phototransduction and temporal differentiation at the first retinal synapse inspired the first generation of neuromorphic sensors, but processing in downstream retinal circuits, much of which has been discovered in the past decade, has not been implemented in image sensor technology. We present a technology-circuit co-design solution that implements two motion computations occurring at the output of the retina that could have wide applications for vision based decision making in dynamic environments. Our simulations on Globalfoundries 22nm technology node show that, by taking advantage of the recent advances in semiconductor chip stacking technology, the proposed retina-inspired circuits can be fabricated on image sensing platforms in existing semiconductor foundries. Integrated Retinal Functionality in Image Sensors (IRIS) technology could drive advances in machine vision applications that demand robust, high-speed, energy-efficient and low-bandwidth real-time decision making.
BibTeX
@misc{yin2022iris,
  author = {Zihan Yin and Md Abdullah-Al Kaiser and Lamine Ousmane Camara and Mark Camarena and Maryam Parsa and Ajey P. Jacob and Gregory W. Schwartz and Akhilesh Jaiswal},
  title = {{IRIS: Integrated Retinal Functionality in Image Sensors}},
  year = {2022},
  doi = {10.1101/2022.08.14.503909},
  howpublished = {bioRxiv},
}
Figure