Traditional computer vision pipelines separate light acquisition from data processing. An image sensor captures photons, digitizes them, and streams millions of raw pixels across a physical bus to an external processor, such as a CPU or GPU. This architecture introduces a severe communication bottleneck—often referred to as the "memory wall"—where the energy and latency required to move data dwarf the cost of the computation itself. Furthermore, conventional cameras route raw data through an Image Signal Processor (ISP) designed to optimize images for human aesthetics (via demosaicing, color correction, and sharpening), a step that is computationally expensive and largely redundant for machine vision algorithms.
To bypass this bottleneck, researchers have developed architectures that shift the boundary between sensing and computing. By embedding computational elements directly into an image sensor's pixel array or its monolithic logic layer, the sensor can perform early feature extraction before the data ever leaves the chip. Instead of transmitting a massive grid of raw pixels, an intelligent sensor outputs highly compressed, task-relevant representations—such as edge maps, sparse keypoints, or neural network feature embeddings.
Why It Works: Bandwidth Reduction at the Source
Focal-plane processing achieves its extreme energy efficiency through two primary mechanisms: massive data reduction at the point of capture and low-power analog or near-sensor digital computation. Moving data off-chip is highly energy-intensive; by executing data reduction directly within the sensor hardware, systems minimize the volume of transmitted data.
For example, Datta et al. demonstrate a "Processing-in-Pixel-in-Memory" ($P^2M$) paradigm in A processing-in-pixel-in-memory paradigm for resource-constrained TinyML applications that embeds the initial layers of a convolutional neural network (CNN) directly into a CMOS sensor, reducing data transfer bandwidth by over twentyfold.
This reduction can also be achieved optically or through sparse sampling patterns:
- Optical Computation: Chen et al. in ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels use bio-inspired angle-sensitive pixels to compute the first layer of a CNN optically before the data leaves the sensor, reducing sensor power and data bandwidth by 90% by digitizing only sparse edge data.
- Hardware-Level Sparse Sampling: In a single-author paper, Feng demonstrates in BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling that performing motion detection and random sampling directly within the sensor hardware allows a system to discard 95% of pixel data, yielding an 8.2× energy reduction and 1.4× lower latency.
Exploiting Analog Domain Efficiency
Instead of converting every pixel to a digital value before processing, many architectures exploit the physics of the sensor substrate to perform computations in the analog domain. This eliminates the power overhead of running high-speed Analog-to-Digital Converters (ADCs) on irrelevant pixel data.
In In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision, Datta et al. utilize existing correlated double sampling (CDS) circuits within the pixel array to perform additions and subtractions for the first layer of a Spiking Neural Network (SNN), reducing data bandwidth by up to 96x.
To minimize the physical area overhead of adding processing elements to pixels, Wang et al. show in A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture that sharing transistors across neighboring pixel units maintains a high fill-factor while achieving massive energy efficiency (up to 11.65 TOPS/W).
Bypassing the Human-Centric ISP
Autonomous systems do not require visually pleasing images to execute tasks like object recognition or tracking. By bypassing traditional image signal processing entirely, vision models can operate on "machine-native" interpretations of raw sensor data.
Bronstein et al. explore this concept in ISP Distillation, using knowledge distillation to train vision models directly on RAW camera data, eliminating the latency and power overhead of human-centric ISPs.
At the hardware level, the authors of Enabling ISPless Low-Power Computer Vision integrate analog signal summation directly into the sensor readout hardware using interleaved row-select lines. This mechanism performs basic demosaicing operations in the analog domain, reducing off-chip data bandwidth by up to 12x without sacrificing downstream classification accuracy.
Similarly, a single-author paper by Buckler, Reconfiguring the Imaging Pipeline for Computer Vision, proposes a reconfigurable imaging pipeline with a low-power vision mode that implements pixel subsampling and non-linear quantization directly into the sensor hardware, saving approximately 75% of imaging energy.
Retinomorphic Sensing and Parallel Architectures
Many focal-plane architectures draw inspiration from biological visual systems, such as the primate retina, which processes spatial and temporal changes locally before transmitting sparse spikes via the optic nerve.
Early work by Etienne-Cummings et al. in VLSI Model of Primate Visual Smooth Pursuit demonstrated the power of integrating phototransduction, edge detection, and motion sensing onto a single substrate to achieve stable, predictive target tracking. A subsequent single-author paper by Etienne-Cummings, VLSI Implementation of Motion Centroid Localization for Autonomous Navigation, showed that calculating spatial properties like the center of motion directly on the focal plane enables tracking systems to operate at milliwatt power levels (5mW) by bypassing energy-intensive data transfer steps.
Modern implementations scale this parallelism using Pixel Processor Arrays (PPAs) and monolithic neuromorphic chips:
- Pixel Processor Arrays: Bose et al. demonstrate in Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays that embedding both convolutional and fully connected layers directly into a PPA enables complete neural network inference at speeds exceeding 3,000 frames per second.
- Monolithic Neuromorphic Integration: Yao et al. in Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip present a fully asynchronous neuromorphic System-on-Chip (Speck) that monolithically integrates a dynamic vision sensor with a spiking neural network processor core, achieving real-time power consumption as low as 0.70 mW.
By combining sensing and processing into a single physical substrate, focal-plane processing addresses the fundamental physical limitations of traditional vision pipelines, enabling sub-millisecond latency and milliwatt-level power envelopes required for next-generation edge AI and robotics.
Key Foundational Work
VLSI Model of Primate Visual Smooth Pursuit (1995) — contributed 2 concept elements to this direction.