Processing in Pixel Arrays for Low Power Perception

84 concepts75 papers1 proposal

In traditional computer vision systems, the image sensor is a passive observer. It captures photons, converts them into analog electrical signals, and immediately hands them off to an Analog-to-Digital Converter (ADC). From there, millions of raw, multi-bit digital pixels are streamed across a physical bus to a separate processor (a CPU, GPU, or accelerator) where feature extraction and neural network inference finally begin.

This architecture suffers from what researchers call the "data tax." As highlighted by Gourav Datta et al. in A processing-in-pixel-in-memory paradigm for resource-constrained TinyML applications, the energy required to toggle physical wires and shuttle high-resolution raw data off-chip often dwarfs the energy consumed by the actual AI computation. Furthermore, power-hungry ADCs spend massive amounts of energy digitizing redundant background information before the system can even determine if the visual data is relevant.

To break this bottleneck, a paradigm shift has emerged: in-sensor and focal-plane computing. By embedding computational capabilities directly into the sensor's pixel array, these systems perform initial feature extraction, filtering, or compression before data crosses the chip boundary or undergoes full digitization. The sensor no longer outputs raw pixels; it outputs intelligence.


The Physics of Early Filtering: Optical and Analog Computing

The most intuitive way to eliminate the sensor-to-processor bottleneck is to perform computations before the data is even digitized. This can be achieved in either the photonic domain (using light) or the analog electronic domain (using raw pixel voltages).

Photonic Convolution

Instead of capturing a high-resolution image and using a digital processor to slide a convolutional kernel across it, the first layer of a neural network can be computed at the speed of light using passive optics.

Julie Chang et al. demonstrated this in Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification by placing an optimized diffractive optical element (DOE) in front of the sensor. The DOE acts as a physical phase mask that naturally convolves incoming light according to learned weights. Because this happens passively as light passes through the lens, it consumes zero power and reduces the downstream electronic workload by roughly 88%. Similarly, Huaijin Chen et al. utilized bio-inspired "Angle Sensitive Pixels" in ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels to extract low-level features like edges directly in the optical domain, slashing data bandwidth by 90%.

Analog In-Pixel Processing

If data makes it past the optics, it arrives at the pixel array as an analog charge. Traditional sensors read this charge out row-by-row. In-sensor architectures, however, perform mathematical operations directly within the pixel circuit.

In a single-author paper on extreme edge intelligence, Gourav Datta proposed replacing standard ADCs with nanoscale voltage-controlled magnetic tunnel junctions (Voltage controlled magnetic tunnel junction based ADC less processing-in-pixel for extreme edge intelligence). By performing "one-time-step" spiking convolutions inside the analog pixel array, the system bypasses traditional digitization for non-essential data, mitigating the primary energy drain of edge vision sensors.


Pixel Processor Arrays (PPAs) and Spatial Data Management

When computation moves directly into the digital or mixed-signal circuitry of the pixel array, the hardware is referred to as a Pixel Processor Array (PPA). In a PPA, each pixel or local cluster of pixels contains its own processing element and memory.

Laurie Bose demonstrated the extreme speed advantages of this approach in Visual Odometry for Pixel Processor Arrays, executing high-speed motion tracking entirely on the sensor at over 1,000 frames per second using only 2W of power. By keeping the entire visual odometry pipeline on the focal plane, the system eliminates the communication latency that typically cripples real-time robotic control.

Traditional Pipeline:
[Sensor] --(Raw Analog)--> [ADC] --(Massive Digital Stream)--> [Bus] --> [Processor]

In-Sensor Pipeline:
[Sensor + Analog/Optical Compute] --(Extracted Features/Spikes Only)--> [ADC/Bus] --> [Processor]

However, scaling deeper neural networks within PPAs introduces a unique challenge: spatial data management. As Laurie Bose et al. discovered in Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays, when a complete Convolutional Neural Network (CNN) is embedded into a PPA, the time and energy spent moving, resizing, and duplicating feature maps across the physical pixel grid can actually exceed the time spent on arithmetic operations. The bottleneck shifts from raw arithmetic throughput to internal spatial routing.


Biological Data Economy: Sparsity and Foveation

Human eyes do not stream uncompressed raw frames to the brain; they operate on principles of efficiency, transmitting data only when something changes or focusing resolution only where it matters. In-sensor computing heavily borrows these concepts.

Asynchronous Event-Driven Sensing

Dynamic Vision Sensors (DVS) mimic the mammalian retina by transmitting pixel data asynchronously, generating a "spike" or event only when a pixel detects a change in log-intensity. Man Yao et al. paired this sensing mechanism with a spiking neural network processor on a single chip named Speck (Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip). Because the architecture is fully asynchronous and event-driven, it achieves sub-millisecond latency and milliwatt-scale power consumption, processing data only when environmental stimuli trigger a response.

Learned In-Sensor Sparse Sampling

For tasks where continuous frame capture is required, such as eye tracking in AR/VR headsets, sensors can be designed to discard redundant data actively. Yu Feng et al. introduced this capability in BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling. Instead of a "read-all" paradigm, BlissCam performs motion detection and random sparse sampling directly within the sensor hardware, dropping up to 95% of the pixel data before it ever reaches the processor bus.


Vertical Integration and Co-Design

To physically realize these architectures without sacrificing the "fill-factor" (the light-sensitive area of a pixel), researchers rely on 3D integration technology.

As detailed by Nassim Abderrahmane et al. in the NimbleAI project (NimbleAI: Towards Neuromorphic Sensing-Processing 3D-integrated Chips), 3D integration allows designers to stack the sensing layer, memory elements, and neural processing units vertically. This vertical interconnectivity provides massive parallel bandwidth between the photodetectors and the processing elements, bypassing the traditional planar bus bottleneck.

Ultimately, maximizing the efficiency of these systems requires a tight hardware-software co-design. Frameworks like SplitNets, developed by Xin Dong et al. (SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems), use neural architecture search to jointly optimize a network's structure alongside its optimal partition point. By splitting the network intelligently between the smart sensor's focal plane and a central processor, the system minimizes data transmission volume while strictly respecting the tight memory and power constraints of the sensor hardware.

Key Foundational Work

Analog VLSI Cellular Implementation of the Boundary Contour System (1998) — contributed 1 concept elements to this direction.
2D Motion Vector Detection on Image Sensor Plane. (1998) — contributed 1 concept elements to this direction.
A New Image Sensor with Space Variant Sampling Control on a Focal Plane (1998) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 74 papers
  2. Version 1 · 2026-10-04 · 100 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_sensor_to_processor_data_bottleneck_c40336d7,
  title = {sensor-to-processor data bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensor-to-processor-data-bottleneck/rdr_c40336d743f2430c804b37ebf8fb007d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_sensor_to_processor_data_bottleneck_c40336d7_v2,
  title = {sensor-to-processor data bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensor-to-processor-data-bottleneck/rdr_c40336d743f2430c804b37ebf8fb007d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake