Traditional computer vision follows a rigid, energy-expensive pipeline: a sensor captures millions of raw pixels, converts them to digital signals, and moves them across a bus to a processor where a neural network extracts features. In this paradigm, the most significant cost isn't the math—it's the movement. Moving high-resolution data between the sensor and the processor creates an "electronic bottleneck" that limits the speed and battery life of edge devices like drones, AR glasses, and autonomous sensors.
Researchers are now collapsing this pipeline by moving computation directly into the sensing hardware. By performing feature extraction in the optical or analog domains before the data is even digitized, these systems can achieve orders-of-magnitude improvements in energy efficiency and latency.
The Data Movement Bottleneck
In a standard camera, the Image Signal Processor (ISP) spends significant energy making images look "pretty" for human eyes—adjusting colors, denoising, and sharpening. However, as Mark Buckler, Suren Jayasuriya, and Adrian Sampson argue in Reconfiguring the Imaging Pipeline for Computer Vision, computer vision algorithms don't need these aesthetics. By bypassing the traditional ISP and using a specialized "vision mode," they demonstrated that systems can save ~75% of imaging energy without losing accuracy.
The core intuition is dimensionality reduction at the source. If a sensor can filter out 95% of redundant data before it reaches the Analog-to-Digital Converter (ADC), the entire system becomes faster and leaner.
Computing with Light: Optical Encoders
The earliest point to process information is before it even hits the silicon. Using "meta-optics" or physical masks, researchers can perform the first layer of a neural network at the speed of light for near-zero energy.
- Angle Sensitive Pixels: Huaijin Chen and Ashok Veeraraghavan developed ASP Vision, which uses bio-inspired pixels to compute the first layer of a Convolutional Neural Network (CNN) optically. This reduces data bandwidth by 90%.
- Metasurface Encoders: In Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders, Maksim Makarenko and Bernard Ghanem show how AI-designed nanoscale surfaces can compress high-dimensional spectral data into low-dimensional signals. This allows a standard monochrome sensor to capture complex hyperspectral video at over 20 frames per second.
- Snapshot Compressive Imaging: Xin Yuan and colleagues in Snapshot Compressive Imaging: Theory, Algorithms, and Applications describe how physical masks modulate signals in the optical domain, allowing a 2D sensor to capture 3D temporal or spectral information at a fraction of the usual data rate.
Processing in the Analog Focal Plane
Once light is converted to an electrical charge, but before it is digitized, there is a window for high-speed analog computing. This "focal-plane processing" mimics biological systems like the primate retina.
Early work by Timothy K. Horiuchi and Christof Koch in Analog VLSI Circuits for Attention-Based, Visual Tracking demonstrated that implementing "winner-take-all" logic directly in hardware allows for real-time tracking of stimuli exceeding 2,000 pixels per second. Similarly, Ralph Etienne-Cummings implemented models of smooth pursuit directly at the focal plane to eliminate the communication bottleneck.
Modern approaches like the "Compute Sensor" by Sai Zhang and Naresh Shanbhag use charge-sharing circuits to perform feature extraction within the sensor's analog fabric. In Reducing the Energy Cost of Inference via In-sensor Information Processing, they achieved up to a 17x reduction in energy consumption by digitizing only low-dimensional features rather than raw pixels.
Processing-in-Pixel-in-Memory (P²M)
The most recent frontier involves embedding digital or mixed-signal neural network layers directly into the pixel array itself. This is often referred to as Processing-in-Pixel (PiP).
Gourav Datta and Souvik Kundu have pioneered the P²M-DeTrack architecture, which offloads the initial convolution and pooling layers of a CNN into the sensor. By transmitting compressed feature maps instead of raw pixels, they reduced data transfer bandwidth by 24x.
To further push efficiency, researchers like Kejie Huang have proposed reconfigurable convolution-in-pixel architectures using "pixel splicing," achieving computational efficiencies as high as 11.65 TOPS/W. Others, such as Md Abdullah-Al Kaiser, are exploring ADC-less designs using nanoscale magnetic tunnel junctions to act as binary neurons, further slashing power.
Selective Attention: Only Reading What Matters
If the goal is to save energy, the most efficient operation is the one you don't perform. Selective readout allows a sensor to skip redundant pixels entirely.
In Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping, Md Abdullah-Al Kaiser and Peter A. Beerel describe a system where a lightweight Vision Transformer predicts "important" regions in real-time. The sensor then power-gates redundant columns, saving 53% of energy by never reading the pixels that don't contain useful information.
The Path to Spatial AI
As Andrew Davison notes in FutureMapping, the evolution of "Spatial AI" for robotics and AR requires a move away from general-purpose CPUs toward a graph-based co-design of hardware and algorithms. By integrating sensing and computing—whether through neuromorphic chips like Speck or on-sensor optical flow accelerators for UAVs—the field is moving toward "smart pixels" that don't just see the world, but understand it at the point of capture.