For decades, the design of physical imaging systems and the development of digital image processing algorithms were treated as entirely separate engineering disciplines. Optical engineers focused on creating the "perfect" lens—minimizing physical aberrations like chromatic distortion and spherical blur to project a pristine image onto a digital sensor. Once captured, software engineers wrote algorithms to clean up, compress, or extract semantic meaning from those pixels.
This decoupled paradigm is fundamentally suboptimal. A physical lens does not need to produce a visually pleasing image for a human observer if the ultimate consumer of that data is a machine learning algorithm. By forcing the hardware to eliminate all blur and distortion, traditional design discards valuable physical degrees of freedom.
Gordon Wetzstein et al. challenged this separation in Deep Optics for Monocular Depth Estimation and 3D Object Detection, introducing a framework where physical hardware and digital neural networks are optimized simultaneously, end-to-end. In this paradigm, the physical world acts as the first layer of a neural network. By co-designing the hardware and software, we can intentionally allow the hardware to introduce complex, deterministic distortions that encode high-dimensional information (such as depth, spectrum, or time) into a single 2D snapshot. A paired digital backend then effortlessly decodes this information, surpassing the physical limitations of traditional hardware.
The Physical Layer as a Trainable Encoder
To understand why this approach works, it helps to view the physical hardware as a physical "encoder" and the downstream neural network as a digital "decoder."
Consider a standard camera aperture. In a traditional setup, it is a simple circular opening. However, if we replace it with a custom phase mask or a coded aperture, we can alter how light strikes the sensor. Y. Inagaki et al. demonstrated this in Learning to Capture Light Fields Through a Coded Aperture Camera, treating the camera's optical mask as a layer of learnable weights. Instead of capturing a standard photo, the optimized mask multiplexes a complex 4D light field into a couple of 2D images, which the digital network reconstructs with high fidelity.
Physical World ---> [ Trainable Physical Layer ] ---> Digital Sensor ---> [ Neural Network ] ---> Task Output
(Lens, Mask, Pulse, etc.) (Decoder/Inference)
This co-design unlocks capabilities that are impossible with software or hardware alone:
- Computational Efficiency: Julie Chang et al. showed in Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification that an optimized diffractive optical element (DOE) can perform the first convolutional layer of a CNN purely through the physics of light propagation. This offloads massive computational burdens to the optical domain, consuming zero power and operating at the speed of light.
- Hardware-Level Privacy: Instead of capturing sensitive data and blurring it digitally, Carlos Hinojosa et al. in Learning Privacy-preserving Optics for Human Pose Estimation optimized a physical lens to selectively destroy identity-revealing features (like faces) at the hardware level while preserving the structural information necessary for downstream pose estimation networks.
- Breaking Dimensional Trade-offs: Standard monocular cameras lose metric depth due to scale ambiguity. Sachin Shah et al. resolved this in CodedVO: Coded Visual Odometry by using a learned phase mask that physically encodes distance directly into the image blur, allowing a single passive camera to perform visual odometry with high precision.
Passing Gradients Through Physical Laws
The core mechanism that makes this joint optimization possible is differentiable forward modeling. To train a physical parameter (like the thickness of a lens or the timing of an electronic shutter) using backpropagation, we must be able to compute the gradient of the final task loss with respect to that physical parameter. This requires translating the laws of physics into differentiable computational operators.
In optics, this means modeling light transport using wave optics or the angular spectrum method within standard deep learning frameworks. Josué Page et al. formalized this with WaveBlocks in Learning to Model and Calibrate Optics Via a Differentiable Wave Optics Simulator, implementing physical propagation as trainable operators. When the network experiences an error during training, the gradient flows backward through the digital layers, across the simulated sensor, and into the physical parameters of the lens or mask, reshaping its geometry.
This differentiable physics framework has proven highly versatile across different domains:
- Holographic Displays: Grace Kuo et al. in Multisource Holography and Beichen Li et al. in End-to-end learning of 3D phase-only holograms for holographic display integrated differentiable wave propagation models directly into the training loop. This allows the system to compute optimal phase-only patterns that pre-compensate for hardware-specific artifacts and speckle noise.
- Compressive Imaging: Edwin Vargas in Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems utilized a differentiable optical path simulator to jointly optimize time-varying masks and sensor shutter sequences, maximizing the information density of compressed snapshots.
- Beyond Optics: The concept generalizes to any physical acquisition chain. In ultrasound imaging, Ortal Senouf in Learning beamforming in ultrasound imaging modeled acoustic wave propagation as a differentiable pipeline to optimize physical transmit beam patterns. In medical resonance imaging, as reviewed by Hongjia Yang et al. in Artificial intelligence for neuro MRI acquisition: a review, treating the MRI physics chain as a differentiable program allows algorithms to directly optimize radiofrequency pulse sequences and sampling trajectories.
Bridging the Sim-to-Real Gap
A common failure mode of purely simulation-driven design is the "sim-to-real gap." An optimization algorithm might find a mathematically perfect lens profile in simulation, but real-world manufacturing tolerances, thermal expansion, alignment errors, and sensor noise will degrade its performance.
Differentiable co-design mitigates this issue through two primary mechanisms: learned digital compensation and hardware-in-the-loop optimization.
Learned Digital Compensation
Because the digital backend is trained alongside the hardware, it naturally learns to be robust against the specific physical limitations of the system. For instance, flat meta-optics traditionally suffer from severe chromatic aberrations across broad bandwidths. Rather than trying to engineer a physically perfect achromatic lens, Johannes E. Fröch et al. showed in Beating spectral bandwidth limits for large aperture broadband nano-optics that the metasurface can be optimized simply to ensure that high-frequency spatial information is preserved across the spectrum, even if the raw image looks completely blurred. The neural network backend, aware of this deterministic blur, easily restores full-color details.
Similarly, Luocheng Huang in Broadband thermal imaging using meta-optics designed hardware to produce a consistent, computable point spread function rather than a perfect image, relying on a backend deconvolution algorithm to handle the heavy lifting.
Hardware-in-the-Loop Optimization
When a physical system is too complex or non-linear to model analytically, researchers place the physical hardware directly into the optimization loop. In Auto-Tuning Structured Light by Optical Stochastic Gradient Descent, the authors bypassed analytical models by projecting patterns onto physical targets, measuring the real-world sensor response, and numerically estimating the system's Jacobian. This allows standard stochastic gradient descent to optimize illumination patterns while automatically absorbing real-world lens distortions and hardware non-linearities.
To maintain this alignment over time, Yanbing Liu et al. in Physics-Constrained Comprehensive Optical Neural Networks introduced a synchronized re-training loop. By updating physical optical weights and digital compensation parameters in tandem, the system dynamically corrects for real-world drift, such as laser instability and exposure mismatches.
When hardware components are completely non-differentiable (such as discrete registers in a commercial image signal processor), hybrid optimization strategies bridge the gap. Nicolas Robidoux demonstrated in End-to-End High Dynamic Range Camera Pipeline Optimization that combining zero-order evolutionary strategies for black-box hardware registers with first-order gradient descent for neural network weights enables successful end-to-end tuning of entire automotive imaging pipelines.
By treating physical hardware not as a fixed constraint, but as a flexible, programmable extension of the computational pipeline, differentiable co-design fundamentally redefines what imaging and sensing systems can achieve.