Traditionally, designing a physical system—like a camera lens, a radio antenna, or a photonic chip—was a separate process from designing the software that processes its data. Engineers would design a "perfect" lens to minimize aberrations, and computer scientists would then write algorithms to handle the resulting images.
A new research direction, often termed Deep Optics or Differentiable Physics, collapses this wall. By modeling physical wave propagation as a differentiable layer within a neural network, researchers can use backpropagation to optimize physical hardware and digital algorithms simultaneously. In this framework, the physical world becomes the first few layers of the neural network.
The Physical World as a Computational Graph
The core intuition behind this approach is that many physical processes, particularly wave propagation, can be described by linear or near-linear differential equations. If you can write a "forward model" of how light or radio waves travel through a medium, and if that model is differentiable, you can calculate how a tiny change in the hardware (like the thickness of a lens or the phase of a metasurface) affects the final output (like classification accuracy).
Gordon Wetzstein and Julie Chang pioneered this in Deep Optics for Monocular Depth Estimation and 3D Object Detection. Instead of designing a lens that produces a pretty picture for humans, they optimized a "coded" lens that produces a specific blur pattern that is easy for a neural network to decode into 3D depth. The lens isn't "better" by traditional standards, but the system is far more accurate.
Building a Digital Twin of Physics
To make this work, researchers create a "digital twin"—a high-fidelity simulation of the physics that lives inside a deep learning framework like PyTorch or TensorFlow.
- Differentiable Forward Models: Shane Colburn demonstrated in Inverse design and flexible parameterization of meta-optics using algorithmic differentiation that by embedding rigorous electromagnetic solvers (like RCWA) into an algorithmic differentiation pipeline, we can treat optical components as trainable weights.
- Wave Propagation as Layers: For diffractive systems, researchers like Xuhao Luo use the Rayleigh-Sommerfeld or Angular Spectrum methods to model light moving between layers. In Metasurface-Enabled On-Chip Multiplexed Diffractive Neural Networks in the Visible, these physical layers are optimized to perform image recognition at the speed of light, achieving massive neuron densities.
- Handling Non-idealities: Real-world hardware is messy. Mustafa Yildirim and Ilker Oguz showed in Nonlinear processing with linear optics that even "linear" systems can be trained to perform complex nonlinear computations by co-training the physical parameters with a digital twin.
Computing in the Air
This concept extends beyond light to the radio spectrum. In wireless communications, researchers are moving away from treating the environment as a "noise" to be overcome. Instead, they treat it as a programmable computer.
Guojun Huang and colleagues introduced Stacked Intelligent Metasurfaces (SIM) in Stacked Intelligent Metasurfaces for Task-Oriented Semantic Communications. Here, multiple layers of metasurfaces act as an "Electromagnetic Neural Network" (EMNN). As the radio waves pass through these layers, the physical structure itself performs the "math" of feature extraction and semantic encoding.
Similarly, Meng Hua and Deniz Gunduz proposed AirCNN (AirCNN via Reconfigurable Intelligent Surfaces), which uses reflective surfaces in a room to perform 2D convolutional operations "in the air" before the signal even reaches the receiver.
Why It Works: The Power of Joint Optimization
The reason this approach consistently outperforms traditional design is task-specificity.
In Real-time Terahertz Compressive Optical-Digital Neural Network Imaging, Shao‐Hsuan Wu and Shang-Hua Yang used 3D-printed diffractive layers as a "physical encoder" in a variational autoencoder. Because the hardware was trained alongside the digital decoder, the physical layers learned to compress the most important information for reconstruction, allowing high-resolution imaging with low-resolution sensors.
This joint optimization also provides robustness. Chao Qian showed in Performing optical logic operations by a diffractive neural network that by treating logic gates as a spatial classification task, the system becomes far more resilient to the phase and intensity fluctuations that typically break optical computers.
Beyond Vision and Radio
The "differentiable hardware" paradigm is now spreading to other domains where complex physical interactions are the bottleneck:
- Holography: Beichen Li and Wojciech Matusik use differentiable wave propagation to train networks that generate 3D holograms in real-time, pre-compensating for hardware artifacts (End-to-end learning of 3D phase-only holograms).
- Quantum Computing: Xiaotong Ni and researchers at Alibaba have applied these same gradient-based frameworks to jointly optimize quantum processor layouts and the control pulses that drive them (Integrating Quantum Processor Device and Control Optimization).
By making the physical world differentiable, we stop designing hardware to be "perfect" in a vacuum and start designing it to be the perfect partner for the digital intelligence that follows.