Physics-Aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks

Yingjie Li, Ruiyang Chen, Weilu Gao, Cunxi Yu

TL;DR

This paper introduces a hardware-software codesign framework that bridges the gap between high-level AI training and the messy reality of physical optical hardware. By using the Gumbel-Softmax trick to make discrete physical device responses differentiable, the authors enable high-accuracy Diffractive Optical Neural Networks (DONNs) even when using low-precision, non-uniform optical components.

What Problem Does This Paper Address?

Diffractive Optical Neural Networks (DONNs) promise incredible speed and energy efficiency by performing computations using light diffraction. However, there is a major "reality gap":

  1. Digital vs. Physical: We train models on computers using continuous floating-point numbers, but physical devices like Spatial Light Modulators (SLMs) operate on discrete voltage levels.
  2. Non-Ideal Responses: Real-world optical devices don't have linear or monotonic responses. A specific voltage might change both the phase and the amplitude of light in complex, non-uniform ways.
  3. Gradient Breakage: Standard quantization (rounding weights) breaks the "gradient chain" needed for backpropagation, leading to accuracy drops of up to 30% when moving from simulation to a real physical system.

The authors aim to create a training framework that is "physics-aware," meaning it knows exactly how the hardware behaves and optimizes the model specifically for those constraints.

Methods

The core innovation is the use of Gumbel-Softmax (GS) to enable differentiable discrete mapping:

  • Discrete-to-System Mapping: Instead of training continuous weights and then rounding them, the framework treats the selection of a discrete voltage level as a categorical probability.
  • Differentiable Approximation: Gumbel-Softmax allows the system to "sample" from these discrete levels during the forward pass while maintaining a differentiable path for the backward pass (backpropagation).
  • Temperature Scheduling: The authors use a "temperature" parameter ($\tau$). Early in training, a high $\tau$ allows the model to explore different discrete levels (coarse search). As training progresses, $\tau$ is annealed (lowered), forcing the model to commit to specific hardware voltage levels.
  • Physics Integration: The model incorporates the actual measured phase and amplitude responses of the SLMs, ensuring the "neurons" in the simulation behave exactly like the pixels in the lab.

Datasets

The framework was evaluated using two standard computer vision benchmarks, adapted for optical resolution (interpolated to $200 \times 200$ pixels):

  • MNIST: Handwritten digits.
  • FashionMNIST (FMNIST): Images of clothing items. The authors also utilized a custom dataset of experimentally measured device responses to characterize their SLMs.

Key Insights and Ideas

  • Linear is King: After testing various temperature schedules (Cosine, Step, Static), the authors found that Linear temperature decay provided the most stable and efficient convergence for DONNs.
  • Reviving Shallow Networks: Previous literature suggested that shallow DONNs ($\leq 3$ layers) were fundamentally limited by physics. This paper shows that much of that "limitation" was actually poor training; their GS-based approach significantly boosted the performance of shallow architectures.
  • Physical Feasibility: Traditional quantization methods like Weight Sharing Quantization (WSQ) often produce "illegal" negative values that must be truncated, tanking accuracy. Because the GS approach is aware of the hardware's physical limits from day one, it avoids this "post-processing" penalty entirely.

Key Results

  • High Accuracy: The framework achieved 98% accuracy on MNIST and 89% on FMNIST using 8-level discrete devices.
  • Superiority over Quantization: While standard quantization methods (like WSQ) saw accuracy drops of over 27% after being forced into physical constraints, the GS-based method maintained its high performance.
  • Experimental Validation: The authors successfully deployed their trained models onto a physical 3-layer visible-range DONN system. The experimental results closely matched the simulations, proving the framework's real-world utility.
  • Robustness: Deeper networks (5 layers) trained with this method showed high "confidence" and were significantly more robust to noise and errors compared to single-layer systems.

Limitations and Future Work

  • Calibration Overhead: The framework relies on precise experimental measurements of each SLM. If a device's response drifts over time or varies significantly across its surface, the model might need frequent re-calibration.
  • Scalability of Discrete Levels: While 8-level devices were tested, the computational cost of the Gumbel-Softmax approach increases with the number of discrete levels (classes) available in the hardware.
  • Complex Non-linearities: The current work focuses on linear diffraction and phase/amplitude modulation. Future work could explore integrating non-linear optical activation functions into this differentiable codesign framework.

Summary updated

Abstract

Diffractive optical neural networks (DONNs) have attracted lots of attention as they bring significant advantages in terms of power efficiency, parallelism, and computational speed compared with conventional deep neural networks (DNNs), which have intrinsic limitations when implemented on digital platforms. However, inversely mapping algorithm-trained physical model parameters onto real-world optical devices with discrete values is a non-trivial task as existing optical devices have non-unified discrete levels and non-monotonic properties. This work proposes a novel device-to-system hardware-software codesign framework, which enables efficient physics-aware training of DONNs w.r.t arbitrary experimental measured optical devices across layers. Specifically, Gumbel-Softmax is employed to enable differentiable discrete mapping from real-world device parameters into the forward function of DONNs, where the physical parameters in DONNs can be trained by simply minimizing the loss function of the ML task. The results have demonstrated that our proposed framework offers significant advantages over conventional quantization-based methods, especially with low-precision optical devices. Finally, the proposed algorithm is fully verified with physical experimental optical systems in low-precision settings.
BibTeX
@inproceedings{li2022physics,
  author = {Yingjie Li and Ruiyang Chen and Weilu Gao and Cunxi Yu},
  title = {{Physics-Aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks}},
  booktitle = {ICCAD},
  year = {2022},
  url = {https://openreview.net/forum?id=rOOE2j19-udu},
}

Loading related content...

Figure