Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks

Yingjie Li, Ruiyang Chen, Weilu Gao, Cunxi Yu

Abstract

Diffractive optical neural networks (DONNs) have attracted significant attention due to their potential for extreme power efficiency, parallelism, and computational speed compared to conventional digital deep neural networks (DNNs). However, a major challenge remains: the "reality gap" involved in mapping algorithm-trained physical model parameters onto real-world optical devices. Existing optical hardware often features non-unified discrete levels and non-monotonic properties that are difficult to model using standard optimization techniques. This work proposes a novel device-to-system hardware-software codesign framework that enables efficient physics-aware training of DONNs using arbitrary experimentally measured optical device responses. By employing the Gumbel-Softmax trick, we enable differentiable discrete mapping from real-world device parameters into the forward function of the DONN. Our results demonstrate that this framework offers significant advantages over conventional quantization-based methods, particularly for low-precision optical devices, and is fully verified through physical experimental optical systems.

Introduction

Diffractive Optical Neural Networks (DONNs) represent a frontier in AI hardware, utilizing the speed of light and the physics of diffraction to perform computations. While theoretically superior in energy efficiency, transitioning these networks from high-level simulations to physical hardware—such as Spatial Light Modulators (SLMs)—is fraught with difficulty. Most training algorithms rely on continuous floating-point weights and smooth gradients, whereas physical devices operate on discrete voltage levels and exhibit complex, non-linear phase and amplitude responses.

This paper addresses the "reality gap" by introducing a differentiable codesign framework that incorporates physical hardware constraints directly into the training loop. By making the discrete selection of hardware states differentiable, we allow the network to learn the optimal configuration for a specific piece of hardware, rather than attempting to force a pre-trained model onto a non-ideal physical substrate.

The Challenges of Optical Hardware

Three primary obstacles hinder the deployment of DONNs on real-world hardware:

  1. Digital vs. Physical Discrepancy: AI models are trained in continuous spaces, but SLMs and other diffractive elements operate on discrete voltage levels (e.g., 8-level or 16-level precision).
  2. Non-Ideal Device Responses: Real-world optical devices rarely exhibit linear or monotonic responses. A specific voltage change might affect both phase and amplitude in a coupled, non-uniform manner that standard simulators fail to capture.
  3. Gradient Breakage: Standard quantization methods, such as rounding weights after training, break the "gradient chain" required for backpropagation. This often leads to accuracy drops of up to 30% when moving from a computer model to a physical system.

Methodology: Differentiable Discrete Mapping

The core of our proposed framework is the use of the Gumbel-Softmax (GS) trick to bridge the gap between discrete hardware states and differentiable optimization.

Gumbel-Softmax Integration

Instead of treating weights as continuous values to be quantized, our framework treats the selection of a discrete voltage level as a categorical probability distribution. During the forward pass, the system "samples" from these discrete levels. To maintain differentiability for the backward pass, the Gumbel-Softmax distribution provides a continuous, differentiable approximation of this discrete sampling.

Temperature Scheduling

A critical component of this approach is the "temperature" parameter ($\tau$).

  • Early Training: A high $ au$ creates a smooth distribution, allowing the model to explore various discrete levels across the hardware's range.
  • Annealing: As training progresses, $ au$ is lowered (annealed), forcing the model to converge and commit to specific hardware voltage levels.

Through empirical testing of various schedules (Cosine, Step, and Static), we determined that Linear temperature decay provides the most stable and efficient convergence for DONNs.

Physics-Aware Training

The framework incorporates actual measured phase and amplitude responses from the SLMs used in the lab. By integrating these physical profiles into the forward function, the "neurons" in the simulation behave identically to the pixels in the physical experimental setup.

Experimental Results

We evaluated the framework using the MNIST and FashionMNIST (FMNIST) datasets, with images interpolated to $200 \times 200$ pixels to match optical resolution requirements.

Performance Benchmarks

The GS-based framework achieved high accuracy even with low-precision (8-level) discrete devices:

  • MNIST: 98% accuracy.
  • FashionMNIST: 89% accuracy.

Comparison with Quantization

We compared our approach against Weight Sharing Quantization (WSQ), a common technique for discretizing neural networks. While WSQ suffered accuracy drops of over 27% when forced to adhere to physical hardware constraints (due to "illegal" values and non-monotonicity), our GS-based method maintained its performance because it was optimized within those constraints from the start.

Physical Validation

The framework was validated on a physical 3-layer visible-range DONN system. The experimental results closely matched our simulations, proving that the differentiable codesign effectively accounts for the complexities of real-world light diffraction and device behavior. Furthermore, we found that deeper networks (e.g., 5 layers) trained with this method exhibited higher confidence and greater robustness to environmental noise compared to single-layer systems.

Conclusion

This work demonstrates that the limitations of shallow or low-precision DONNs are often a result of suboptimal training rather than fundamental physical constraints. By using a physics-aware, differentiable discrete mapping approach, we can unlock the potential of high-speed optical computing on existing, non-ideal hardware. Future work will focus on scaling this approach to higher discrete levels and integrating non-linear optical activation functions to further enhance the computational power of DONNs.

Summary updated

Abstract

Diffractive optical neural networks (DONNs) have attracted lots of attention as they bring significant advantages in terms of power efficiency, parallelism, and computational speed compared with conventional deep neural networks (DNNs), which have intrinsic limitations when implemented on digital platforms. However, inversely mapping algorithm-trained physical model parameters onto real-world optical devices with discrete values is a non-trivial task as existing optical devices have non-unified discrete levels and non-monotonic properties. This work proposes a novel device-to-system hardware-software codesign framework, which enables efficient physics-aware training of DONNs w.r.t arbitrary experimental measured optical devices across layers. Specifically, Gumbel-Softmax is employed to enable differentiable discrete mapping from real-world device parameters into the forward function of DONNs, where the physical parameters in DONNs can be trained by simply minimizing the loss function of the ML task. The results have demonstrated that our proposed framework offers significant advantages over conventional quantization-based methods, especially with low-precision optical devices. Finally, the proposed algorithm is fully verified with physical experimental optical systems in low-precision settings.
BibTeX
@inproceedings{li2022physics,
  author = {Yingjie Li and Ruiyang Chen and Weilu Gao and Cunxi Yu},
  title = {{Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks}},
  booktitle = {ICCAD},
  year = {2022},
  url = {https://openreview.net/forum?id=rOOE2j19-udu},
}
Figure