Fully forward mode training for optical neural networks

Zhiwei Xue, Tiankuang Zhou, Zhihao Xu, Shaoliang Yu, Qionghai Dai, Lu Fang

TL;DR

Researchers have developed a novel "fully forward mode" (FFM) learning approach that enables optical neural networks (ONNs) to train themselves directly on physical hardware, bypassing the need for computationally expensive digital simulations. This breakthrough allows for significantly faster and more efficient training of ONNs, leading to state-of-the-art performance in tasks like focusing through scattering media and non-line-of-sight imaging.

What Problem Does This Paper Address?

Training machine learning models, especially deep neural networks, typically relies on extensive simulations performed on digital computers. This "in silico" approach is computationally intensive and requires large datasets. When applied to optical systems, this problem is exacerbated by the complexity of modeling light propagation and system imperfections, leading to a mismatch between simulated and real-world performance. Existing methods for training optical systems onsite are often limited by the need for backward propagation of light fields, which requires stringent calibration and can be difficult to implement physically. This paper addresses the challenge of efficiently training optical neural networks directly on their physical hardware, overcoming the limitations of offline modeling and complex calibration.

Methods

The core innovation is the Fully Forward Mode (FFM) learning approach. Instead of relying on traditional backpropagation, FFM learning leverages spatial symmetry and Lorentz reciprocity to eliminate the need for backward light propagation. The training process is entirely "onsite," meaning it happens directly on the physical optical system.

Here's how it works:

  • Mapping Optical Systems to Neural Networks: Optical systems are re-parameterized as differentiable, embedded photonic neural networks. The tunable components (like refractive index) in the optical system correspond to the "weights" of the neural network.
  • Data and Error Propagation: Input data is propagated through the physical optical system (data propagation). An "error field" is generated based on the difference between the system's output and the desired target. This error field is then also propagated forward through the system.
  • Gradient Calculation: By cleverly using the symmetry of the system, the gradients required for training are calculated from the measured outputs of both the data and error propagations.
  • Onsite Gradient Descent: The calculated gradients are used to update the tunable optical parameters (e.g., refractive index) directly on the physical system, iteratively improving its performance.

This method allows for the entire training process, including gradient calculation, to be performed optically or optoelectronically, significantly speeding up learning and reducing reliance on digital simulations. The paper also introduces nonlinear FFM learning, where nonlinear activations are incorporated into the training process, further enhancing the capabilities of the ONNs.

Datasets

The paper utilizes several standard benchmark datasets for evaluating the performance of their FFM learning approach:

  • MNIST: A dataset of handwritten digits, used for classification tasks.
  • Fashion-MNIST: A dataset of clothing items, also used for classification.
  • CIFAR-10: A dataset of small images of objects, used for classification.
  • ImageNet: A large-scale image dataset, used for classification.
  • Iris dataset: A dataset describing Iris flower species, used for classification in the integrated photonic circuit experiments.
  • MWD dataset: A dataset of weather condition images, used for classification.
  • Chromium target dataset: A custom dataset for non-line-of-sight (NLOS) imaging, featuring letter-shaped targets.

Key Insights and Ideas

  • Onsite Training is Key: The most significant insight is that complex optical systems can be trained directly on their physical hardware, bypassing the limitations of offline simulations. This is a paradigm shift for optical AI.
  • FFM Learning Eliminates Backward Propagation: By exploiting symmetry and reciprocity, the authors have found a way to train optical systems without the need for backward light propagation, a major experimental hurdle.
  • Optical Systems are Inherently Differentiable: The work demonstrates that optical systems can be viewed as differentiable neural networks, opening up new avenues for designing and controlling them.
  • Achieving Diffraction-Limited Imaging: FFM learning can optimize optical systems to achieve imaging resolutions very close to the theoretical diffraction limit, even through scattering media.
  • Photon-Efficient NLOS Imaging and Processing: The method enables efficient imaging and all-optical classification of objects hidden from direct view, requiring remarkably few photons.
  • Model-Free Design: FFM learning allows for the design of optical systems without requiring an explicit analytical model of the physics involved, making it applicable to complex and unconventional scenarios.

Key Results

  • Deep ONNs: FFM learning enabled the training of free-space ONNs with up to eight layers and millions of parameters, achieving accuracies comparable to ideal in-silico models. For example, on the Fashion-MNIST dataset, FFM achieved accuracies of 92.5% for an 8-layer network, significantly outperforming in-silico trained networks (around 58.8%).
  • Scattering Imaging: The FFM approach achieved focusing through scattering media with a resolution approaching the diffraction limit (65.6 µm FWHM achieved vs. 64.5 µm theoretical limit). It also significantly outperformed Particle Swarm Optimization (PSO) in terms of convergence speed and final loss.
  • Non-Line-of-Sight (NLOS) Imaging: FFM enabled parallel imaging of dynamic NLOS scenes with millisecond exposure times and achieved high classification accuracies even at very low photon counts (<1 photon per pixel), demonstrating an energy efficiency of 5.40 × 10¹⁸ OPS/W.
  • Integrated Photonic Circuits: FFM learning was successfully applied to an integrated photonic neural network, achieving classification accuracies of up to 94.2% on the Iris dataset, comparable to ideal simulations and significantly better than in-silico trained networks (around 71.7%).
  • Exceptional Point Search: The method was numerically demonstrated to self-design non-Hermitian systems to traverse exceptional points without prior physical models.

Limitations and Future Work

  • Computational Resources: While FFM learning is faster than in-silico training, the physical propagation and measurement steps still require time and resources. The paper mentions that compute-intensive propagations are executed on site, implying a need for efficient optical components.
  • Scalability of Physical Systems: The paper discusses scaling to deeper networks and larger parameter counts, but the physical implementation of such large-scale optical systems remains a challenge.
  • Generalization to Complex Non-Hermitian Systems: While the paper demonstrates FFM learning for exceptional point search, extending this to more complex non-Hermitian topological systems might require further investigation.
  • Future Work: The authors suggest that FFM learning has the potential to surpass GPU-accelerated learning implementations with advancements in optoelectronic devices. They anticipate significant improvements in high-speed FFM learning, potentially reaching efficiencies orders of magnitude higher. The work also opens up possibilities for large-scale, high-efficiency physical AI in the post-Moore era.

Summary updated

Abstract

Optical computing promises to improve the speed and energy efficiency of machine learning applications1–6. However, current approaches to efficiently train these models are limited by in silico emulation on digital computers. Here we develop a method called fully forward mode (FFM) learning, which implements the compute-intensive training process on the physical system. The majority of the machine learning operations are thus efficiently conducted in parallel on site, alleviating numerical modelling constraints. In free-space and integrated photonics, we experimentally demonstrate optical systems with state-of-the-art performances for a given network size. FFM learning shows training the deepest optical neural networks with millions of parameters achieves accuracy equivalent to the ideal model. It supports all-optical focusing through scattering media with a resolution of the diffraction limit; it can also image in parallel the objects hidden outside the direct line of sight at over a kilohertz frame rate and can conduct all-optical processing with light intensity as weak as subphoton per pixel (5.40 × 1018-&nbsp;operations-per-second-per-watt energy efficiency) at room temperature. Furthermore, we prove that FFM learning can automatically search non-Hermitian exceptional points without an analytical model. FFM learning not only facilitates orders-of-magnitude-faster learning processes, but can also advance applied and theoretical fields such as deep neural networks, ultrasensitive perception and topological photonics. We present fully forward mode learning, which conducts machine learning operations on site, leading to faster learning and promoting advancement in numerous fields.
BibTeX
@article{xue2024fully,
  author = {Zhiwei Xue and Tiankuang Zhou and Zhihao Xu and Shaoliang Yu and Qionghai Dai and Lu Fang},
  title = {{Fully forward mode training for optical neural networks}},
  journal = {Nat.},
  year = {2024},
  doi = {10.1038/s41586-024-07687-4},
}
Figure