Optical Neural Networks Should Be Evaluated by Function, Not Only Physics

Shi-Yuan Ma, Xingjian Bai, Ryan Hamerly, Mahmoud Jalali MehrabadarXiv 2026 · arXiv

TL;DR

Optical Neural Networks (ONNs) are often hyped for their speed and energy efficiency, but their performance is frequently measured using physics-native metrics that don't translate to real-world machine learning. This paper proposes a "function-oriented" evaluation: define the optical module's input/output boundary and replace it with standard digital modules (like MLPs or CNNs) to see what it actually contributes. By identifying a "performance-matched" digital substitute, researchers can finally make fair, apples-to-apples comparisons between exotic optical hardware and standard digital silicon.

What Problem Does This Paper Address?

We are currently seeing a "billion-dollar paradox" in optical computing. Startups have raised over $4 billion to build light-based AI accelerators, yet the ML community often finds it impossible to interpret their results.

The core issue is a language barrier. Optics researchers describe their systems in terms of light propagation, diffractive masks, and interferometer meshes. They might call a physical component a "layer" or a "neuron," but in ML terms, a stack of ten optical "layers" might only implement a single linear transformation. Without a common frame of reference, it’s unclear if an ONN’s impressive task performance comes from the "magic" of the optics or simply from the digital pre- and post-processing surrounding it.

Methods

The authors propose a simple, three-step evaluation methodology summarized in an Expressivity Card (EC):

  1. Define the Boundary: Treat the optical module as a black box that maps real-valued input tensors to real-valued output tensors. Ignore the internal physics and focus on the digital interface.
  2. Run an Ablation Study: Replace the optical module with a "ladder" of standard digital substitutes (e.g., a Linear layer, a 2-layer MLP, a small CNN). Train these substitutes within the exact same pipeline and task.
  3. Identify the Match: Find which digital module matches the optical system's performance. This module becomes the "anchor" for all system-level claims.
Figure 1. Evaluating optical modules by function, not only physics. An optical module (yellow) sits within a hybrid digital– optical pipeline. Internally, the module involves complex light propagation through various physical substrates (interferometer meshes, wavelength-division multiplexing, diffractive elements, programmable waveguides, etc.). However, when used as a com- ponent in a classical ML pipeline, the surrounding digital system interacts with it only through real-valued tensors: input x ∈X and output y ∈Y. The mapping fopt is therefore the relevant object for ML comparison. Its empirical expressivity is measured by an ablation study: the optical module is replaced by standard digital modules with the same inputs and outputs, and the one that matches its task performance is identified.

Datasets

The paper demonstrates this methodology using case studies on the MNIST handwritten digit dataset. While the methodology is designed to be task-agnostic (applicable to ImageNet, CIFAR-10, etc.), the authors chose MNIST because it is a common benchmark in existing ONN literature, allowing them to retrospectively analyze published work.

Key Insights and Ideas

  • Physics $\neq$ Function: Just because an optical process is physically complex (like simulating wave propagation) doesn't mean it's computationally expressive. The "functional" value of a module is what it contributes to the task, not how hard it is to simulate on a computer.
  • The "Anchor" for Claims: If an optical module performs exactly like a shallow 2-layer MLP, then claims about its speed or energy efficiency should be compared against a digital 2-layer MLP—not against a massive GPU or a "physics-native" operation count that inflates the numbers.
  • Diagnostic Value: This approach helps hardware designers. If a tiny linear layer outperforms your complex optical setup, you know you have either a hardware capacity limit or an optimization bottleneck in your training process.

Key Results

The authors applied their "Expressivity Card" to a state-of-the-art Optical Generative Model (OGM).

  • The Match: They found that the optical module in the OGM performed similarly to a shallow 2-layer MLP (with a hidden dimension of 256).
  • The "Crusher": A very small CNN (only 75K parameters) significantly outperformed the optical module, achieving a much better FID score (35.7 vs. 92.9).
  • Efficiency Reality Check: While simulating the light propagation for this module would take ~58 million FLOPs, the performance-matched MLP only requires ~1 million FLOPs. This suggests that the "effective" computational throughput of the optics is much lower than physics-based counts might suggest.
Figure 2. Ablation study for OGM, MNIST generation. FID (lower is better) against the number of trainable parameters of the module in the optical module’s place. Markers: digital substitutes, all trained with the same encoder architecture, data, loss, and training setup; mean ± std over 4 seeds, FID on 5,000 samples at epoch 150. Dashed line and yellow band: the optical module simulated with the authors’ released code under the same setup (FID 92.9 ± 2.9). Solid line: the experimental FID reported by the authors (131.08), computed on 1,000 generated images and therefore not strictly comparable with the other values. The optical module’s 160K phase-mask pixels are not parameters in the ML sense and are not placed on the horizontal axis.
Table 1. Empirical expressivity on MNIST.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Retrospective Difficulty: Many ONN papers do not release enough code or data to allow for independent ablation studies. The authors urge the community to adopt the Expressivity Card as a standard reporting norm for future papers.
  • Task Dependence: A module's expressivity isn't a single number; it can change depending on the task (e.g., a module might be great for sensing but poor for generative modeling).
  • Fairness to Optics: The authors note that their protocol might actually be too kind to optics, as the surrounding digital pipelines in these papers are often specifically co-designed to make the optical module look good.

Ultimately, the goal isn't to dismiss optics, but to provide a "minimal translation layer" that allows the ML and optics communities to finally speak the same language.

Summary updated

Abstract

Optical neural networks (ONNs) are attracting significant attention as alternative platforms for machine learning (ML). In an ONN, an optical module often takes over part of the computation inside a digital ML pipeline. How much of the task does the optical module itself perform, and how much does it contribute to any claimed system-level advantages, such as throughput or energy? This paper offers a simple, function-oriented evaluation methodology. Whatever its internal physics, the optical module appears to the surrounding digital pipeline as a mapping between real-valued tensors. We therefore evaluate it in three steps: (i) define its boundary in the pipeline, as real-valued tensors in and out; (ii) replace it with standard digital modules to identify those that match the optical system's performance on the same task and metric; and (iii) state system-level claims, such as speed or energy, per inference against the performance-matched digital module. We summarize steps (i) and (ii) in an Expressivity Card and demonstrate its use on published ONN systems. This helps isolate the optical module's contribution and provides a concrete digital reference for assessing its claimed system-level advantages, without the need to understand the physics. While we focus on optical systems as the most prominent example, the same approach applies to any physical analog computing platform used in ML, wherever a physical module takes over part of a digital pipeline.
BibTeX
@misc{ma2026optical,
  author = {Shi-Yuan Ma and Xingjian Bai and Ryan Hamerly and Mahmoud Jalali Mehrabad},
  title = {{Optical Neural Networks Should Be Evaluated by Function, Not Only Physics}},
  year = {2026},
  eprint = {2611.23610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23610},
}
Figure