TL;DR
Optical Neural Networks (ONNs) are often hyped for their speed and energy efficiency, but their performance is frequently measured using physics-native metrics that don't translate to real-world machine learning. This paper proposes a "function-oriented" evaluation: define the optical module's input/output boundary and replace it with standard digital modules (like MLPs or CNNs) to see what it actually contributes. By identifying a "performance-matched" digital substitute, researchers can finally make fair, apples-to-apples comparisons between exotic optical hardware and standard digital silicon.
What Problem Does This Paper Address?
We are currently seeing a "billion-dollar paradox" in optical computing. Startups have raised over $4 billion to build light-based AI accelerators, yet the ML community often finds it impossible to interpret their results.
The core issue is a language barrier. Optics researchers describe their systems in terms of light propagation, diffractive masks, and interferometer meshes. They might call a physical component a "layer" or a "neuron," but in ML terms, a stack of ten optical "layers" might only implement a single linear transformation. Without a common frame of reference, it’s unclear if an ONN’s impressive task performance comes from the "magic" of the optics or simply from the digital pre- and post-processing surrounding it.
Methods
The authors propose a simple, three-step evaluation methodology summarized in an Expressivity Card (EC):
- Define the Boundary: Treat the optical module as a black box that maps real-valued input tensors to real-valued output tensors. Ignore the internal physics and focus on the digital interface.
- Run an Ablation Study: Replace the optical module with a "ladder" of standard digital substitutes (e.g., a Linear layer, a 2-layer MLP, a small CNN). Train these substitutes within the exact same pipeline and task.
- Identify the Match: Find which digital module matches the optical system's performance. This module becomes the "anchor" for all system-level claims.
Datasets
The paper demonstrates this methodology using case studies on the MNIST handwritten digit dataset. While the methodology is designed to be task-agnostic (applicable to ImageNet, CIFAR-10, etc.), the authors chose MNIST because it is a common benchmark in existing ONN literature, allowing them to retrospectively analyze published work.
Key Insights and Ideas
- Physics $\neq$ Function: Just because an optical process is physically complex (like simulating wave propagation) doesn't mean it's computationally expressive. The "functional" value of a module is what it contributes to the task, not how hard it is to simulate on a computer.
- The "Anchor" for Claims: If an optical module performs exactly like a shallow 2-layer MLP, then claims about its speed or energy efficiency should be compared against a digital 2-layer MLP—not against a massive GPU or a "physics-native" operation count that inflates the numbers.
- Diagnostic Value: This approach helps hardware designers. If a tiny linear layer outperforms your complex optical setup, you know you have either a hardware capacity limit or an optimization bottleneck in your training process.
Key Results
The authors applied their "Expressivity Card" to a state-of-the-art Optical Generative Model (OGM).
- The Match: They found that the optical module in the OGM performed similarly to a shallow 2-layer MLP (with a hidden dimension of 256).
- The "Crusher": A very small CNN (only 75K parameters) significantly outperformed the optical module, achieving a much better FID score (35.7 vs. 92.9).
- Efficiency Reality Check: While simulating the light propagation for this module would take ~58 million FLOPs, the performance-matched MLP only requires ~1 million FLOPs. This suggests that the "effective" computational throughput of the optics is much lower than physics-based counts might suggest.
Limitations and Future Work
The authors acknowledge several hurdles:
- Retrospective Difficulty: Many ONN papers do not release enough code or data to allow for independent ablation studies. The authors urge the community to adopt the Expressivity Card as a standard reporting norm for future papers.
- Task Dependence: A module's expressivity isn't a single number; it can change depending on the task (e.g., a module might be great for sensing but poor for generative modeling).
- Fairness to Optics: The authors note that their protocol might actually be too kind to optics, as the surrounding digital pipelines in these papers are often specifically co-designed to make the optical module look good.
Ultimately, the goal isn't to dismiss optics, but to provide a "minimal translation layer" that allows the ML and optics communities to finally speak the same language.
Summary updated