Optical neural networks promise very low energy per inference, but they need a nonlinearity. In ordinary materials that nonlinearity only appears under intense light. One route around this synthesizes the nonlinearity from purely linear optics: the input is written onto a spatial light modulator (a programmable pixelated phase mask), light is bounced back and forth through it several times, and each pass multiplies in another copy of the data, so the output becomes a polynomial of the input whose degree equals the number of passes (Nonlinear processing with linear optics). This works with weak continuous-wave light. It leaves open how weak the light can get. Every extra bounce adds expressive power, but it also loses light. When only tens to thousands of photons reach the detectors per inference, the random arrival of photons, known as Poisson shot noise, can erase the polynomial terms that the extra depth was meant to provide.
This note proposes to measure and predict that trade-off directly. The claim is that, under a fixed photon budget per inference, classification accuracy of a linear-optics nonlinearity synthesizer is non-monotone in depth. The best depth grows roughly with the logarithm of the photon budget, divided by the per-bounce loss. Training through a faithful Poisson detection model moves this optimum deeper than training through noise-free or Gaussian-noise models. If the claim holds, "how many passes?" gets an answer an engineer can compute from two device numbers. It would also tell us whether low-power synthetic nonlinearity actually buys accuracy per photon compared with simply detecting the input and classifying it digitally.
Core Claim
In the multi-bounce design of Yildirim et al., reported accuracy rises as a power law in the number of passes $N$. Each pass through the modulator and mirror transmits only a fraction $\eta<1$ of the light. If $\alpha_0$ photons enter per inference, roughly $\alpha_0\eta^N$ are detected. We model the test error as
$$E(N) \approx A\,N^{-\gamma} + \frac{B}{\alpha_0\,\eta^{N}},$$
where $A$ and $\gamma$ describe the noise-free power-law improvement with depth that Yildirim et al. observed. The constant $B$ measures how sensitive the trained readout is to count variance; the second term follows from Poisson variance scaling as the inverse of the detected count. Setting the derivative to zero gives $N^*$ growing approximately as $\ln(\alpha_0)/\ln(1/\eta)$, up to slowly varying corrections.
This form borrows directly from the U-shaped error analysis in Resolution Limit of Single-Photon LiDAR. That paper shows that splitting a fixed photon flux over more pixels trades discretization bias falling as $1/N^2$ against Poisson variance growing as $N/\alpha_0$. Our setting differs in two ways. The bias term is model capacity rather than spatial discretization. The variance penalty grows exponentially in depth rather than linearly, because light is lost multiplicatively per bounce rather than divided among pixels. The model is a hypothesis to test, not a result. Its sharpest prediction is quantitative: halving the per-bounce transmission in log terms should roughly halve the optimal depth at fixed budget.
The second part of the claim concerns training. Below roughly ten to twenty counts per detector, Poisson noise is skewed and integer-valued, and Gaussian noise models misdescribe it. We expect networks trained with sampled Poisson counts in the loop to tolerate larger $N$ than networks trained with variance-matched Gaussian noise. That would push $N^*$ outward.
Relation to Prior Work
The proposal takes its device from the linear-synthesis line of work. Yildirim et al. demonstrate polynomial nonlinearity from repeated encoding on MNIST, Fashion-MNIST and Imagenette, always at illumination bright enough to treat the field as deterministic. Nonlinear Computation with Linear Optics via Source-Position Encoding reaches nonlinearity instead by encoding data as emitter position in a topology-optimized passive medium, again with a noise-free readout. Neither paper asks what happens when photons are scarce, and that is the entire question here.
Resonant material routes motivate the problem rather than supply the method. Free space few-photon nonlinearity in critically coupled polaritonic metasurfaces reaches nonlinear absorption at about 6.12 photons per cavity lifetime, but only at 6 K. At that count Poisson fluctuations are around 40% of the mean, a point the device paper does not analyze. Coherent Ising machines—optical neural networks operating at the quantum limit runs at the quantum limit, but uses a strong χ(2) parametric process and electronic feedback, and targets binary optimization rather than feedforward inference.
We choose the linear synthesizer because it runs at room temperature with off-the-shelf parts. That makes the photon-budget question experimentally clean.
Two papers constrain the design:
- Robust Generalization Against Photon-Limited Corruptions via Worst-Case Sharpness Minimization shows that robust training on scarce Poisson-corrupted data falls into sharp minima. Its SharpDRO method (flattening the loss around the worst-case noise level) is the strongest training-side competitor we know of. We reuse it as a baseline rather than assume noise injection is enough.
- Fully forward mode training for optical neural networks shows that in-silico models transfer poorly to hardware: about 58.8% versus 92.5% for on-hardware training on Fashion-MNIST. We therefore treat simulation-to-hardware mismatch as a first-class risk and fit the final layer on hardware.
Research Program
Simulator. We first build a simulator of the multi-bounce synthesizer. It includes:
- per-bounce transmission $\eta$;
- modulator phase quantization and flicker;
- detector quantum efficiency;
- exact Poisson sampling of detector counts, with the total expected entering photons per inference $\alpha_0$ swept from about $10^1$ to $10^6$.
Training is differentiable through the Poisson layer using a straight-through or score-function estimator. We use it to fit $A$, $\gamma$ and $B$, and to predict $N^*(\alpha_0,\eta)$ before any hardware run.
Hardware. We then repeat the sweep on a tabletop synthesizer following the Yildirim et al. geometry. The laser is attenuated with calibrated neutral-density filters, and outputs are read on a photon-counting camera, so each inference has a known, measured photon count. The per-bounce transmission is changed deliberately, by inserting a calibrated partial attenuator in the cavity arm. This lets us test the depth-scaling prediction at two or three values of $\eta$, rather than only fitting it at one.
Training conditions. At each $(\alpha_0,\eta,N)$ we train under four conditions: noise-free, variance-matched Gaussian noise, sampled Poisson noise, and Poisson noise with SharpDRO. We report accuracy as a function of detected photons per inference. The final linear readout is re-fit on hardware counts to absorb calibration drift.
Baselines
Photon-matched direct detection. This is the naive reference and the one that matters most. The input image itself is detected with the same total photon count and classified by an electronic model that sees only those Poisson counts. We run both a linear classifier and a small convolutional network, trained with Poisson augmentation and given the same hyperparameter budget as the optical system.
This is the strongest competitor because optical preprocessing cannot create information. It can only choose a different measurement, one that concentrates photons onto a few output detectors instead of spreading them over 784 pixels. Beating it at equal detected photons, even with a far weaker downstream model, is the evidence that low-power synthetic nonlinearity earns its place.
Single-pass linear system. The second baseline is the same optics at $N=1$, which removes all synthesized nonlinearity at the lowest optical loss. It is the right comparator because at small budgets it may simply win. In that case the answer to "how deep?" is "not at all," and we would report it.
Strongest existing optical results. These come from deterministic-light settings, since no reported result found on Lacuna gives accuracy for any optical network under a controlled photon-per-inference budget. For the synthesizer itself, Yildirim et al. report Imagenette test accuracy of 35% experimentally with 4 layers, and about 46% in simulation. We use their 4-layer configuration, run at high light level on our hardware, as the reference ceiling. Reproducing roughly 35% there validates the setup.
For wider context, Richardson et al. report 83.1% on MNIST and 53.1% on Fashion-MNIST, against linear baselines of 75.8% and 47.1%. Fully forward mode training reports 92.5% on Fashion-MNIST. These are not photon-matched and serve only as upper anchors.
Training-side baselines. SharpDRO, from the released zhuohuangai/SharpDRO code, is the strongest robust-training competitor; it reports 84.7% versus 80.5% for Risk Extrapolation on CIFAR-10 under severe Gaussian noise. It is adapted to the optical parameters with an identical tuning budget.
Ablation. The key ablation isolates the proposal's training ingredient. Poisson-in-the-loop training is compared against Gaussian noise with exactly matched per-detector variance. Any advantage then comes from the discrete, skewed shape of the noise, not its size.
What counts as success. The proposal wins if two things hold:
- at some budget below about $10^4$ detected photons per inference, the synthesizer at its predicted $N^*$ beats photon-matched direct detection with the linear digital classifier, and beats both its own $N=1$ and deepest configurations, by more than the seed-to-seed spread;
- the measured $N^*$ shifts with $\eta$ in the predicted direction and roughly the predicted amount.
Why not just…?
Why not just raise the laser power until shot noise is negligible? The motivation for linear-synthesized nonlinearity is energy per inference. The Direction 1 argument is that once light power dominates, the optical system loses to electronics, and remote or eye-safe sensing is photon-limited regardless, as in the LiDAR analysis of Chan et al. Our results are reported as accuracy versus photons, so a reader can see directly how many photons buy each point of accuracy. If the curves show the synthesizer only pays off above budgets where electronics would be cheaper, that is a negative result we would state plainly.
Why not just inject Poisson noise during training and pick depth by grid search? This is a real threat to the training claim. It is also exactly what our Poisson-trained condition does. The contribution that survives, even if noise-aware training is all one needs, is the predictive law for $N^*$ in terms of $\alpha_0$ and $\eta$. That law replaces a hardware grid search with two calibration measurements.
The Gaussian-versus-Poisson ablation settles whether noise shape matters beyond noise size. If it does not, we drop the second half of the claim. SharpDRO tests whether sharpness control adds anything beyond plain injection. Huang et al. suggest it should under scarce corruptions, but their evidence is from digital CNNs, not optical parameters.
Why not just detect the dim image and denoise or classify it digitally? Denoising cannot recover information lost at detection, and Chan et al. show that post-hoc smoothing trades variance for bias. However, a Poisson-trained digital classifier on raw counts has no such handicap. This is why photon-matched direct detection is our primary baseline, not a strawman. The question is empirical: does routing light through several optical passes before detection yield a more informative measurement than detecting pixels directly? It might not at the smallest budgets, and the experiment is designed to show where the crossover lies.
Why not just use a genuine few-photon material nonlinearity instead? The polaritonic metasurface of Fang et al. needs 6 K operation and narrow resonances. Its ~40% per-lifetime fluctuation would itself need exactly the photon-budget analysis proposed here. We treat it as a later extension: modeling the metasurface as a noisy saturable activation in the same simulator, so the two routes to low-power nonlinearity can be compared at equal photons.
Decisive Checks
The first check is whether simulated accuracy-versus-depth curves are U-shaped at low budgets and monotone at high budgets on MNIST, Fashion-MNIST and Imagenette. If depth never hurts down to tens of photons, the core premise is wrong.
The second check is whether the fitted law, calibrated at one $\eta$, predicts the hardware $N^*$ at a second $\eta$ to within one bounce.
The third check is whether any depth of the synthesizer beats photon-matched direct detection at equal detected photons. A consistent loss would mean that, under photon scarcity, linear-optics nonlinearity is a way to spend photons rather than save them.
Any of these outcomes is informative. Together they fix a regime where "low-power optical nonlinearity" is currently asserted but not measured.
References
- Nonlinear processing with linear optics
- Resolution Limit of Single-Photon LiDAR
- Nonlinear Computation with Linear Optics via Source-Position Encoding
- Free space few-photon nonlinearity in critically coupled polaritonic metasurfaces
- Coherent Ising machines—optical neural networks operating at the quantum limit
- Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization
- Fully forward mode training for optical neural networks