Switching Experts with Light: Sparse Mixtures in Static Diffractive Optics

A mixture-of-experts model is a network built from many sub-networks, called experts. A small gating model picks which few experts handle each input. On GPUs the main deployment cost of this design is memory. Every expert must sit in fast memory, or the system pays to fetch it when it is chosen. A diffractive optical network computes by passing light through fabricated phase plates. It has the opposite property: every "weight" is permanently present in glass and costs nothing to keep. The catch is that it can never change after fabrication. We propose to put these two facts together. All experts of a mixture would be fabricated into one shared, static diffractive stack. Each expert would be "addressed" by the angle of the light that illuminates it. That angle is selected by switching on one emitter of a laser array placed in the Fourier plane of a lens, which is the plane where position maps to propagation angle. Selecting an expert then costs one laser switching event, not a memory transfer.

This trades the sparse-model memory problem for a modulation-bandwidth problem. That is a better problem to have, because directly modulated lasers switch at tens of gigahertz, while the spatial light modulators currently used to reconfigure diffractive networks switch at kilohertz to megahertz. The trade also creates two new difficulties that the proposal treats as central. First, the gating decision has to be made before the light is switched. Second, training one shared stack to behave as several distinct experts invites optimization collapse. Either all illumination angles converge to nearly the same function, or one angle wins and the rest go unused.

Core Claim

A single passive diffractive stack, illuminated by one of $K$ discrete angles chosen per input by an electronic gate, can act as a true input-dependent sparse mixture of experts. We claim this can beat two existing designs on the same task mixture. It should beat static multi-task diffractive networks controlled by amplitude masks on accuracy, and on light throughput by roughly an order of magnitude. It should also beat always-on parallel optical experts at equal optical power and aperture. Two further claims concern training. The local-minimum problem of jointly training experts in shared optics can be split into two parts. One is a low-dimensional outer problem: where the emitters sit in the Fourier plane and how bright they are, a few dozen parameters. The other is a high-dimensional inner problem: the phase pixels. Gradient-informed Bayesian optimization handles the outer problem well, and ordinary gradient descent with continuation is adequate for the inner one. Finally, the laser's switching rate becomes a budget the gate can be trained to respect, analogous to how digital mixtures are trained to respect memory residency.

What Exists and Where This Departs

The direct precursor is Illumination Angular Spectrum Encoding for Controlling the Functionality of Diffractive Networks. It places a programmable amplitude mask in a 2f Fourier system so that one static diffractive stack performs several tasks. On a four-dataset suite (MNIST, FashionMNIST, KMNIST, EMNIST) it reports 82.35% average accuracy, against 79.95% for wavelength encoding and 73.5% with no encoding. That paper supplies our physical mechanism and our main baseline. It differs from this proposal in three ways. The task is chosen from outside the system rather than inferred per input. The mask blocks most of the light, so throughput is only 1% to 8%. And reconfiguration runs at the speed of the mask. Replacing the blocking mask with an addressable emitter array tackles the throughput loss directly, because a lit emitter is the angle rather than a filtered piece of a broad beam. It also moves switching onto laser time scales.

The second relevant system is the optically parallel mixture of experts in Optical Spiking Neurons Enable High-Speed and Energy-Efficient Optical Neural Networks. It runs all experts at once along a shared optical path and combines their outputs with error-correcting codes. It reaches 96.27% on KTH action recognition. Its adaptive training raised physical MNIST accuracy from 35.7% to 90.2%, which shows that fabrication noise must be handled in training. That work motivates our claim that optics can host experts cheaply. It also constrains it, since it is the strongest argument that sparsity is unnecessary in optics. We address this head on in the baselines below.

On the digital side, MoE-ERAS: Expert Residency Aware Selection biases gating toward experts already in fast memory. BlockFFN adds losses that make consecutive tokens share experts. Both supply the idea that the gate should be trained against a hardware cost. We reuse that idea, but the cost here is the number of emitter switches per second rather than the number of bytes moved. ERAS also warns that biasing a gate away from its preferred expert degrades quality: C4 perplexity rose from 8.044 to 8.221. We should expect a similar accuracy-versus-switch-rate curve and measure it rather than assume it away.

For optimization, BONNI couples a neural-ensemble surrogate trained on adjoint gradients with an interior-point optimizer. On a 62-parameter grating coupler it reaches −2.2 dB against −2.9 dB for interior-point optimization alone. The authors say it stops paying off beyond roughly 75 to 100 parameters. Gaussian continuation (Mobahi et al.) smooths a rugged objective and tracks the solution as the smoothing is removed. Neither has been applied to multi-expert diffractive training. Our contribution there is the decomposition: BONNI is used only where its dimensional limit is respected.

Research Program

The system has three parts. The optical part is a stack of two to five phase plates illuminated through a 2f lens. A $K$-element emitter array sits in the lens's front focal plane. $K$ is between 4 and 16, limited by the sub-0.5° paraxial cone in which the angular-encoding paper found angle control to work. The electronic part is a small gate. It reads a "probe" pass, in which a fixed reference emitter is lit and a coarse detector readout is taken, and it outputs which emitter to light for the main pass. For video, the gate may instead reuse the previous frame's features, which avoids the probe pass. Two-pass latency is a cost we report rather than hide. The third part is the training procedure.

Training has an inner and an outer loop. The inner loop optimizes the phase pixels $\phi$ for fixed emitter geometry $g$, by gradient descent through a differentiable angular-spectrum propagation model. Gaussian continuation is applied to $\phi$: phase noise of decreasing width $\sigma_t$ is injected, and the same noise doubles as fabrication-robustness training in the spirit of the spiking-neuron paper. To counter expert collapse, the inner objective includes a load-balancing term and a switching term:

$$ \mathcal{L}(\phi, \theta; g) = \mathbb{E}_x\big[\ell(f_{\phi,g}(x, e_\theta(x)), y)\big] + \alpha\, \mathrm{KL}(\bar p_\theta \,\|\, u) + \beta\, \mathbb{E}_t\big[\mathbf{1}[e_\theta(x_t) \neq e_\theta(x_{t-1})]\big]. $$

The symbols are defined as follows. $\theta$ are the gate weights. $e_\theta(x)$ is the emitter chosen for input $x$, relaxed with a Gumbel-softmax (a differentiable stand-in for a hard choice) during training. $\ell$ is the classification loss on label $y$. $\bar p_\theta$ is the average gate distribution and $u$ is the uniform distribution over the $K$ emitters. The last term counts emitter switches between consecutive inputs in a stream. Sweeping $\beta$ traces accuracy as a function of switch rate.

The outer loop runs BONNI over $g$, which holds emitter positions and relative powers: at most 48 parameters for $K=16$. The objective is validation loss after a short inner run, and the gradient with respect to $g$ comes from differentiating through the last inner steps. This deliberately keeps BONNI inside the regime where its authors found it effective.

The physical model includes three non-idealities. Emitter wavelength drift is modelled as a small random angle and phase perturbation. Finite extinction is modelled as leakage from emitters that are switched off. Detector-side capacitance sets a ceiling on how fast the readout feeding the gate can run, following the receiver-parasitics discussion in Seung-Hoon Kim et al.. We expect this electronic step, not the laser, to set the real routing rate. Measuring where that crossover happens is one of the program's outputs.

Baselines

The naive reference point is a single diffractive stack of the same size with one fixed illumination angle, trained on the pooled four-dataset mixture. The angular-encoding paper's "non-encoded" number, 73.5% average accuracy, approximates this. We re-run it in our simulator so that the comparison uses identical layers, pixel pitch, detector layout and noise model.

The strongest existing method on this question is the amplitude-mask angular-encoding network of Illumination Angular Spectrum Encoding, which reports 82.35% average accuracy on the four-dataset suite. We reimplement it with the same layer stack and the same number of control patterns $K$. We run it in two conditions. In the first, the task identity is supplied, as in that paper's setting. In the second, our gate must infer it from the probe pass. Beating this baseline means exceeding 82.35% in the matched simulation in at least one of two ways: in the supplied-identity condition at equal $K$, or in the inferred-identity condition. In addition, throughput at the detector must be at least five times higher at equal source power. A throughput gain alone with lower accuracy counts as a partial result, not a win.

The second strong competitor is the always-on parallel mixture of Optical Spiking Neurons. We reproduce it as $K$ spatially multiplexed sub-apertures, all illuminated, with outputs combined by error-correcting codes. Total aperture, total source power and pixel count are held equal to our system. Beating it means higher accuracy at equal power and aperture under the same detector noise. On the four-dataset suite, its cited number is not comparable: the file reports 96.27% only on KTH. We therefore compare on KTH as well, against that figure, using the gate's previous-frame variant.

Two ablations isolate the key ingredients. In the first, the gate is replaced by random emitter assignment at the same switch rate, which tests whether input-dependent selection matters. In the second, the outer BONNI loop is replaced by multi-start Adam over the emitter geometry and phases jointly, at equal simulator-call budget. All methods get the same hyperparameter search budget, counted in simulator forward-and-adjoint evaluations, with five seeds each.

Why not just…?

Why not just light all experts at once, since optics is parallel anyway? This is the most serious threat. The spiking-neuron system shows parallel optical experts working well. Our argument is a resource one. Spatially multiplexing $K$ experts divides either the aperture or the power among them, so each expert sees $1/K$ of the light or of the pixels. Sparse illumination gives the single chosen expert everything. Whether this matters depends on whether accuracy is limited by detector noise or by expressivity. The iso-power, iso-aperture comparison settles it. If the parallel mixture matches us at moderate $K$, we will report that sparsity buys nothing in optics at that scale and narrow the claim to the low-light regime.

Why not just keep the amplitude mask and make its choice input-dependent? That is exactly the inferred-identity condition of the main baseline, so we test it directly. Two things argue against it. The reported 1% to 8% throughput comes from blocking light, which an emitter array avoids by construction. And spatial light modulators reconfigure at kilohertz to megahertz rates, which caps per-input routing for video or streaming inputs. If an idealized fast mask matched our accuracy, the remaining case for our design would rest on throughput alone, and we would say so.

Why not just use a static optical encoder with a digital backend that does the routing? Transferable polychromatic optical encoder shows that a fixed metasurface plus a digital calibration layer transfers across datasets. It reaches about 73.2% on CIFAR-10 and about 60% on the ImageNet High-10 subset, roughly 5–10% below unconstrained digital networks. A digital mixture after a shared optical front end would recover flexibility, but it puts the experts back into electronic memory and compute, which is the cost we are trying to remove. We include this hybrid as an extra comparison and report both accuracy and backend multiply-accumulate count.

Why not just use multi-start gradient descent instead of the BONNI outer loop? On small problems multi-start may be enough. BONNI's own paper notes that its 5-second surrogate overhead per iteration loses to plain gradients when evaluations are cheap. The second ablation answers this at equal simulator budget. If multi-start matches, we drop BONNI and keep continuation plus load balancing. The scientific claim about the decomposition would then fail, but the optical system would still stand.

Decisive Experiments

All primary experiments run in a calibrated angular-spectrum simulator, with fabrication phase noise and emitter drift drawn from ranges consistent with the cited hardware papers. A small tabletop check would use a commercial VCSEL array and one 3D-printed or lithographic phase plate on MNIST.

The first decisive check is the four-dataset mixture with task identity hidden. The proposal succeeds if the gated emitter system beats the matched reimplementation of the 82.35% amplitude-mask result while delivering at least five times its throughput.

The second is the iso-power comparison against parallel experts, swept over $K \in \{4, 8, 16\}$ and over source power. We expect sparse illumination to win in the low-power regime, and the crossover point is the finding.

The third is KTH video with the switching penalty swept. It produces an accuracy-versus-switches-per-second curve against the reported 96.27%. That curve also locates the rate at which detector readout, not laser modulation, becomes the binding limit.

The fourth measures expert collapse directly: the effective number of distinct experts used (exponentiated entropy of $\bar p_\theta$) and pairwise output dissimilarity between emitters. We report these with and without continuation, and with BONNI versus multi-start for the outer geometry. This check tests the claim that the optimization decomposition, rather than added capacity, is what keeps the experts distinct.