Defect-Aware Scheduling for Photonic All-to-All Switches in Mixture-of-Experts Training

Large language models built as a mixture of experts send each token to a few specialist sub-networks, which are spread across many GPUs. Every layer therefore requires an all-to-all exchange, in which each GPU ships tokens to every other GPU and then receives the results back. This exchange often takes more time than the computation itself. Reconfigurable optical switches are a promising way to make it cheaper. Integrated photonic switches, however, are built from nanometer-scale waveguides, and small fabrication errors in width or etch depth shift optical phases. As a result, some switch configurations on a given chip leak light into the wrong output port or lose power. Current proposals treat the switch as perfect. The usual hardware answer is to demand tighter fabrication or to trim every element back to ideal, which is costly in both effort and power.

We propose the opposite direction. Measure each fabricated chip's imperfections once, then let the communication scheduler and the placement of experts on GPUs work around them. Mixture-of-experts traffic is skewed, local, and fairly predictable from step to step. A training job therefore needs only a small set of switch configurations, and the scheduler can choose a set that a particular imperfect chip realizes well. The central question is quantitative: how much fabrication variation can a switch tolerate before MoE training throughput drops, with and without scheduling that knows about the defects?

Core Claim

All-to-all schedules are already built from conflict-free one-to-one stages. FLASH splits the inter-server traffic matrix into a sequence of permutations using Birkhoff decomposition, a method that writes a traffic matrix as a weighted sum of one-to-one matchings. A photonic switch carries out one permutation per stage. On a chip with fabrication errors, each permutation $\pi$ has its own achievable line rate $r(\pi)$, which is set by worst-path insertion loss and crosstalk-induced bit errors. The rate is high when $\pi$ uses well-behaved paths and low when it uses defective ones. We claim that replacing the ideal decomposition with a rate-weighted one recovers most of the throughput lost to fabrication error. A useful form is to choose stage permutations $\pi_s$ and durations $t_s$ to

$$\min \sum_s t_s \quad \text{subject to} \quad \sum_s t_s\, r(\pi_s)\, P_{\pi_s} \ge D,$$

where $D$ is the measured or forecast traffic matrix and $P_{\pi_s}$ is the permutation matrix of stage $s$. The scheduler can also reassign experts to GPU ports, so that the heaviest-communicating pairs map onto the best optical paths. Our claim is that the two levers together relax the tolerable phase-error spread by a measurable factor, at fixed throughput and fixed trimming-power budget.

This approach has precedent in the fabrication literature, which supplies the design philosophy rather than the method. Optimal-control pulse design for silicon donor qubits relaxed the required donor spacing from 10–20 nm to about 30 nm by shaping control signals instead of tightening placement (Optimal control of the silicon-based donor-electron-spin quantum computing). Broken-symmetry hBN metasurfaces spread field enhancement over a wide region, so that emitter placement and etch errors matter less (Cao et al.). Both accept the imprecision and design the device or its control around it. We apply the same idea one layer higher, in the communication runtime.

Relation to Prior Work

MixNet is the closest work. It adds optical circuit switches to an electrical fabric, forecasts per-layer MoE traffic, and points optical circuits at the heaviest GPU pairs. It assumes a fixed switching latency and uniform loss, and it does not model per-device imperfections. We keep its traffic-forecasting idea and its practice of hiding switch reconfiguration behind attention computation. We change the switch model from ideal to measured-per-chip, and we move from coarse circuit switching to integrated photonic meshes, where fabrication error is the dominant cost.

FLASH supplies the scheduling primitive, intra-server pre-balancing followed by Birkhoff stages. It assumes every permutation runs at full link rate, and our contribution is to weight permutations by what the hardware can actually do.

Several lines of work reduce or hide the traffic instead. Head Parallel makes communication deterministic and balanced. Grouter freezes a distilled router so that schedules can be precomputed. FoldMoE overlaps dispatch with attention. These methods are complementary: the more predictable the traffic, the smaller the set of switch configurations that must be realized well. We use them as traffic regimes to test, not as competitors.

Research Program

The work has three parts.

The first part is a switch model. We will build a Monte Carlo simulator of Mach-Zehnder interferometer meshes at 16 to 64 ports, in which each element's phase error is drawn from a distribution parameterized by a spread $\sigma$ that stands in for waveguide width and etch variation. The simulator outputs per-path insertion loss and crosstalk. These map through a standard receiver model to bit error rate and to the highest rate that keeps forward error correction overhead bounded, which gives $r(\pi)$. We will also model per-element thermo-optic trimming under a total power budget, because trimming is the conventional fix.

The second part is the scheduler. It will be a rate-weighted Birkhoff decomposition, solved greedily by repeatedly extracting the maximum-weight matching under $r$. We will combine it with an expert-to-port assignment step that solves a quadratic assignment problem over the forecast traffic matrix and the measured path quality. Characterizing a chip needs only the transmission of individual paths, which on-chip power monitors can read in a calibration pass, and the pass can be repeated when thermal drift is detected.

The third part is a co-simulation. MoE traffic traces drive the switch model and the scheduler, and the output is all-to-all completion time and end-to-end iteration time.

Baselines

The naive reference applies the ideal-switch FLASH-style Birkhoff schedule, with the stock expert placement, to a defective chip. This is what current systems would do. Its throughput as a function of $\sigma$ shows how much fabrication error costs when nothing is done about it.

The strongest existing method is MixNet-style forecasting combined with FLASH scheduling, run with full per-element trimming up to the same power budget our method is allowed and with electrical fallback for paths that fail. It is the strongest competitor because it combines the best published traffic handling with the standard hardware remedy. All methods receive the same traces, chip instances (the same random seeds), power budget, and forecast quality. For reference numbers, MixNet reports a performance-per-dollar gain over non-blocking fat-trees of 1.2×–1.5× at 100 Gbps and 1.9×–2.3× at 400 Gbps (MixNet). FLASH reports up to 4.48× training throughput on MI300X and a 221 µs schedule for 64 GPUs (FLASH). We found no reported result on Lacuna for MoE all-to-all throughput on integrated photonic switches with fabrication error, so the defect-sensitive baseline curve will come from our own simulation.

Beating the strongest baseline means one of two things. Either we reach the same all-to-all completion time at a $\sigma$ at least 1.5× larger, or we match its throughput at a substantially lower trimming power, with schedule computation that stays within a few hundred microseconds so it remains comparable to FLASH's overhead.

Two ablations isolate the ingredients: defect-aware scheduling with stock placement, and defect-aware placement with ideal scheduling. An oracle that knows the true $r(\pi)$ for every permutation bounds the loss caused by sparse calibration measurements.

Why not just…?

Why not just trim every switch element back to ideal? This is the most serious threat. If trimming power and calibration time are cheap relative to the throughput at stake, scheduling adds little. The experiments sweep the trimming budget explicitly. If the full-trimming baseline matches our method at every budget we can realistically assign, the claim is false and we will report that. Our expectation is that trimming fails in the regime of large $\sigma$ and many ports, where the number of elements and their drift make full correction expensive. Scheduling only has to make the few configurations a job actually uses work well, not every possible configuration.

Why not just send bad paths over the electrical network? MixNet already keeps an electrical fabric, and it reports a 3–6% impact when traffic is rerouted around faults (MixNet). That figure, however, concerns discrete failures, not a continuous degradation spread across many paths. We include electrical fallback inside the strongest baseline. If fallback alone closes the gap at realistic $\sigma$, the optical defect model is unnecessary, and the experiment will show this directly.

Why not just remove the traffic with Head Parallel or Grouter? Head Parallel's reported gains were measured on 4–8 H100s and came mostly from compute kernels. Grouter needs a pretrained teacher model. Neither eliminates cross-node exchange at cluster scale (Head Parallel, Grouter). We run their traffic patterns through the same pipeline. If deterministic, balanced traffic makes the defect-aware gain disappear, that would be an informative negative result.

Proposed Experiments

Traffic comes from two sources. The first is Megatron-LM MoE traces and synthetic Zipfian-skewed traces, matching FLASH's evaluation. The second is traces we generate by training small MoE models on FineWeb-Edu, with standard routing, Grouter-style frozen routing, and Head-Parallel-style balanced exchange. The primary metrics are all-to-all completion time, iteration time, the largest $\sigma$ that keeps throughput within 5% of an ideal switch, and trimming power. Each configuration is evaluated over at least 50 chip instances per $\sigma$, and we report the spread across instances.

The decisive check is the curve of tolerable $\sigma$ against trimming budget for our method and the strongest baseline. A consistent rightward shift supports the claim, and overlapping curves refute it.

A second check tests forecast robustness. We inject traffic forecasting errors, and we also test stale chip characterizations that simulate thermal drift, to see whether the gain survives the imperfect knowledge a real deployment would have.

A third check scales the mesh from 16 to 64 ports. The goal is to find whether the benefit grows with port count, as expected if the fraction of permutations that land on degraded paths rises with mesh size, or whether it shrinks.