How Much of a Neural Dispersion Equalizer's Gain Survives Contact with Hardware?

Optical fiber spreads a short light pulse out in time because different colors in the pulse travel at slightly different speeds. This effect is called chromatic dispersion, and at 100 Gb/s it smears neighboring symbols into each other. A growing body of work trains neural networks to undo this smearing. These networks report large improvements in bit error rate (BER) over classical linear filters. Almost all of those numbers, however, come from software simulators that assume three convenient things: infinite numerical precision, clean white receiver noise, and a fiber whose properties never change.

This proposal measures how much of a neural equalizer's simulated advantage survives as each assumption is removed, ending on a physical fiber link. It also tests whether the surviving fraction can be predicted from one physical number: how far the link sits beyond the reach at which dispersion, rather than noise, becomes the limiting impairment. The short version is a falsifiable claim. Neural gains reported in simulation are inflated most where dispersion is weakest. Making them transfer requires carrying a lesson from classical filter design into network training: keep the equalizer's response small outside the signal band.

Core Claim

The neural equalizer work this note scrutinizes evaluates exclusively in simulation:

Both papers name physical validation as open. Neither says which simulator assumption hides which loss.

Two other papers supply the reason to expect a structured, predictable gap rather than uniform degradation.

PAM-N—Fundamental Limits in Chromatic Dispersive-Uncompensated Channels shows that uncompensated PAM-N links are limited by the product $\beta_2 B^2 L$. Here $\beta_2$ is the fiber's dispersion coefficient, $B$ the symbol rate and $L$ the length. The ceiling is $C(N)\approx 0.212$ for PAM-4, which is roughly 3.7 km at 53 GBaud in the C-band.

Electrical Equalization Analysis of PAM-4 Transmission in Short-Reach Optical Systems finds the opposite emphasis at 0.5–3.5 km. There, the noise admitted by the receiver filter, not dispersion, dominates: a wider receiver filter costs 1.6–1.8 dB.

We reconcile the two papers with a single coordinate, the normalized dispersion

$$ \delta = \frac{\beta_2 B^2 L}{C(N)}, $$

so that $\delta<1$ means the link is inside the uncompensated dispersion limit and $\delta>1$ means it is beyond it. We also define gain retention

$$ R(\delta) = \frac{G_{\text{real}}(\delta)}{G_{\text{sim}}(\delta)}, $$

where $G$ is the neural equalizer's improvement over the strongest linear baseline. We express $G$ in dB of required received optical power at a fixed pre-forward-error-correction BER target. $G_{\text{sim}}$ is measured in an idealized simulator and $G_{\text{real}}$ on hardware.

We hypothesize the following:

  • Below the limit. For $\delta<1$, $R$ collapses toward zero, because much of the simulated "dispersion gain" is actually noise shaping. A tuned FFE plus a well-chosen receiver filter recovers that gain.
  • Beyond the limit. For $\delta>1$, $R$ stays substantially positive but is cut by fixed-point quantization.
  • Training that transfers. A network trained on randomized link parameters, quantized during training, and penalized for out-of-band gain retains most of its gain on hardware. Unconstrained networks do not.

The out-of-band penalty comes from Dispersion Compensation FIR Filter With Improved Robustness to Coefficient Quantization Errors. That work shows that linear dispersion filters with large gain outside the signal band amplify rounding errors when taps are stored in 3–5 bits. It is the method-supplying citation here. We apply its constraint to the locally linearized frequency response of a neural equalizer, which that paper, being linear-only, never considered.

Research Program

The program has three parts: a reality ladder, a gap attribution analysis, and a transfer-oriented training recipe.

The reality ladder. This is a sequence of five evaluation environments for the same intensity-modulation direct-detection PAM-4 link at about 53 GBaud. Fiber spools give lengths spanning $\delta$ from about 0.3 to above 3: 1, 2, 3.5, 5, 8 and 11 km, the last two matching the Shao et al. reaches.

  • Rung 0: dispersion only. This is the "CD-only" VPItransmissionMaker scenario of Shao et al., with floating-point equalizers.
  • Rung 1: the "realistic" simulation. This is Shao et al.'s scenario with Kerr nonlinearity, modulator chirp and timing jitter.
  • Rung 2: measured noise and converters. Rung 1's synthetic noise is replaced by noise recorded from the physical photodiode and amplifier chain. Analog-to-digital converter quantization at the testbed's resolution is added.
  • Rung 3: fixed-point equalizers. Rung 2 is run with the equalizer itself in bit-exact fixed point (4- and 6-bit weights and activations), mirroring the precision regime of Sheikh et al.
  • Rung 4: the physical link. A lab link with an electro-absorption modulated laser, spools, a variable optical attenuator, and real-time oscilloscope capture. The physical link is captured repeatedly over several days, so slow thermal and polarization drift is present.

Rung 4 processing is offline, but it uses the same bit-exact fixed-point code as Rung 3. That makes Rung 3 to Rung 4 a clean test of what simulation still misses.

The attribution analysis. For each rung and each $\delta$, we report $G$ for every equalizer. The drop between consecutive rungs is a loss attributable to one assumption. The output is a map of where along the ladder each method's gain disappears, plotted against $\delta$.

The training recipe. We train the network on Rung-1 simulation with three changes:

  • Randomized link parameters. Fiber length, chirp, receiver bandwidth and noise spectral shape are drawn from ranges wider than the testbed.
  • Quantization-aware training. Training runs at the target bit width.
  • An out-of-band penalty. We add a penalty on the energy of the network's input-output Jacobian response outside the signal bandwidth. This is the neural analogue of the constraint in Sheikh et al.'s least-squares constrained FIR design.

The recipe deliberately never sees hardware data, so that any transfer it achieves is attributable to the training choices.

Baselines

The naive reference is the strongest linear equalizer a practitioner would deploy at low cost. It is an FFE of roughly 6 taps, the point beyond which Ayoub et al. report no further gain, adapted by least mean squares, with the receiver filter bandwidth swept and chosen per $\delta$. Choosing the receiver filter per link is what makes this reference strong rather than a straw man: Ayoub et al. attribute 1.6–1.8 dB to that choice alone. It gets the same captured data and the same filter-sweep budget as every other method.

The strongest linear dispersion-specific competitor is the least-squares constrained FIR filter of Sheikh et al. It is closed-form, robust to coarse quantization, and reported to beat the industry-standard band-limited least-squares filter by 0.44 dB at BER $10^{-3}$ for 16-QAM with 4-bit taps. That result comes from a 250 km coherent simulation and is not directly transferable to direct detection, so we re-derive the filter for our link. It counts as beaten only if a neural method exceeds it at Rung 4 under equal bit width.

The strongest existing neural method is FC-SCINet, reimplemented with Shao et al.'s architecture. It is trained on the same Rung-1 data with the same hyperparameter search budget, measured in trials and GPU-hours, as our recipe. Its reported simulated advantage is an 88.87% BER improvement over FFE at 5 km and 87.5% at 11 km, from A Novel Machine Learning-based Equalizer for a Downstream 100G PAM-4 PON. We found no reported result on Lacuna for any neural chromatic-dispersion equalizer on a physical testbed, so its Rung 4 number is itself a finding.

Ablations remove each of the three training changes (randomization, quantization-aware training, out-of-band penalty) one at a time from our recipe. The out-of-band ablation isolates the key ingredient. Our recipe "wins" only if two things hold:

  • at $\delta>1$ on Rung 4, it beats the tuned FFE, the constrained FIR filter and FC-SCINet at matched bit width and matched multiplications per symbol;
  • its retention $R$ exceeds FC-SCINet's by a margin larger than the day-to-day spread of repeated testbed captures.

Why not just…?

Why not just tune the FFE and receiver filter and skip neural equalizers entirely? This is a real threat, and for short links we expect it to win. Ayoub et al. find that receiver filter bandwidth, not dispersion, governs performance below a few kilometers. The ladder tests this directly: if tuned-FFE matches every neural method at $\delta<1$ on Rung 4, that is a reportable result, not a failure. Beyond the dispersion limit the alternative should break. Ayoub et al. report BER around 0.02 at 3.5 km even with decision feedback, and Granot and Katz place the uncompensated PAM-4 ceiling near 3.7 km. If tuned-FFE still matches at 5–11 km on hardware, the case for neural equalizers in this regime fails, and we will say so.

Why not just add colored noise and quantization to VPItransmissionMaker and stay in simulation? This is exactly Rungs 2 and 3. The question is whether they close the gap to Rung 4. Sheikh et al. show that precision effects are invisible in floating-point simulation. The networking evidence in ACM MMSys 2024 Bandwidth Estimation in Real Time Communications Challenge suggests that sophisticated emulators still miss non-stationary real conditions. That challenge reports emulator-trained bandwidth estimators failing on the live internet. If Rung 3 predicts Rung 4 within capture-to-capture variation, then richer simulation suffices. That would be a useful, cheaper conclusion, and the ladder is designed to detect it.

Why not just train directly on captured testbed data, as the bandwidth-estimation community moved to real telemetry? On a single lab link this is a strong option, and we include it as an oracle: a network trained on day-one captures and tested on later days. Its weaknesses are the per-link data cost and drift across days, which repeated captures expose. We report a data-efficiency curve showing how many real captured symbols the oracle needs to match the simulation-trained recipe. If very few suffice, the practical recommendation becomes "fine-tune on a short capture," and our recipe's value shrinks to providing a better starting point.

Why not just quantize FC-SCINet after training? Post-training quantization is the cheapest route to hardware and is run as an ablation at 4 and 6 bits. Sheikh et al.'s analysis predicts that equalizers with large out-of-band response degrade sharply under coarse quantization. Comparing post-quantized FC-SCINet against our recipe, with and without the penalty, tests whether that linear-filter result carries over to deep networks.

Proposed Experiments and Decisive Checks

All methods are evaluated on the same symbol sequences. We reuse the long pseudo-random pattern structure and the 28.7 dB path loss of the Shao et al. setup for Rungs 0–1 at 5 and 11 km. We extend the reach grid down into Ayoub et al.'s 0.5–3.5 km range and use 53 GBaud to match Granot and Katz's operating point.

For each method, rung and $\delta$, the metrics are:

  • BER versus received optical power;
  • the required-power penalty at a fixed pre-forward-error-correction BER target;
  • multiplications per symbol at the deployed bit width;
  • the standard deviation of BER across days of Rung 4 captures.

There are four decisive checks.

The retention curve. Does $R(\delta)$ for FC-SCINet rise with $\delta$, near zero below the dispersion limit and substantially positive above it? A flat curve would refute the noise-versus-dispersion account and mean the gap is not organized by the Granot–Katz coordinate.

Rung-by-rung attribution. Which single rung transition accounts for most of each method's loss? This tells the field which simulator assumption to fix first.

The out-of-band penalty at hardware. Does the penalty raise Rung 4 retention at 4 bits by more than the capture-to-capture spread? Gains that appear in simulation but not on hardware would count against it.

Sim-to-hardware prediction. Does Rung 3 predict Rung 4 well enough that future papers could report a cheap, standardized, bit-exact, measured-noise simulation in place of a testbed? This would address the missing cross-environment benchmark that the simulation-validation literature identifies.

Alongside the code and Rung 2 noise recordings, we will release the Rung 4 captures. As far as the file shows, these would be the first shared physical dataset for comparing neural dispersion equalizers.