In traditional communications engineering, the recipe for overcoming noise is simple: turn up the transmit power. Higher power yields a higher signal-to-noise ratio (SNR), which translates directly to greater channel capacity.
This linear assumption underpins most standard digital signal processing (DSP) models, such as standard finite impulse response (FIR) filters designed to counteract chromatic dispersion. As [Alireza Sheikh] et al. note in Dispersion Compensation FIR Filter With Improved Robustness to Coefficient Quantization Errors, these linear models perform beautifully in simulations but fail to account for the complex, high-power distortions that occur over transoceanic distances.
When applied to physical glass fibers, the linear assumption hits a hard physical ceiling known as the Kerr nonlinearity. Glass alters its refractive index in response to the intensity of the light passing through it. As a result, high-power optical pulses inadvertently modulate their own phases (Self-Phase Modulation) and interfere with neighboring channels (Cross-Phase Modulation). Instead of clarifying the signal, increasing the power past a certain threshold introduces severe nonlinear phase noise, jitter, and cross-talk. This phenomenon creates a fundamental performance ceiling often called the "capacity crunch."
To push past this limit, researchers are shifting away from treating nonlinearity as a system failure. Instead, they are developing methods to predict, counteract, or directly exploit the nonlinear physics of optical transmission.
Physical Interventions: Balancing Dispersion and Nonlinearity
One way to defeat fiber nonlinearity is to use it as a stabilizing force rather than an impairment. In a standard fiber, chromatic dispersion causes different wavelengths of light to travel at different speeds, melting a crisp pulse into a smeared blur. Concurrently, the Kerr effect chirps the pulse's frequencies.
If engineered precisely, these two destructive forces can perfectly cancel each other out, creating a self-reinforcing wave called a soliton.
[Dispersion: Spreads Pulse] <---> [Kerr Effect: Compresses Pulse]
|
[Perfect Balance]
|
v v v v v v v v
Stable Soliton Wave
However, as solitons travel thousands of kilometers, they accumulate noise from optical amplifiers, causing their arrival times to fluctuate—a phenomenon known as non-linear jitter accumulation. To stabilize these pulses over transoceanic distances, [A. Gray] et al. demonstrated the use of in-line Nonlinear Optical Loop Mirrors (NOLMs) in 40GB/S Transmission over 4000KM of standard fibre using in-line nonlinear optical loop mirrors.
A NOLM acts as an all-optical threshold gate. High-intensity soliton pulses pass through cleanly, while low-intensity background noise and timing jitter are reflected away. By establishing this "autosoliton" regime, the authors suppressed exponential jitter growth over a 4,000-to-6,000 km reach, as further detailed by [A. Gray] et al. in Evolution of timing jitter in nonlinearly-guided, dispersion managed transmission systems.
Other physical approaches target the amplification and transmission medium directly:
- Phase-Sensitive Amplifiers (PSAs): Traditional amplifiers boost the signal but inject significant noise. [Samuel L. I. Olsson] et al. showed in Long-haul optical transmission link using low-noise phase-sensitive amplifiers that PSAs can simultaneously minimize amplifier noise and mitigate deterministic fiber nonlinearities, extending error-free transmission reach by more than five times.
- Hollow-Core Fibers (AR-HCF): Instead of fixing the distortions caused by glass, some architectures eliminate the glass core entirely. [Dawei Ge] et al. achieved a 1-Tb/s single-channel transmission over 10,714 km in 1-Tb/s/λ Transmission Over Record 10714-km AR-HCF by guiding light through an air-filled hollow core, drastically reducing the material-level Kerr interactions that constrain solid-core fibers.
Computational Equalization: Mirroring Physics with Photonic Computing
When physical mitigation is insufficient, receivers must untangle the distorted waveforms computationally. While linear noise approximations suffice for short-reach metro networks, they fail over long-haul links where nonlinearities dominate, as observed by [Hideki Nishizawa] et al. in Fast WDM provisioning with minimal probing: the first field experiments for DC exchanges.
However, executing deep neural networks at line rates of hundreds of gigabits per second introduces an unsustainable electronic power and latency penalty. To solve this, researchers have turned to Photonic Reservoir Computing (PRC).
PRC offloads the heavy mathematical lifting from electronic silicon to optical hardware. Instead of calculating complex nonlinear inversions digitally, a PRC system passes the distorted optical signal directly through an analog nonlinear medium—such as cascaded, injection-locked semiconductor lasers. Because the laser hardware possesses its own rich, high-speed nonlinear dynamics, it naturally mirrors and untangles the Kerr distortions experienced by the signal in the fiber.
Distorted Signal ---> [ Analog Laser Reservoir ] ---> Simple Linear Layer ---> Clean Signal
(High-speed optical physics (Low-power digital
unwinds Kerr distortion) matrix multiplication)
This architecture has evolved rapidly:
- Deep Architectures: [Yi-Wei Shen] et al. developed a multi-layered PRC in Deep photonic reservoir computing recurrent network, demonstrating that cascaded optical layers outperform shallow electronic alternatives.
- Wavelength Multiplexing: To scale processing bandwidth, [Rui-Qian Li] et al. introduced a scalable architecture in Scalable wavelength-multiplexing photonic reservoir computing that leverages the concurrent light modes of a single Fabry-Perot laser to process multiple data channels in parallel.
- Advanced Modulation Support: This parallel processing capability was extended to complex multi-level signals (16-QAM) at data rates of 240 Gbps by [Ruiqian Li] et al. in Deep photonic reservoir computer for nonlinear equalization of 16-level quadrature amplitude modulation signals.
For environments where all-optical computing is unavailable, researchers focus on reducing digital complexity. [Xinyu Xiao] et al. proposed Meta-DSP: A Meta-Learning Approach for Data-Driven Nonlinear Compensation in High-Speed Optical Fiber Systems, using meta-learning to allow a digital model to generalize across different data rates without costly retraining. For short-reach systems, [Junde Lu] et al. bypassed neural networks entirely in A Low-Complexity Receiver-Side Lookup Table Equalization Method for High-Speed Short-Reach IM/DD Transmission Systems, replacing intensive computations with a hardware-efficient receiver-side lookup table.
Waveform Engineering: Designing for a Nonlinear Medium
A third paradigm seeks to prevent nonlinearities from manifesting by altering the shape and structure of the transmitted waveforms.
Constellation and Modulator Optimization
Standard modulation formats like 16-QAM arrange data points uniformly in a grid. However, because the Kerr effect causes amplitude-dependent phase rotation, outer constellation points experience more distortion than inner ones.
In Design of APSK Constellations for Coherent Optical Channels with Nonlinear Phase Noise, the authors addressed this by engineering custom Amplitude Phase-Shift Keying (APSK) constellations. By strategically placing signal points on concentric rings and optimizing their spacing, they minimized the impact of nonlinear phase noise. They demonstrated that intentionally sacrificing or reshaping the outer points yields superior overall system performance in uncompensated links.
Similarly, hardware components can be driven intentionally into nonlinear states if the input waveform is pre-compensated. [Eduard Sonkin] et al. demonstrated in MZM Optimization of PAM-4 Transmission in Data Center Interconnect that operating a Mach-Zehnder Modulator deep within its nonlinear regime—combined with digital pre-distortion—can significantly optimize the optical signal-to-noise ratio for data center links.
PAPR Reduction
High Peak-to-Average Power Ratio (PAPR) is a major catalyst for nonlinear distortion. In multi-carrier systems like coherent optical OFDM, independent subcarriers occasionally align in phase, generating massive power spikes. These spikes drive the fiber and optical components into severe nonlinear distortion.
To combat this, [Omar Alnaseri] et al. designed deep learning autoencoders in Deep Learning Autoencoders for Reducing PAPR in Coherent Optical Systems. The autoencoder learns an optimized mapping that bounds the peak power of the transmitted signal, preventing the waveform from ever triggering the severe nonlinearities of the optical components.
The Nonlinear Fourier Transform
Perhaps the most radical approach to waveform engineering is to abandon the linear Fourier transform altogether. Standard wavelength-division multiplexing treats the fiber as a collection of independent linear frequencies. But because the channel is nonlinear, these frequencies inevitably mix and cause cross-talk.
[Sander Wahls] proposed a solution to this in Fast inverse nonlinear Fourier transform for generating multi-solitons in optical fiber by utilizing the Nonlinear Fourier Transform (NFT).
Linear Fourier Transform: Maps data to independent frequencies ---> Mixed by Kerr Effect (Cross-talk)
Nonlinear Fourier Transform: Maps data to nonlinear eigenvalues ---> Propagate independently through fiber
The NFT mathematically decomposes a signal into "nonlinear frequencies" (eigenvalues) based on the nonlinear Schrödinger equation. When these custom-shaped multi-soliton waveforms propagate through the fiber, they do not interfere with one another; they travel down the nonlinear channel completely isolated. By reducing the computational complexity of the inverse NFT algorithm from quadratic to log-linear ($O(D \log^2 D)$), this method provides a viable mathematical framework for bypassing the capacity crunch entirely.
Key Foundational Work
Evolution of timing jitter in nonlinearly-guided, dispersion managed transmission systems (2004) — contributed 1 concept elements to this direction.
40GB/S Transmission over 4000KM of standard fibre using in-line nonlinear optical loop mirrors (2003) — contributed 1 concept elements to this direction.