In classical optimization and statistical learning, nonlinearity is often treated as an obstacle. We linearize complex systems to make them tractable, or we rely on smooth, Lipschitz-continuous functions to guarantee stable gradient descent. However, avoiding nonlinearity can introduce severe performance bottlenecks. When a system relies purely on linear dynamics or linear parameter relationships, it frequently suffers from sluggish convergence—especially when approaching an equilibrium point or trying to disentangle interacting parameters.
To overcome these limitations, a growing body of research across distributed optimization, control theory, and deep learning intentionally injects non-linear mechanisms to accelerate convergence. Whether through non-Lipschitz signum functions in decentralized networks or non-linear gating transformations in Mixture-of-Experts (MoE) models, the core intuition remains the same: well-designed nonlinearities break the mathematical symmetries and dependencies that stall linear systems.
Breaking Parameter Degeneracy in Mixture-of-Experts
In statistical machine learning, Mixture-of-Experts (MoE) architectures use a gating network to route inputs to specialized "expert" sub-networks. Intuitively, we expect the model to learn efficiently by adjusting the gates and experts in tandem. However, when the gating or expert mechanisms are linear or polynomial, a severe mathematical bottleneck emerges: the gradients of the gating weights and the expert parameters become linearly dependent.
This linear dependence triggers a phenomenon known as parameter collapse. Because the optimization algorithm cannot distinguish the statistical contributions of the gate from those of the expert, the parameter estimation rate plummets from a fast polynomial rate to an agonizingly slow logarithmic rate.
Linear/Polynomial Gating:
Input ──> [ Linear Gate ] ──┐
├──> Linear Dependency ──> Parameter Collapse (Logarithmic Convergence)
Input ──> [ Linear Expert ] ┘
Non-Linear Gating Transformation:
Input ──> [ Non-Linear Act ] ──> [ Softmax Gate ] ──┐
├──> Linearly Independent ──> Fast Polynomial Convergence
Input ──> [ Expert Module ] ───────────────────────┘
To resolve this, researchers introduce non-linear transformations directly into the architectural pipeline. As shown by Huy Nguyen et al. in On Least Square Estimation in Softmax Gating Mixture of Experts, selecting non-linear activation functions like sigmoid or tanh for expert modules avoids the linear dependencies that plague polynomial experts, keeping the model within a well-behaved convergence regime.
Similarly, in Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?, Huy Nguyen et al. demonstrated that inserting a non-linear activation function (such as Sigmoid or GeLU) before the softmax operation decouples the learning of gating weights from temperature parameters. This architectural choice prevents linear dependencies between parameter gradients that otherwise lead to sub-polynomial convergence rates.
This strategy of using nonlinearity to enforce parameter identifiability extends across several modern architectures:
- Classification MoEs: In A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts, Huy Nguyen et al. applied a non-linear transformation to the input signal before it enters the gating mechanism. This breaks the linear dependence between gates and experts, ensuring parameters remain distinguishable and restoring efficient polynomial convergence rates.
- Parameter-Efficient Fine-Tuning (PEFT): When tuning small prompts attached to massive frozen models, Fanqi Yan et al. showed in Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts that incorporating non-linear activations into the trainable components prevents the coupling of location and scale parameters. This structural distinction ensures that the trainable parameters remain mathematically distinguishable from the base model.
- Continual Learning: In prompt-based continual learning, Minh Le et al. developed Mixture of Experts Meets Prompt-Based Continual Learning, introducing Non-linear Residual Gates (NoRGa) to replace traditional linear gating functions, bypassing the typical logarithmic convergence bottleneck and significantly boosting statistical efficiency.
Overcoming Equilibrium Sluggishness in Distributed Optimization
A parallel challenge occurs in distributed optimization and resource allocation over networks. In a standard linear consensus protocol, agents update their states based on a linear combination of their neighbors' states. As the network approaches the optimal equilibrium, the differences between agent states shrink toward zero. Because the update step size is proportional to this shrinking difference, the convergence rate decays exponentially, leading to a long, slow asymptotic tail.
To accelerate this process, researchers replace linear consensus updates with non-Lipschitz, signum-based functions (such as $\text{sgn}(x)$ or $\text{sgn}^v(x)$).
$$\text{Linear Update: } \Delta x \propto e \quad \implies \quad \text{As } e \to 0, \, \Delta x \to 0 \quad \text{(Slowing Down)}$$ $$\text{Signum Update: } \Delta x \propto \text{sgn}(e) \quad \implies \quad \text{As } e \to 0, \, \Delta x = \text{Constant} \quad \text{(Full Speed)}$$
The intuition is straightforward: a signum function extracts the direction of the error rather than its magnitude. As the system nears equilibrium and the error becomes tiny, a linear update would shrink to near-zero, but a signum function maintains a robust push toward the target, driving the system to equilibrium much faster.
In Using Non-Lipschitz Signum-Based Functions for Distributed Optimization and Machine Learning: Trade-Off Between Convergence Rate and Optimality Gap, Mohammadreza Doostmohammadian et al. explored this mechanism in distributed gradient tracking. They demonstrated that non-Lipschitz dynamics significantly accelerate convergence near the equilibrium point. However, this constant push introduces a fundamental trade-off: because the step size does not naturally decay to zero, it can introduce a steady-state error or "optimality gap" where the system oscillates slightly around the exact optimum.
Despite this trade-off, non-linear coordination protocols offer massive practical advantages in real-world networks:
- Anytime Feasibility: In resource allocation tasks (like power grid balancing or CPU scheduling), the total amount of resources must remain perfectly balanced at every single iteration. Alireza Aghasi et al. in Accelerated Distributed Allocation and Hamid R. Rabiee et al. in Distributed Automatic Generation Control subject to Ramp-Rate-Limits: Anytime Feasibility and Uniform Network-Connectivity utilized power-of-sign functions and signum nonlinearities to achieve rapid convergence while strictly maintaining this "anytime feasibility" constraint, even over intermittent networks and subject to physical hardware limitations.
- Handling Hardware Constraints: Real-world systems suffer from communication quantization and actuator saturation. Mohammadreza Doostmohammadian et al. in Fast-Convergent Anytime-Feasible Dynamics for Distributed Allocation of Resources Over Switching Sparse Networks with Quantized Communication Links and Alireza Aghasi et al. in Distributed CPU Scheduling Subject to Nonlinear Constraints showed that incorporating sign-based nonlinearities inherently accommodates these constraints, driving networks to an optimal state more rapidly than standard linear consensus methods.
- High-Order Optimization: In continuous-time systems solving high-order optimization problems, Xuezhong Wang et al. showed in Modified gradient dynamic approach to the tensor complementarity problem that integrating monotonically increasing odd nonlinear activation functions, such as power-sigmoid variants, allows the system state to rapidly approach equilibrium points and achieve lower residual errors compared to standard linear models.
Actively Exploiting Internal System Nonlinearities
While the previous examples focus on injecting artificial nonlinearities to improve algorithms, a similar philosophy applies to physical systems with inherent nonlinearities. Traditional control engineering treats internal nonlinearities and cross-channel couplings as disturbances to be canceled out or compensated for through linearization.
However, recent work flips this paradigm, treating internal nonlinearities as an asset rather than a liability. For example, in A Fast Robust Integrated Guidance and Control Scheme for Flight Vehicles Based on Convergence Rate Estimation Mechanism, Tianyu Ye et al. developed a convergence rate indicator that actively quantifies how internal model nonlinearities affect convergence speed. Instead of wasting control effort trying to suppress these complex dynamics, the control system actively transforms and injects them into the controller loop, turning potentially detrimental dynamics into performance enhancers that accelerate convergence.
Similarly, in hardware controller optimization, Ta-Chung Wang et al. demonstrated in Polynomial Method for PLL Controller Optimization that leveraging a non-linear approach via sum-of-squares techniques allows for a precise estimation of the domain of attraction. This leads to controllers that exhibit less overshoot and achieve convergence rates up to five times faster than traditional linear design methods.
This philosophy also manifests in robust signal processing. In Two Improved Normalized Subband Adaptive Filter Algorithms with Good Robustness Against Impulsive Interferences, Yi Yu et al. utilized non-linear cost functions, such as the Maximum Correntropy Criterion and the Minimum Logarithm Criterion. By utilizing these non-linear criteria, adaptive filters can instantly shrink their learning step sizes during sudden noise spikes, providing a superior trade-off between stability and speed without sacrificing convergence velocity under normal conditions.
Conclusion
The unifying insight across these diverse fields is that linearity is often a bottleneck in disguise. In statistical learning, linear relationships cause parameter gradients to collapse into unidentifiability, stalling optimization. In distributed systems, linear updates suffer from diminishing returns near equilibria. By strategically embracing non-linear transformations—whether through architectural choices like non-linear gating or algorithmic choices like non-Lipschitz dynamics—we can break these dependencies, unlock faster convergence, and build more robust systems.
Key Foundational Work
Polynomial Method for PLL Controller Optimization (2011) — contributed 1 concept elements to this direction.
Two Improved Normalized Subband Adaptive Filter Algorithms with Good Robustness Against Impulsive Interferences (2016) — contributed 1 concept elements to this direction.