Abstract
This paper presents the Regularized Mixed Newton Method (RMNM), an optimization framework for complex-valued spaces. By extending real-valued objective functions into the complex plane and applying a novel "complex-repulsive" regularization, the proposed algorithm effectively bypasses local minima that typically trap standard second-order methods. Experimental results demonstrate that RMNM achieves superior global convergence and significant computational speedups compared to the Ordinary Newton Method (ONM).
Introduction
Optimization in non-convex landscapes is frequently hindered by local minima and saddle points. While second-order methods like the Ordinary Newton Method (ONM) offer powerful convergence properties, they are often computationally expensive and prone to getting stuck in sub-optimal regions. This research introduces a regularized version of the Mixed Newton Method (MNM), specifically designed to leverage complex-space optimization for real-world engineering problems.
The core challenges addressed include the potential degeneracy of the "mixed Hessian" and the need to apply complex-space advantages to problems defined in real space ($\mathbb{R}^n$).
Methodology: The Regularized Mixed Newton Method (RMNM)
The authors introduce several technical innovations to enhance the Mixed Newton Method:
- Mixed Hessian Regularization: By adding a positive definite matrix $P$ to the mixed derivative $\frac{\partial^2 f}{\partial z \partial \bar{z}}$, the authors ensure matrix invertibility and enable step-size damping, akin to the Levenberg-Marquardt algorithm.
- Complex Extension with Penalty: Real-valued problems $F(x)$ are extended to the complex plane $f(z)$ with an added regularization term: $2\gamma^2 \sum \cosh(2 \text{Im}(z_l))$. This term acts as a penalty that pushes the optimizer back toward real parameters while allowing it to "tunnel" through complex space to find global paths.
- Computational Efficiency: MNM requires computing only one-fourth of the full Hessian matrix entries compared to standard Newton methods, significantly reducing the per-iteration cost.
Key Insights: Saddle Point Repulsion and Shortcuts
A critical insight of this work is that many local minima in real space transform into saddle points when the function is extended to complex space. Theorem 2 in the paper proves that RMNM is mathematically "repulsive" at saddle points, allowing the optimizer to naturally flow away from these traps toward the global minimum. Furthermore, by allowing parameters to temporarily become complex, the optimizer can find "shortcuts" through the complex plane that are unavailable in a real-only landscape.
Experimental Results
The RMNM was evaluated across synthetic non-convex polynomials, a real-world Digital Pre-Distortion (DPD) task, and the Abalone regression dataset.
Global Convergence
In synthetic toy examples, RMNM demonstrated remarkable robustness, reaching the global minimum from 100% of tested initial points. In contrast, the standard ONM frequently stalled in local minima.
Speed and Model Performance
For the DPD telecommunications task, the Levenberg-Marquardt variant (LM-MNM) was 3.5 to 5 times faster per iteration than the standard Newton approach. In the Abalone regression task, the complex-valued model trained with RMNM achieved an $R^2$ score of 0.669, outperforming the best real-valued model's score of 0.639.
Conclusion and Future Work
The Regularized Mixed Newton Method offers a powerful alternative for non-convex optimization by exploiting the properties of complex space. While currently optimized for functions expressible as sums of squares of holomorphic functions, future research into diverse complex-space regularizations may broaden its applicability. Additionally, addressing sensitivity to initial imaginary parameter values remains an area for further refinement.
Summary updated