Abstract
In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Moreover, we provide a novel theoretical characterization of a failure mode of eliminating suboptimal local minima via an additional theorem and several examples. This paper also introduces a novel proof technique based on the perturbable gradient basis (PGB) necessary condition of local minima, which provides new insight into the elimination of local minima and is applicable to analyze various models and transformations of objective functions beyond the elimination of local minima.
Introduction
Training deep neural networks is notoriously difficult because their objective functions are non-convex and high-dimensional. In classical optimization, non-convexity usually implies the existence of "bad" local minima—valleys in the loss landscape where an optimizer, such as Gradient Descent, can become trapped despite the existence of superior global solutions.
While deep learning performs remarkably well in practice, theoretical guarantees for avoiding these traps have historically relied on restrictive assumptions, such as massive over-parameterization or simplified loss functions. This research addresses a fundamental question: Can we modify any network architecture to ensure that every local minimum is a global one?
Methodology: The "One Neuron" Modification
The authors propose a simple yet powerful architectural modification. For a standard neural network $f(x; \theta)$, they define a modified network: $$\tilde{f}(x; \tilde{ heta}) = f(x; \theta) + g(x; a, b, W)$$ where $g$ is a "special" neuron added to each output unit. This added neuron utilizes an exponential activation function: $a_k \exp(w_k^\top x + b_k)$.
To analyze this modification, the researchers introduced the Perturbable Gradient Basis (PGB) necessary condition. This mathematical tool examines the tangent space of the network's outputs. They demonstrate that the high expressivity of the added exponential neurons effectively "fills in" the gaps in the loss landscape that typically harbor local minima.
Crucially, the authors prove (Theorem 1) that under the mild assumption of a convex loss function (like Mean Squared Error), every local minimum of the modified network is a global minimum of the original network. Furthermore, at any local minimum, the coefficient $a$ of the extra neuron automatically becomes zero. Consequently, the modification is self-correcting; the extra neuron "vanishes" upon reaching a solution, leaving a global minimum of the original, unmodified network.
Key Insights and Failure Modes
- Automatic Vanishing: The modification does not require manual removal of the extra neurons post-training. The mathematical properties of the system force their influence to zero at any valid local minimum.
- Divergence as a Signal: The paper identifies a specific "failure mode" (Theorem 4). If a global minimum is not easily reachable, the weights of the extra neurons ($a, b, W$) may diverge toward infinity. Rather than a drawback, the authors argue this is a diagnostic feature: detecting exploding weights is significantly easier than identifying entrapment in a subtle, suboptimal local minimum.
- PGB Condition: This new proof technique offers a more general framework than previous methods, potentially applicable to analyzing other network transformations and regularization strategies.
Empirical Validation
The theoretical findings were validated using synthetic "bad-case" examples and real-world datasets:
- Semeion: A dataset of 1,593 handwritten digits.
- KMNIST (Kuzushiji-MNIST): A challenging dataset of 70,000 Japanese cursive characters.
In experiments on Semeion, the modified "elimination" method significantly shifted the distribution of training losses toward zero compared to standard training. On KMNIST, the authors demonstrated a hybrid optimization strategy: by monitoring weight norms, they could detect divergence, trigger restarts, and successfully navigate to superior optimization subspaces that standard methods failed to reach.
Conclusion and Future Work
This work provides a robust theoretical foundation for eliminating bad local minima in deep learning. However, practical implementation faces challenges such as numerical stability—exponential functions can lead to exploding gradients in standard 32-bit floating-point arithmetic. Furthermore, while the existence of bad local minima is eliminated, the speed of convergence for Gradient Descent is not guaranteed. Future research will focus on automated hyperparameter tuning for the added neurons' regularizers and improving the numerical robustness of the implementation.
Summary updated