Numerical Analysis near Singularities in RBF Networks

Weili Guo, Haikun Wei, Yew-Soon Ong, Jaime Rubio Hervas, Junsheng Zhao, Hai Wang, Kanjian ZhangJMLR 2018 · OpenReview

TL;DR

This paper delves into the impact of "singularities" in the parameter space of Radial Basis Function (RBF) networks on their learning dynamics. By deriving the Fisher information matrix and conducting extensive numerical simulations, the authors demonstrate that these singularities, particularly "elimination singularities," significantly hinder learning, often leading to plateaus and slow convergence. This work provides a foundation for understanding and potentially mitigating these issues in both shallow and deep neural networks.

What Problem Does This Paper Address?

Training neural networks can sometimes lead to peculiar behaviors like extremely slow learning or getting stuck in "plateaus." These phenomena are often linked to "singularities" in the network's parameter space, where the mathematical properties of the learning process break down. While it's known that these singularities exist and affect learning, their precise influence and the mechanisms behind them, especially in different types of neural networks, are not fully understood. This paper aims to systematically analyze how two specific types of singularities – overlap and elimination singularities – affect the learning dynamics of RBF networks, a common type of feedforward neural network. Understanding these effects is crucial for improving the training of not only RBF networks but also more complex deep learning models.

Methods

The authors employ a two-pronged approach:

  1. Theoretical Analysis: They derive the explicit analytical expression for the Fisher information matrix (FIM) of RBF networks. The FIM is a key tool in information geometry that quantifies the amount of information a statistical model carries about its parameters. Singularities in the parameter space correspond to regions where the FIM degenerates.
  2. Numerical Simulations: They use the derived analytical form of the Fisher information matrix and "averaged learning equations" (ALEs) to simulate the learning process of RBF networks under various conditions. ALEs are ordinary differential equations that describe the average behavior of the network's parameters during training. They conduct experiments with different numbers of hidden units (from two to ten) and in various scenarios designed to trigger overlap and elimination singularities. They also analyze a real-world dataset to assess the impact in a more practical setting.

Datasets

The paper primarily uses artificial datasets for its numerical simulations. For the real-world experiment, they utilize the Extended Complex Scene Saliency Data set (ECSSD) for salient object detection. The ECSSD dataset is described as a benchmark for saliency detection, providing images with associated "conspicuity maps" indicating salient regions.

Key Insights and Ideas

  • Singularities are a major bottleneck: The study strongly suggests that singularities are not just theoretical curiosities but have a tangible and often detrimental impact on the learning process of RBF networks.
  • Elimination singularities are more disruptive: The research highlights that elimination singularities (where a weight becomes zero) have a more significant negative impact on learning dynamics compared to overlap singularities (where two neuron centers coincide).
  • Plateaus are linked to elimination singularities: The paper provides evidence that the frustrating "plateau phenomena" observed during training are primarily caused by elimination singularities.
  • Low-dimensional vs. High-dimensional impact: Overlap singularities are shown to be more influential in low-dimensional RBF networks, while elimination singularities pose a significant challenge across both low and high-dimensional settings.
  • Fisher Information Matrix as a diagnostic tool: The FIM's condition number (or its inverse) serves as a crucial indicator of how close the network is to a singularity, allowing researchers to track the learning process's proximity to these problematic regions.

Key Results

  • Explicit FIM for RBFs: The paper successfully derives the analytical expression for the Fisher information matrix of RBF networks, a foundational step for their analysis.
  • Significant impact of singularities: Numerical simulations reveal that a substantial portion of learning processes are affected by singularities. For RBF networks with two hidden units, approximately 32% of experiments were affected by singularities (7.5% by overlap, 12.5% by elimination, and 12% by output weight becoming zero). In a general case with ten hidden units, about 20.2% of experiments were affected by singularities.
  • Elimination singularities dominate: In high-dimensional scenarios, elimination singularities were found to be the most prevalent cause of learning difficulties, affecting 21% of experiments.
  • Plateau phenomena confirmed: The simulations visually demonstrate that plateaus in the learning curve are strongly associated with elimination singularities.
  • Real-world data validation: Experiments on the ECSSD dataset for salient object detection showed that elimination singularities continue to be a major factor affecting learning in a practical application.

Limitations and Future Work

  • Focus on RBF Networks: While the findings are suggested to be applicable to other feedforward networks, the primary analysis is conducted on RBF networks. Extending this analysis to other architectures, especially deep neural networks, is a natural next step.
  • Theoretical vs. Practical: The paper uses artificial examples extensively to isolate and study specific singularity types. While a real-world dataset is used, further investigation into how these theoretical findings translate to diverse real-world applications would be beneficial.
  • Mitigation Strategies: The paper identifies the problem and its impact but does not propose specific methods to overcome or mitigate the effects of singularities. The authors suggest that their work can serve as a foundation for developing such strategies, particularly for deep neural networks.
  • Future Research Directions: The authors explicitly call for more research into elimination singularities due to their significant impact and the relative lack of theoretical analysis compared to overlap singularities. They also suggest that future research should focus on designing treatments for both traditional and deep neural networks to address singularity issues.

Summary updated

Abstract

The existence of singularities often affects the learning dynamics in feedforward neural networks. In this paper, based on theoretical analysis results, we numerically analyze the learning dynamics of radial basis function (RBF) networks near singularities to understand to what extent singularities influence the learning dynamics. First, we show the explicit expression of the Fisher information matrix for RBF networks. Second, we demonstrate through numerical simulations that the singularities have a significant impact on the learning dynamics of RBF networks. Our results show that overlap singularities mainly have influence on the low dimensional RBF networks and elimination singularities have a more significant impact to the learning processes than overlap singularities in both low and high dimensional RBF networks, whereas the plateau phenomena are mainly caused by the elimination singularities. The results can also be the foundation to investigate the singular learning dynamics in deep feedforward neural networks.
BibTeX
@article{guo2018numerical,
  author = {Weili Guo and Haikun Wei and Yew-Soon Ong and Jaime Rubio Hervas and Junsheng Zhao and Hai Wang and Kanjian Zhang},
  title = {{Numerical Analysis near Singularities in RBF Networks}},
  journal = {J. Mach. Learn. Res.},
  year = {2018},
  url = {https://openreview.net/forum?id=d4VeqR1XUL},
}
Figure