TL;DR
This paper introduces Network Connectivity Gradients (NCG), a novel method for optimizing the connectivity patterns of Restricted Boltzmann Machines (RBMs) by jointly learning network weights and connection structures. NCG enables RBMs to learn faster and achieve better performance in both generative and classification tasks, outperforming traditional dense networks and other pruning methods.
What Problem Does This Paper Address?
Deep neural networks have shown that sparse connectivity can lead to improved performance and efficiency. However, finding optimal sparse connectivity patterns, especially for simpler models like Restricted Boltzmann Machines (RBMs), remains a challenge. Traditional methods often rely on explicit penalization or treat sparsity as a hyperparameter, and network connectivity is often determined after training or in separate pruning phases. This paper addresses the problem of efficiently and jointly learning optimal network connectivity and model parameters for RBMs, aiming to improve their learning performance without altering their fundamental energy-based objective function.
Methods
The core contribution is the Network Connectivity Gradient (NCG) method. NCG introduces two new parameters: a binary connection activation matrix (A) and a continuous connection strength matrix (A').
- Network Gradients: NCG computes gradients for every possible connection, even those currently absent. These gradients drive the continuous connection strength parameter.
- Joint Learning: The connection strength is updated using these gradients, and then thresholded to determine the binary connection pattern at each training epoch. This allows for truly joint learning of network weights and connectivity.
- Continuous Approximation: To handle the discrete nature of connections, NCG uses a continuous strength parameter and a thresholding mechanism, avoiding the need for discrete optimization methods.
- Initialization: The method explores different initialization strategies for the connection activation matrix (p), investigating how initial sparsity affects learning.
- No Objective Function Change: Crucially, NCG does not require modifications to the RBM's standard energy-based objective function or additional regularization terms.
Datasets
- MNIST: Used for both generative (sample generation) and classification tasks. Images were converted to binary.
- Mushrooms: A dataset from the UCI Evaluation Suite, used for classification.
- Connect-4: Another dataset from the UCI Evaluation Suite, used for classification.
Key Insights and Ideas
- Connectivity is as Important as Weights: The research strongly suggests that the network's connectivity pattern is a critical hyperparameter that significantly impacts learning performance, comparable to the number of hidden units.
- Dynamic Connectivity Learning: NCG's ability to dynamically add and remove connections during training is key to its success, allowing it to adapt to different initial sparsity levels.
- Robustness to Initialization: NCG demonstrates robustness to the initial connectivity pattern, achieving similar performance across different initial sparsity levels (p values).
- Decoupled Learning Rates: The ability to use different learning rates for connectivity (α₁) and model parameters (α) is crucial for NCG's effectiveness, allowing connectivity to adapt more quickly.
- Moderate Threshold is Key: A moderate threshold (γ) for activating connections strikes a good balance between sparsity and learnability, with very low or high thresholds leading to suboptimal performance.
Key Results
- Generative Task (MNIST): NCG achieved significantly lower average Negative Log-Likelihood (NLL) compared to a dense RBM, outperforming it by a substantial margin. For example, NCG with p=0.1 achieved an NLL of 103 ± 1, compared to 149 ± 11 for the dense network.
- Classification Task (MNIST, Mushrooms, Connect-4): NCG consistently achieved higher classification accuracy than the dense RBM and other static connectivity patterns. For MNIST, NCG achieved accuracies around 93.5-93.7%, while the dense network was around 93.1%.
- Comparison with SET: NCG outperformed the SET method in both generative and classification tasks, demonstrating superior robustness and learning curves.
- Dynamic Connectivity: NCG was shown to both add and remove connections during training, adapting to initial sparsity levels.
- Learning Speed: NCG generally learned faster, achieving better performance earlier in training compared to baseline methods.
Limitations and Future Work
- Hyperparameter Sensitivity: The performance of NCG is sensitive to hyperparameters like the connectivity learning rate (α₁) and the connectivity threshold (γ). Finding optimal values for these is important.
- RBM Specificity: The current formulation of NCG is specific to RBMs. Extending the core ideas to other neural network architectures (e.g., using backpropagation) is a promising future direction.
- Initialization Strategy: While NCG is robust to initialization, exploring more principled methods for initializing connectivity could further improve performance.
- Computational Cost: While NCG optimizes connectivity, it adds complexity. Further research into computational efficiency might be beneficial.
- CD Approximation: The use of Contrastive Divergence (CD) for gradient approximation can affect results, especially with fewer steps (CD-1). The paper notes that NCG's relative advantage is still present even with CD-1, though diminished.
Summary updated