Towards constructive and destructive dynamic network configuration

Stefan Wermter, Manuela MeurerESANN 1996 · OpenReview

Abstract

Designing optimal neural network architectures traditionally involves significant manual trial and error. This paper introduces a hybrid dynamic configuration algorithm that combines constructive and destructive phases to automatically determine the ideal number of hidden units for recurrent neural networks (RNNs). By utilizing a sliding average of the error rate to mitigate the inherent noise in RNN learning curves, the algorithm grows the network to meet performance goals and subsequently prunes redundant units to enhance generalization. Experimental results on a semantic context classification task demonstrate that this dynamic approach yields networks that are smaller, faster to train, and more accurate on unseen data than those designed through manual static configuration.

Introduction

In the mid-1990s, determining the optimal number of hidden units in a neural network was a persistent challenge. An insufficient number of units leads to underfitting, where the model lacks the capacity to learn the task, while an excessive number leads to overfitting and increased computational costs. While constructive (growth-based) and destructive (pruning-based) methods existed independently, their combined application—particularly for recurrent neural networks (RNNs)—remained under-explored. RNNs present unique difficulties because their error rates often fluctuate non-monotonically, making it difficult to determine the appropriate timing for architectural adjustments.

Methods

The proposed 3-Phase Dynamic Configuration algorithm addresses these challenges through a structured approach:

  1. Preprocessing: To account for the erratic learning curves of RNNs, the algorithm employs a sliding average ($E_{sa}$) of the error over a specific window. This smoothing allows for a more reliable assessment of the learning trend.
  2. Phase 1: Constructive Learning: Training begins with a single hidden unit. If the learning slope flattens before reaching the performance target, a new unit is added, and training continues.
  3. Phase 2: Destructive Learning: Once the performance goal is met, the algorithm attempts to simplify the architecture by removing hidden units sequentially. If the error does not increase significantly, the unit is permanently removed, forcing the network to retain only essential connections.
  4. Phase 3: Post-learning: A final fine-tuning phase optimizes the weights of the streamlined architecture.

Experimental Results

The algorithm was evaluated using a Semantic Context Classification task involving a corpus of 795 phrases categorized into 10 semantic classes (e.g., law, music, computer science). The dataset was split into 277 training phrases and 518 test phrases.

The dynamic approach significantly outperformed the best manually tuned static networks. As shown in the results below, the dynamic network achieved a more compact architecture with superior generalization capabilities.

Comparison of performance metrics between the best manually designed static network and the dynamically configured network.

The dynamic network utilized 25% fewer hidden units (9 vs. 12) and reduced training time by approximately 14% (257:07 vs. 299:17). Crucially, it achieved a test accuracy of 81%, surpassing the 79% achieved by the static network, indicating that the pruning phase effectively served as a form of regularization.

Discussion and Conclusion

The success of the dynamic configuration highlights several key insights. First, the use of a smoothed error window is essential for managing the "internal loops" of RNNs that cause non-monotonic learning. Second, starting with a small network and growing it is computationally more efficient than starting with a large network and shrinking it. Finally, the destructive phase does more than just improve speed; it forces the model to learn more robust, general features, leading to better performance on novel data.

Future work should address the sensitivity of the algorithm to the thresholds used for adding and removing units. Additionally, while effective for the specific RNN architectures and language tasks studied, further research is needed to determine if these principles scale to modern architectures or different domains such as computer vision.

Summary updated

BibTeX
@inproceedings{wermter1996towards,
  author = {Stefan Wermter and Manuela Meurer},
  title = {{Towards constructive and destructive dynamic network configuration}},
  booktitle = {ESANN},
  year = {1996},
  url = {https://openreview.net/forum?id=dMbTfBkux1},
}
Figure