Abstract
Neural networks have been shown to improve performance across a range of natural-language tasks. However, designing and training them can be complicated. Frequently, researchers resort to repeated experimentation to pick optimal settings. In this paper, we address the issue of choosing the correct number of units in hidden layers. We introduce a method for automatically adjusting network size by pruning out hidden units through $\ell_{\infty,1}$ and $\ell_{2,1}$ regularization. We apply this method to language modeling and demonstrate its ability to correctly choose the number of hidden units while maintaining perplexity. We also include these models in a machine translation decoder and show that these smaller neural models maintain the significant improvements of their unpruned versions.
Introduction
In deep learning, architecture design is often more of an art than a science. A primary challenge is determining the optimal number of units in a hidden layer. Insufficient capacity prevents learning, while excessive capacity leads to overfitting and wasted computational resources. Traditionally, researchers rely on grid search—running numerous experiments with different configurations—which is computationally expensive and slow. This work proposes an "auto-sizing" approach where the network determines its own optimal size during training.
Methodology
The proposed method starts with an intentionally oversized network and prunes it automatically using Group Sparsity Regularization. Unlike standard $\ell_1$ regularization which sparsifies individual weights, group sparsity targets entire hidden units. By treating the weights of a single unit as a group, the network can drive all incoming weights of a unit to zero simultaneously, effectively deactivating it.
The authors utilize $\ell_{2,1}$ and $\ell_{\infty,1}$ mixed norms. The $\ell_{\infty,1}$ norm, for instance, penalizes the maximum weight in a group; if the maximum reaches zero, the entire group is eliminated. Because these regularizers are non-differentiable at zero, the authors employ a proximal gradient method. Specifically, they introduce a novel linear-time algorithm for the $\ell_\infty$ proximal operator to "snap" weight groups to zero.
Evaluation Setup
The method was evaluated on two major corpora:
- Europarl v7 (English): ~56 million tokens of European Parliament proceedings.
- English Gigaword 5 (AFP): ~870 million tokens of news articles.
The approach was tested on $n$-gram neural language models and integrated into a statistical machine translation decoder.
Key Insights and Results
- Robustness: A single regularization strength ($\lambda = 0.1$) proved effective across different datasets and vocabulary sizes, reducing the need for hyperparameter tuning.
- Convergence: Models initialized with different sizes (e.g., 900 vs. 1000 units) tended to prune down to a similar final capacity, suggesting the method identifies an inherent task-specific optimal size.
- Simultaneous Pruning and Fitting: Training curves indicate that the model fits data while pruning units in parallel, allowing remaining units to adapt to the reduced architecture in real-time.
- Performance: On the Europarl dataset, the first hidden layer was reduced from 1,000 to 499 units (~50% reduction) with negligible perplexity increase (103 to 105). In machine translation, auto-sized models maintained BLEU score gains (up to +2.0) while significantly reducing memory and latency.
Conclusion
This study demonstrates that group sparsity regularization is a viable path for automating neural network architecture design. While focused on feedforward $n$-gram models, the principles are general. Future research should explore applying these techniques to recurrent architectures like LSTMs or Transformers to determine if similar efficiency gains can be achieved in sequence-to-sequence modeling.
Summary updated