Abstract
This paper introduces a novel theoretical framework for the analysis of vector-valued neural networks through the development of vector-valued variation spaces, a new class of reproducing kernel Banach spaces. These spaces emerge from studying the regularization effect of weight decay in training networks with activations like the rectified linear unit (ReLU). This framework offers a deeper understanding of multi-output networks and their function-space characteristics. A key contribution of this work is the development of a representer theorem for the vector-valued variation spaces. This representer theorem establishes that shallow vector-valued neural networks are the solutions to data-fitting problems over these infinite-dimensional spaces, where the network widths are bounded by the square of the number of training data. This observation reveals that the norm associated with these vector-valued variation spaces encourages the learning of features that are useful for multiple tasks, shedding new light on multi-task learning with neural networks. Finally, this paper develops a connection between weight-decay regularization and the multi-task lasso problem. This connection leads to novel bounds for layer widths in deep networks that depend on the intrinsic dimensions of the training data representations. This insight not only deepens the understanding of the deep network architectural requirements, but also yields a simple convex optimization method for deep neural network compression. The performance of this compression procedure is evaluated on various architectures.
Introduction
While the theoretical foundations of scalar-valued neural networks have matured significantly, our understanding of multi-output (vector-valued) networks has lagged behind. These networks are the workhorses of modern machine learning, powering everything from multi-class classification to complex multi-task learning systems. This paper bridges the gap by introducing vector-valued variation spaces, a class of Reproducing Kernel Banach Spaces (RKBS) that characterize the functions learned by multi-output networks under weight-decay regularization.
Theoretical Framework: Vector-Valued Variation Spaces
The authors develop a framework based on the regularization effects of weight decay in networks using positively homogeneous activations, such as the Rectified Linear Unit (ReLU). A foundational component of this theory is the Neural Balance Theorem (NBT), which posits that weight decay forces a mathematical balance between the magnitudes of a neuron's input and output weights.
The Representer Theorem
A central achievement of this work is the formulation of a Representer Theorem for vector-valued networks. The theorem proves that for any finite dataset of size $N$, there exists an optimal shallow multi-output network whose width is bounded by $N^2$. Crucially, this bound is independent of the input or output dimensions, providing a theoretical explanation for why neural networks remain effective in high-dimensional settings without succumbing to the "curse of dimensionality."
Multi-Task Learning and the "Neuron Sharing" Phenomenon
One of the most significant insights of this research is the mathematical mapping between the weight-decay objective and the multi-task Lasso problem. This connection reveals that weight decay does more than just promote sparsity; it specifically encourages neuron sharing.
In a multi-task environment, the variation norm favors solutions where individual neurons contribute to multiple output tasks simultaneously, rather than having disjoint sets of neurons for different tasks. This provides a principled explanation for the inductive bias that makes multi-task learning successful in deep learning.
Principled Network Compression
The link to the multi-task Lasso problem enables a novel, optimization-based approach to network compression. By viewing a pre-trained layer as a solution to a convex optimization problem, the authors can identify the "sparsest" equivalent layer that produces the same output features.
Key aspects of this method include:
- Data-Dependent Bounds: The required width of a layer is shown to depend on the intrinsic rank (dimension) of the learned feature representations rather than static architectural defaults.
- Convex Optimization: Unlike heuristic pruning or dropout, this method uses convex optimization to find a narrower version of a layer that preserves the exact mathematical optimality of the original trained model.
Experimental Results
The authors validated their compression method on standard architectures using the CIFAR-10 dataset, focusing on the final layers of established models:
- VGG-19: The final hidden layer was compressed from 512 to 99 neurons with virtually no impact on performance (test accuracy shifted from 93.91% to 93.90%).
- AlexNet: The final layer was reduced from 4096 to 555 neurons while maintaining a stable test accuracy of approximately 72.14%.
These results demonstrate that the compressed models maintain nearly identical weight-decay objectives and training losses, proving that the compression preserves the integrity of the learned representations.
Conclusion and Future Work
This paper provides a rigorous functional analysis of multi-output networks, offering a new lens through which to view multi-task learning and architectural efficiency. While the current compression approach is applied layer-by-layer and is restricted to homogeneous activations (ReLU, Leaky ReLU), it lays the groundwork for global optimization strategies that could further streamline deep learning architectures. The established $N^2$ bound and the connection to multi-task Lasso represent a significant step forward in the quest for a unified theory of deep learning.
Summary updated