Abstract
This paper investigates the role of connectivity in the expressive efficiency of neural networks. By leveraging the framework of tensor decompositions, we demonstrate that interconnecting dilated convolutional networks (DCNs) with distinct dilation patterns significantly enhances their representational power. We introduce "mixed tensor decompositions" to model these interconnected architectures and prove that they can represent complex functions with an almost quadratic reduction in channel requirements compared to standard DCNs. Empirical results on phoneme classification tasks confirm that these connectivity patterns improve accuracy without increasing parameter counts.
Introduction
In deep learning theory, "expressive efficiency" typically focuses on network depth—the idea that deep networks can represent functions that shallow ones cannot without an exponential increase in neurons. However, modern architectures like ResNet and Inception rely heavily on complex connectivity patterns, such as skip connections and branching. This research addresses whether such connectivity provides a formal boost in expressive power or is merely a training heuristic. The study specifically examines dilated convolutional networks (DCNs), which are fundamental to sequence models like WaveNet.
Methods: Bridging Networks and Tensors
The authors bridge deep learning and multi-linear algebra by treating the input-output mapping of a network as a high-dimensional "grid tensor." The core innovation lies in the relationship between network architecture and tensor decomposition structures.
Dilated Convolutional Networks
Standard DCNs use a hierarchy of convolutional layers with increasing dilation factors to capture long-range dependencies. Figure 1 illustrates the architecture of a size-2 dilated convolutional network, where the receptive field grows exponentially with depth.
Mode Trees and Mixed Decompositions
A key theoretical insight is that a specific dilation pattern corresponds to a "mode tree"—a hierarchical structure dictating how tensor dimensions are combined. By mixing different dilation patterns, the authors create "mixed tensor decompositions."
In neural network terms, this mixing involves running two dilated networks in parallel and swapping or sharing features between their intermediate layers. To make the mathematical proofs tractable, the authors utilize "convolutional arithmetic circuits," where non-linearities are products rather than ReLUs, allowing for rigorous tensor rank analysis.
Key Insights and Results
Connectivity as a Multiplier
Interconnecting two networks does not merely result in the sum of their parts. Instead, it generates an ensemble of "hybrid trees"—computational paths that exist in neither original network. The authors prove an "almost quadratic" gap in efficiency: a mixed network with $r$ channels can represent functions that would require nearly $r^2$ channels in a standard architecture. This is the difference between a practical model and one that exceeds memory limits.
Empirical Validation
The theoretical findings were validated using the TIMIT Acoustic-Phonetic Continuous Speech Corpus for phoneme classification. The results showed:
- Consistent Accuracy Gains: Increasing the number of interconnections between parallel dilated networks led to steady improvements in classification accuracy.
- Zero-Cost Improvement: These gains were achieved without increasing the total number of parameters or computational complexity, simply by optimizing the connectivity of existing capacity.
Conclusion and Future Work
This work demonstrates that connectivity is a fundamental driver of expressive efficiency. While the formal proofs (such as Corollary 8) rely on arithmetic circuits, empirical evidence suggests these principles hold for standard ReLU-based networks. Future research aims to extend this "mixed decomposition" framework to other connectivity schemes like DenseNet, moving toward a more principled mathematical approach to architecture design.
Summary updated