Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions

Nadav Cohen, Ronen Tamari, Amnon Shashua

Abstract

This paper investigates the role of connectivity in the expressive efficiency of neural networks. By leveraging the framework of tensor decompositions, we demonstrate that interconnecting dilated convolutional networks (DCNs) with distinct dilation patterns significantly enhances their representational power. We introduce "mixed tensor decompositions" to model these interconnected architectures and prove that they can represent complex functions with an almost quadratic reduction in channel requirements compared to standard DCNs. Empirical results on phoneme classification tasks confirm that these connectivity patterns improve accuracy without increasing parameter counts.

Introduction

In deep learning theory, "expressive efficiency" typically focuses on network depth—the idea that deep networks can represent functions that shallow ones cannot without an exponential increase in neurons. However, modern architectures like ResNet and Inception rely heavily on complex connectivity patterns, such as skip connections and branching. This research addresses whether such connectivity provides a formal boost in expressive power or is merely a training heuristic. The study specifically examines dilated convolutional networks (DCNs), which are fundamental to sequence models like WaveNet.

Methods: Bridging Networks and Tensors

The authors bridge deep learning and multi-linear algebra by treating the input-output mapping of a network as a high-dimensional "grid tensor." The core innovation lies in the relationship between network architecture and tensor decomposition structures.

Dilated Convolutional Networks

Standard DCNs use a hierarchy of convolutional layers with increasing dilation factors to capture long-range dependencies. Figure 1 illustrates the architecture of a size-2 dilated convolutional network, where the receptive field grows exponentially with depth.

Architecture of a dilated convolutional network showing the connectivity between input, hidden layers, and output across time steps.

Mode Trees and Mixed Decompositions

A key theoretical insight is that a specific dilation pattern corresponds to a "mode tree"—a hierarchical structure dictating how tensor dimensions are combined. By mixing different dilation patterns, the authors create "mixed tensor decompositions."

Comparison of two different dilation patterns (a and b) and their corresponding mode trees, illustrating how connectivity dictates the hierarchical structure of the tensor decomposition.

In neural network terms, this mixing involves running two dilated networks in parallel and swapping or sharing features between their intermediate layers. To make the mathematical proofs tractable, the authors utilize "convolutional arithmetic circuits," where non-linearities are products rather than ReLUs, allowing for rigorous tensor rank analysis.

Key Insights and Results

Connectivity as a Multiplier

Interconnecting two networks does not merely result in the sum of their parts. Instead, it generates an ensemble of "hybrid trees"—computational paths that exist in neither original network. The authors prove an "almost quadratic" gap in efficiency: a mixed network with $r$ channels can represent functions that would require nearly $r^2$ channels in a standard architecture. This is the difference between a practical model and one that exceeds memory limits.

Empirical Validation

The theoretical findings were validated using the TIMIT Acoustic-Phonetic Continuous Speech Corpus for phoneme classification. The results showed:

  • Consistent Accuracy Gains: Increasing the number of interconnections between parallel dilated networks led to steady improvements in classification accuracy.
  • Zero-Cost Improvement: These gains were achieved without increasing the total number of parameters or computational complexity, simply by optimizing the connectivity of existing capacity.

Conclusion and Future Work

This work demonstrates that connectivity is a fundamental driver of expressive efficiency. While the formal proofs (such as Corollary 8) rely on arithmetic circuits, empirical evidence suggests these principles hold for standard ReLU-based networks. Future research aims to extend this "mixed decomposition" framework to other connectivity schemes like DenseNet, moving toward a more principled mathematical approach to architecture design.

Summary updated

Abstract

The driving force behind deep networks is their ability to compactly represent rich classes of functions. The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to realize (or approximate) functions of another. To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones. In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways. We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks. By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency. In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not. Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy. This leads us to believe that expressive efficiency may serve a key role in the development of new tools for deep network design.
BibTeX
@inproceedings{cohen2018boosting,
  author = {Nadav Cohen and Ronen Tamari and Amnon Shashua},
  title = {{Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=S1JHhv6TW},
}
Figure