Doctor of Crosswise: Reducing Over-parametrization in Neural Networks

J. de Curtò, I. de Zarzà, Kris Kitani, Irwin King, Michael R. Lyu

Abstract

Dr. of Crosswise proposes a new architecture to reduce over-parametrization in Neural Networks. It introduces an operand for rapid computation in the framework of Deep Learning that leverages learned weights. The formalism is described in detail providing both an accurate elucidation of the mechanics and the theoretical implications.

Introduction

Modern deep learning models are characterized by massive over-parametrization. A standard fully connected layer with $n$ inputs and $n$ outputs requires $n^2$ weights, leading to significant memory overhead and computational bottlenecks. "Doctor of Crosswise" (Dr. of Crosswise) introduces a novel architecture designed to mitigate these issues by replacing standard dense matrix-vector multiplications with a specialized diagonal operation termed c&z. By constraining weight matrices to a diagonal form, the parameter count per layer is reduced from $O(n^2)$ to $O(n)$, potentially accelerating both training and inference.

The Challenge of Over-parametrization

The current paradigm of large-scale computation relies heavily on dense matrix-vector products ($Wx$). As networks scale in depth and width, the quadratic growth of parameters increases the risk of overfitting and necessitates specialized hardware. Dr. of Crosswise seeks a "fast variant" of deep learning that maintains non-linear expressive power (e.g., via ReLU) while simplifying the underlying arithmetic. This approach draws inspiration from Kernel Methods to streamline neural network mechanics.

The Doctor of Crosswise Architecture

The c&z Operand

The core innovation is the c&z operand ($\{c, x\}_{c\&z}$). In a conventional layer, the transformation is defined as $y = \sigma(Wx + b)$. Dr. of Crosswise substitutes the dense matrix $W$ with a diagonal matrix $\hat{W}$. This allows the operation to be factorized into a component-wise product between the input vector $x$ and a vector $c$, representing the non-zero diagonal elements of $\hat{W}$.

Figure 1: The fundamental c&z operation substituting dense matrix multiplication with a diagonalized vector product.

This transformation is mathematically analogous to a Hadamard product, significantly reducing the number of arithmetic operations required during the forward pass.

Scaling and Dimensionality

Standard neural layers often project data into higher-dimensional spaces (e.g., expanding 4 features to 8). To accommodate this, Dr. of Crosswise utilizes a set of diagonal matrices. By leveraging the Khatri-Rao product—a column-wise Kronecker product—the architecture can expand inputs into higher dimensions while maintaining the $O(n)$ parameter efficiency.

Figure 2: Multi-block diagonal architecture for dimensionality expansion.

For instance, a transformation from 4 inputs to 8 outputs that typically requires 32 weights can be achieved with only 8 learned parameters using this multi-block diagonal approach.

Theoretical Foundations

The Dr. of Crosswise framework is grounded in the "McKernel" construction and the principle that Fourier transforms can diagonalize integral operators like convolutions. By treating the network as a sequence of diagonalized operations, the authors aim to approximate the behavior of complex kernels through simplified arithmetic. The paper formalizes the $\{c, x\}_{c\&z}$ operator and establishes the mathematical properties necessary for implementation via Kronecker products.

Discussion and Future Directions

While the theoretical gains in parameter efficiency are momentous, several challenges remain. The primary limitation is the current lack of empirical validation on standard benchmarks like ImageNet or GLUE. Furthermore, existing hardware (GPUs) is highly optimized for dense BLAS operations; realizing the theoretical speedups of diagonal and Khatri-Rao products may require custom low-level kernel development. Finally, the trade-off between the efficiency of diagonal matrices and the expressivity of dense matrices must be explored to determine if increased network depth can compensate for the reduced width of individual layers.

Summary updated

Abstract

Dr. of Crosswise proposes a new architecture to reduce over-parametrization in Neural Networks. It introduces an operand for rapid computation in the framework of Deep Learning that leverages learned weights. The formalism is described in detail providing both an accurate elucidation of the mechanics and the theoretical implications.
BibTeX
@misc{curto2019doctor,
  author = {J. de Curtò and I. de Zarzà and Kris Kitani and Irwin King and Michael R. Lyu},
  title = {{Doctor of Crosswise: Reducing Over-parametrization in Neural Networks}},
  year = {2019},
  eprint = {1905.10324},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.10324},
}
Figure