Spectral Complexity-scaled Generalization Bound of Complex-valued Neural Networks

Haowen Chen, Fengxiang He, Shiye Lei, Dacheng TaoarXiv 2021 · arXiv

Abstract

Complex-valued neural networks (CVNNs) have demonstrated significant empirical success in domains ranging from signal processing to MRI reconstruction. However, a theoretical understanding of their generalization capabilities has remained elusive. This paper establishes the first mathematical generalization bounds for CVNNs, demonstrating that their performance on unseen data is fundamentally constrained by their "spectral complexity"—the product of the spectral norms of their weight matrices. Through rigorous derivation and extensive empirical validation across vision and sequential datasets, we show that this spectral metric serves as a highly accurate proxy for test performance, independent of the total number of parameters.

1. Introduction

In contemporary deep learning, real-valued weights and activations are the standard. However, Complex-Valued Neural Networks (CVNNs), which utilize complex numbers to represent both amplitude and phase, offer unique advantages. They are biologically plausible—where amplitude can represent firing rates and phase can represent spike timing—and computationally efficient for tasks involving wave-like data or unitary transformations.

Despite their success in specialized fields, CVNNs have lacked the theoretical guarantees that characterize their real-valued counterparts. Specifically, there have been no established bounds to explain how well a CVNN trained on a finite sample will generalize to new, unseen data. This research fills this gap by providing the first rigorous generalization bounds for CVNNs in both independent and identically distributed (i.i.d.) and sequential data settings.

2. Theoretical Framework

The authors approach the problem through the lens of statistical learning theory. The goal is to bound the Rademacher complexity, a measure of a function class's ability to fit random noise, which in turn provides an upper bound on the generalization error.

2.1 Mathematical Tools

To derive these bounds in the complex domain, the research adapts several sophisticated mathematical techniques:

  • Generalized Maurey Sparsification Lemma: Used to create matrix coverings within the complex plane.
  • Dudley Entropy Integral: Employed to bound the Rademacher complexity by integrating the square root of the log-covering numbers.

2.2 The Spectral Complexity Metric

The core finding is that generalization is governed by spectral complexity. This metric is primarily defined by the product of the spectral norms (the largest singular values) of the network's complex-valued weight matrices, modulated by the Lipschitz constants of the activation functions (such as modReLU or complex hyperbolic tangents). Notably, these bounds do not explicitly depend on the number of layers or the total parameter count, suggesting that the "smoothness" of the transformation is more critical for generalization than the sheer size of the model.

3. Generalization Bounds

The paper presents two landmark theorems:

  1. Theorem 1 (i.i.d. Data): Establishes an upper bound on generalization error for static data, such as images.
  2. Theorem 2 (Sequential Data): Extends this logic to sequential data, proving that the spectral complexity remains the dominant factor in time-series or text processing.

Both theorems mathematically prove that the excess risk—the gap between training error and test error—scales with the spectral complexity of the network.

4. Empirical Validation

To verify the theoretical derivations, the authors conducted experiments using complex-valued Convolutional Neural Networks (CV-CNNs) trained via Stochastic Gradient Descent (SGD).

4.1 Datasets and Methodology

The theory was tested across a diverse range of benchmarks:

  • Image Classification (i.i.d.): MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet.
  • Text Classification (Sequential): The IMDB movie reviews dataset.

During training, the researchers tracked the product of the spectral norms of the weights and compared it against the excess risk. Since the networks were trained to near-zero training error, the excess risk effectively mirrored the test error.

4.2 Results

The experimental results showed a near-perfect correlation between the theoretical bound and actual performance. The Spearman’s rank-order correlation coefficients (SCC) were remarkably high:

  • 0.99 for CIFAR-10, MNIST, FashionMNIST, Tiny ImageNet, and IMDB.
  • 0.80 for CIFAR-100.

The associated $p$-values were effectively zero ($p < 10^{-23}$), providing overwhelming evidence that spectral complexity is a reliable predictor of generalization in CVNNs.

5. Discussion and Conclusion

While spectral norm-based bounds are often numerically "loose" (the absolute value of the bound may exceed the actual error), this research demonstrates that the trend of the bound perfectly tracks the trend of the error. This makes the spectral norm product a highly useful metric for monitoring and potentially regularizing CVNNs during training.

By providing the first foundational theoretical step into the generalization properties of complex-valued architectures, this work paves the way for more robust and better-understood CVNNs. Future research may build upon these bounds to develop new regularization techniques specifically tailored for the complex plane, further enhancing the performance of these powerful networks in signal and image processing.

Summary updated

Abstract

Complex-valued neural networks (CVNNs) have been widely applied to various fields, especially signal processing and image recognition. However, few works focus on the generalization of CVNNs, albeit it is vital to ensure the performance of CVNNs on unseen data. This paper is the first work that proves a generalization bound for the complex-valued neural network. The bound scales with the spectral complexity, the dominant factor of which is the spectral norm product of weight matrices. Further, our work provides a generalization bound for CVNNs when training data is sequential, which is also affected by the spectral complexity. Theoretically, these bounds are derived via Maurey Sparsification Lemma and Dudley Entropy Integral. Empirically, we conduct experiments by training complex-valued convolutional neural networks on different datasets: MNIST, FashionMNIST, CIFAR-10, CIFAR-100, Tiny ImageNet, and IMDB. Spearman's rank-order correlation coefficients and the corresponding p values on these datasets give strong proof that the spectral complexity of the network, measured by the weight matrices spectral norm product, has a statistically significant correlation with the generalization ability.
BibTeX
@misc{chen2021spectral,
  author = {Haowen Chen and Fengxiang He and Shiye Lei and Dacheng Tao},
  title = {{Spectral Complexity-scaled Generalization Bound of Complex-valued Neural Networks}},
  year = {2021},
  eprint = {2112.03467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.03467},
}
Figure