Abstract
Complex-valued neural networks (CVNNs) have demonstrated significant empirical success in domains ranging from signal processing to MRI reconstruction. However, a theoretical understanding of their generalization capabilities has remained elusive. This paper establishes the first mathematical generalization bounds for CVNNs, demonstrating that their performance on unseen data is fundamentally constrained by their "spectral complexity"—the product of the spectral norms of their weight matrices. Through rigorous derivation and extensive empirical validation across vision and sequential datasets, we show that this spectral metric serves as a highly accurate proxy for test performance, independent of the total number of parameters.
1. Introduction
In contemporary deep learning, real-valued weights and activations are the standard. However, Complex-Valued Neural Networks (CVNNs), which utilize complex numbers to represent both amplitude and phase, offer unique advantages. They are biologically plausible—where amplitude can represent firing rates and phase can represent spike timing—and computationally efficient for tasks involving wave-like data or unitary transformations.
Despite their success in specialized fields, CVNNs have lacked the theoretical guarantees that characterize their real-valued counterparts. Specifically, there have been no established bounds to explain how well a CVNN trained on a finite sample will generalize to new, unseen data. This research fills this gap by providing the first rigorous generalization bounds for CVNNs in both independent and identically distributed (i.i.d.) and sequential data settings.
2. Theoretical Framework
The authors approach the problem through the lens of statistical learning theory. The goal is to bound the Rademacher complexity, a measure of a function class's ability to fit random noise, which in turn provides an upper bound on the generalization error.
2.1 Mathematical Tools
To derive these bounds in the complex domain, the research adapts several sophisticated mathematical techniques:
- Generalized Maurey Sparsification Lemma: Used to create matrix coverings within the complex plane.
- Dudley Entropy Integral: Employed to bound the Rademacher complexity by integrating the square root of the log-covering numbers.
2.2 The Spectral Complexity Metric
The core finding is that generalization is governed by spectral complexity. This metric is primarily defined by the product of the spectral norms (the largest singular values) of the network's complex-valued weight matrices, modulated by the Lipschitz constants of the activation functions (such as modReLU or complex hyperbolic tangents). Notably, these bounds do not explicitly depend on the number of layers or the total parameter count, suggesting that the "smoothness" of the transformation is more critical for generalization than the sheer size of the model.
3. Generalization Bounds
The paper presents two landmark theorems:
- Theorem 1 (i.i.d. Data): Establishes an upper bound on generalization error for static data, such as images.
- Theorem 2 (Sequential Data): Extends this logic to sequential data, proving that the spectral complexity remains the dominant factor in time-series or text processing.
Both theorems mathematically prove that the excess risk—the gap between training error and test error—scales with the spectral complexity of the network.
4. Empirical Validation
To verify the theoretical derivations, the authors conducted experiments using complex-valued Convolutional Neural Networks (CV-CNNs) trained via Stochastic Gradient Descent (SGD).
4.1 Datasets and Methodology
The theory was tested across a diverse range of benchmarks:
- Image Classification (i.i.d.): MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet.
- Text Classification (Sequential): The IMDB movie reviews dataset.
During training, the researchers tracked the product of the spectral norms of the weights and compared it against the excess risk. Since the networks were trained to near-zero training error, the excess risk effectively mirrored the test error.
4.2 Results
The experimental results showed a near-perfect correlation between the theoretical bound and actual performance. The Spearman’s rank-order correlation coefficients (SCC) were remarkably high:
- 0.99 for CIFAR-10, MNIST, FashionMNIST, Tiny ImageNet, and IMDB.
- 0.80 for CIFAR-100.
The associated $p$-values were effectively zero ($p < 10^{-23}$), providing overwhelming evidence that spectral complexity is a reliable predictor of generalization in CVNNs.
5. Discussion and Conclusion
While spectral norm-based bounds are often numerically "loose" (the absolute value of the bound may exceed the actual error), this research demonstrates that the trend of the bound perfectly tracks the trend of the error. This makes the spectral norm product a highly useful metric for monitoring and potentially regularizing CVNNs during training.
By providing the first foundational theoretical step into the generalization properties of complex-valued architectures, this work paves the way for more robust and better-understood CVNNs. Future research may build upon these bounds to develop new regularization techniques specifically tailored for the complex plane, further enhancing the performance of these powerful networks in signal and image processing.
Summary updated