TL;DR
This paper introduces quadratic neurons as a more expressive alternative to conventional neurons in deep learning. To address their notorious training instability, the authors propose "ReLinear," a novel initialization and regularization strategy that significantly improves performance and stability, making quadratic networks a competitive and potentially more efficient choice for certain tasks.
What Problem Does This Paper Address?
Deep neural networks have achieved remarkable success, but their building blocks, conventional neurons, are fundamentally limited. Biological neural systems exhibit far greater diversity and complexity. While quadratic neurons offer a step towards this diversity by replacing the inner product with a quadratic function, leading to enhanced expressivity (the ability to represent a wider range of functions), they suffer from severe training instability. This instability, characterized by oscillations and potential collapse, has hindered their practical adoption. This paper aims to bridge this gap by theoretically demonstrating the superior expressivity of quadratic networks and introducing a practical solution to their training challenges.
Methods
The authors tackle the problem from two main angles: theoretical expressivity and practical trainability.
Theoretical Expressivity: * Spline Theory: They leverage spline theory to show that quadratic networks can represent piecewise polynomial functions, which are inherently more accurate approximations than the piecewise linear functions produced by conventional networks. * Algebraic Geometry: Using the dimension of algebraic varieties as a measure of expressivity, they prove that quadratic networks have a higher dimension than conventional networks with quadratic activation, indicating a richer function space.
Trainability (ReLinear Strategy): * Initialization: Quadratic neurons are initialized to behave like conventional neurons (effectively setting quadratic terms to zero) to start from a stable point. * Regularization: Quadratic terms are then gradually introduced and controlled during training through two main approaches: * Shrinking Gradients (ReLinear$^{sg}$): Regularizing the gradients of quadratic weights. * Shrinking Weights (ReLinear$^{sw}$): Directly regularizing the quadratic weights. * Weight Transfer: They also explore initializing quadratic networks by transferring weights from pre-trained conventional networks. * ReLinear + ReZero: For residual networks, they combine ReLinear with the ReZero technique for further stabilization.
Datasets
The authors evaluate their methods on several standard datasets:
- Runge Function Approximation: Used for analyzing the behavior of different training strategies on a univariate function known for causing approximation issues.
- CIFAR10: A common image classification benchmark.
- ImageNet: A large-scale image classification dataset.
- KITTI: Used for depth estimation tasks.
- CIFAR100: Used for comparing training stability.
Key Insights and Ideas
- Quadratic Neurons Offer a Richer Function Space: The theoretical analysis strongly suggests that quadratic networks can represent a broader and more complex set of functions than their conventional counterparts, even those with quadratic activations.
- Training Instability is a Major Hurdle: The paper vividly illustrates how small changes in initialization can lead to drastic output variations in untrained quadratic networks, explaining their notorious training difficulties.
- ReLinear is Crucial for Unlocking Potential: The proposed ReLinear strategy is not just a minor tweak; it's presented as essential for making quadratic networks trainable and performant. It effectively bridges the gap between theoretical expressivity and practical utility.
- Shrinking Gradients Outperforms Shrinking Weights: The experiments indicate that regularizing the gradients of quadratic terms (ReLinear$^{sg}$) is more effective than directly shrinking the weights (ReLinear$^{sw}$) for achieving better performance and stability.
- Quadratic Networks Can Be More Parameter-Efficient: In some cases, quadratic networks achieve better performance with fewer parameters than deeper conventional networks, suggesting a potential for more efficient architectures.
Key Results
- Expressivity Theorems: The paper provides theoretical proofs demonstrating the superior expressivity of quadratic networks over conventional ones and conventional networks with quadratic activation.
- ReLinear Performance: The ReLinear strategy significantly stabilizes the training of quadratic networks, leading to performance gains. For instance, on the Runge function approximation, ReLinear$^{sg}$ achieved the lowest RMSE.
- Image Classification: On CIFAR10, QResNet models trained with ReLinear consistently outperformed their ResNet counterparts. For example, QResNet32 achieved an error of 6.38% compared to ResNet110's 6.61%, with fewer parameters.
- Depth Estimation: On the KITTI dataset, Quadratic-LapDepth showed a notable reduction in error compared to the baseline LapDepth, demonstrating the benefit of quadratic neurons in this domain.
- Training Stability: Quadratic networks trained with ReLinear showed significantly less oscillation and better convergence compared to untrained or conventionally trained quadratic networks. VGG16 with quadratic activation without ReLinear failed to converge.
Limitations and Future Work
- Generalizability of ReLinear: While ReLinear shows promise, its effectiveness across all possible quadratic network architectures and tasks might require further investigation.
- Parameter Complexity: Quadratic neurons inherently have more parameters than conventional neurons. While the paper shows potential for efficiency, this remains a consideration.
- Task Suitability: The authors suggest that quadratic networks are particularly suited for tasks where quadratic or nonlinear features are significant. Identifying and characterizing these specific tasks is an area for future work.
- Scaling Up: The paper mentions upscaling quadratic networks to solve more real-world problems and characterizing their robustness and generalizability as future research directions.
Summary updated