Introduction
Traditional deep learning architectures typically rely on scaling laws, stacking billions of identical, simple neurons. This "one-size-fits-all" approach contrasts sharply with biological brains, which exhibit significant neuronal diversity—different neurons possess unique morphologies and functions specialized for specific tasks. Recent "NeuroAI" research has sought to bridge this gap by using Symbolic Regression (SR) to discover task-specific aggregation functions. However, these early methods (NS-SR) often relied on Genetic Programming, which is notoriously unstable, slow, and sensitive to initialization.
As shown in the table above, various formulations have been proposed to enhance neuron expressivity. However, a rigorous theoretical foundation and a stable optimization framework were largely missing until now.
Methodology: NeuronSeek via Tensor Decomposition (NS-TD)
The authors propose NeuronSeek via Tensor Decomposition (NS-TD), moving from discrete, tree-based searches to a continuous, differentiable optimization framework. This approach reformulates the search for optimal neuronal structures into a tensor decomposition problem.
The Three-Stream Architecture
NS-TD models candidate neurons through three distinct mathematical streams:
- Polynomial Stream: Captures independent non-linear contributions of individual features.
- Interaction Stream: Utilizes Canonical Polyadic (CP) Decomposition to capture complex feature correlations. By factorizing high-dimensional tensors into rank-1 components, the model avoids the "curse of dimensionality."
- Periodic Stream: Employs trigonometric functions to detect high-frequency patterns in the data.
Differentiable Search and Optimization
The search process is made differentiable using stochastic binary gates and $L_0$ regularization, allowing for the "pruning" of unnecessary terms during training via standard gradient descent. A two-phase optimization strategy is employed: a "warm-up" phase allows tensors to develop meaningful representations before sparsity constraints are enforced.
Theoretical Contributions: Super-Super-Expressivity
A key theoretical breakthrough of this work is the proof of "super-super-expressivity." While previous theories identified "super-expressive" activations capable of approximating any function with a fixed number of neurons, they often required parameter magnitudes to grow infinitely. NS-TD demonstrates that by adjusting the aggregation function, universal approximation can be achieved with both a fixed number and a fixed magnitude of parameters, leading to significantly higher memory efficiency.
Experimental Results
The NS-TD framework was evaluated across synthetic data, 16 standard tabular benchmarks (e.g., California Housing), high-stakes real-world tasks, and computer vision datasets (CIFAR-10/100, SVHN, STL-10).
- Efficiency: NS-TD achieved performance comparable to or better than Kolmogorov-Arnold Networks (KAN) while requiring 10x fewer FLOPs on datasets like California Housing.
- Robustness and Stability: Unlike genetic methods, NS-TD consistently converges to the same optimal formulas across different runs, even in high-dimensional spaces ($d=100$).
- Computer Vision: Integrating task-driven neurons into standard backbones like ResNet18 provided a consistent accuracy boost, improving CIFAR-100 performance by ~2% and STL-10 by nearly 3%.
- Computational Speed: The CP-based decomposition resulted in a 42% reduction in training time compared to Tucker decomposition methods.
Conclusion and Future Work
NeuronSeek-TD represents a significant step toward stable and expressive task-driven neural architectures. By replacing heuristic searches with differentiable tensor optimization, the authors provide a robust method for discovering specialized neuronal functions. While currently constrained by a predefined template of functional forms, future research will explore expanding the library of functions and scaling these architectures to Large Language Models.
Summary updated