No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks

Feng-Lei Fan, Meng Wang, Hang-Cheng Dong, Jianwei Ma, Tieyong ZengarXiv 2024 · DOI · arXiv

Introduction

In the last decade, deep learning innovation has focused heavily on architectures—the complex ways in which neurons are interconnected, such as in ResNets or Transformers. However, the fundamental unit of these networks, the neuron, has remained largely static: a simple linear dot product followed by a non-linear activation function. This paper challenges this "one-size-fits-all" approach by proposing a framework to design task-based neurons. By introducing neuronal diversity, the authors aim to move beyond homogeneous architectures toward specialized designs that mirror the functional diversity found in biological brains.

Methodology: Discovering Task-Based Neurons

The authors propose a two-step framework to prototype custom neurons tailored to specific datasets.

1. Vectorized Symbolic Regression (VSR)

Traditional symbolic regression (SR) searches for mathematical formulas to fit data but often produces overly complex equations and is computationally expensive. The authors introduce Vectorized Symbolic Regression (VSR), which regularizes the search by requiring all input variables to undergo the same symbolic transformation. This constraint significantly accelerates the search process, facilitates parallel computation, and acts as a powerful regularizer to prevent overfitting.

2. Parameterization

Once VSR identifies a promising "skeleton" formula, it is converted into a learnable neuron. Fixed constants within the formula are replaced with trainable weights (e.g., $\alpha, \beta, \gamma$), allowing the neuron to fine-tune its behavior during standard backpropagation. This parameterized formula serves as a new "aggregation function," replacing the traditional linear dot product.

The two-stage process of discovering a symbolic expression through VSR and converting it into a trainable neuron model via parameterization.

Experimental Evaluation

The researchers evaluated Task-based Networks (TNs) against standard Linear Networks (LNs) across several domains:

  • Synthetic Data: 10 multivariate polynomials were used to verify if VSR could recover underlying mathematical patterns from noisy signals (up to 30% noise).
  • Public Benchmarks: 20 diverse tabular datasets for regression and classification, including California Housing, Wine Quality, and Credit Risk.
  • Real-World Applications: High-energy particle collision data from CERN and NASA/CalTech asteroid diameter prediction.

Key Results and Discussion

The experiments revealed that optimizing the neuron type provides a new dimension for performance gains that architecture tweaks alone cannot unlock:

  • Superior Accuracy and Efficiency: TNs consistently outperformed standard LNs. Notably, TNs often achieved better results while using fewer parameters, as specialized neurons can naturally represent complex relationships (e.g., quadratic or trigonometric) that linear neurons require multiple layers to approximate.
  • State-of-the-Art Performance: On the complex Asteroid Prediction task, the TN achieved an MSE of 0.0513, significantly outperforming specialized models like TabNet (0.0627) and FT-Transformer (0.2136).
  • Inductive Bias: By allowing neurons to utilize functions like $x^3$ or $\cos(x)$, the network starts with a stronger inductive bias regarding the underlying physics or logic of the task.

Conclusion and Future Work

Task-based neurons represent a shift toward more efficient and interpretable AI. While VSR is faster than traditional SR, future work will focus on full GPU acceleration and automating the selection of base function libraries. The ultimate goal is to combine task-based neurons with Neural Architecture Search (NAS) to create fully automated pipelines for designing optimal, specialized networks for any given problem.

Summary updated

Abstract

In the past decade, many successful networks are on novel architectures, which almost exclusively use the same type of neurons. Recently, more and more deep learning studies have been inspired by the idea of NeuroAI and the neuronal diversity observed in human brains, leading to the proposal of novel artificial neuron designs. Designing well-performing neurons represents a new dimension relative to designing well-performing neural architectures. Biologically, the brain does not rely on a single type of neuron that universally functions in all aspects. Instead, in our brain, neurons are often task-based. In this study, we address the following question: since the human brain is a task-based neuron user, can the artificial network design go from the task-based architecture design to the task-based neuron design? Since methodologically there are no one-size-fits-all neurons, given the same structure, task-based neurons can enhance the feature representation ability relative to the existing universal neurons due to the intrinsic inductive bias for the task. Specifically, we propose a two-step framework for prototyping task-based neurons. As the initial step, we evaluate the proposed framework using polynomials as base functions. Empirically, systematic experimental results on synthetic data, classic benchmarks, and real-world applications show that the proposed task-based neuron design is not only feasible but also delivers competitive performance over other state-of-the-art models.
BibTeX
@misc{fan2024no,
  author = {Feng-Lei Fan and Meng Wang and Hang-Cheng Dong and Jianwei Ma and Tieyong Zeng},
  title = {{No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks}},
  year = {2024},
  eprint = {2405.02369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.02369},
}
Figure