Does the Neuron "Learn" Like the Synapse?

Raoul TawelNeurIPS 1988

In the late 1980s, the primary bottleneck for artificial neural networks was the computational burden of supervised learning. Algorithms like standard back-propagation (BP) were notoriously slow to converge, especially on high-order, non-linearly separable tasks like the XOR problem.

This inefficiency stems from a fundamental oversimplification: traditional models treat the neuron as a strictly passive, non-linear "wire," placing the entire burden of knowledge retention on the synaptic weights. Drawing inspiration from biological evidence that information processing also occurs at the neuronal level, Raoul Tawel's paper asks a pivotal question: What if we let the neurons learn, too?

To answer this, the paper proposes the Adaptive Neuron Model (ANM), which treats the "temperature" of a neuron's activation function as a learnable parameter rather than a fixed constant, achieving massive reductions in training time.

The Adaptive Neuron Model (ANM)

In a standard feed-forward network, the activation of a neuron is typically governed by a sigmoid function: $$o = \frac{1}{1 + e^{-\beta s}}$$ Usually, $\beta$ (the gain or inverse temperature) is a fixed global constant. In the ANM, every individual neuron $i$ in every layer $m$ possesses its own characteristic local temperature ($T_i^m = 1/\beta_i^m$).

Sigmoid activation function at various temperature levels.

The key innovation is applying gradient descent to the temperature space. During the training phase, the algorithm updates both the synaptic weights and the neuronal temperatures simultaneously. The paper derives the chain-rule mathematics required to calculate the error signal for these temperatures in both the output and hidden layers, effectively giving the network an additional degree of freedom for every neuron.

Because the sigmoid function is extremely sensitive to small changes in temperature, adjusting this parameter allows a neuron to radically change its firing behavior. This enables the network to navigate the error landscape much more efficiently than by adjusting weights alone. Furthermore, in a network with $N$ neurons, there are $O(N^2)$ synaptic weights but only $O(N)$ neuronal temperatures. Adding this relatively small number of parameters yields a disproportionately large gain in learning speed.

Performance and the "Phase Transition" Effect

The model was evaluated using classic logic-gate benchmarks, including the XOR (Exclusive-OR) and Parity problems. The performance gains reported are significant:

  • Convergence Speed: For the XOR problem, the ANM reached a solution in approximately 1,000 iterations. In contrast, plain back-propagation required over 1,000,000 iterations to reach a comparable state.
  • Precision: The ANM achieved a mean square error (MSE) of $10^{-6}$, while standard BP was still at an MSE of $10^{-3}$ after its million-iteration run.
  • Escaping Local Minima: Standard back-propagation frequently gets trapped in local minima. The ANM consistently found global minima, likely because the learnable temperatures provide a way to "reshape" the activation functions to find a path out of these traps.
Error and standard deviation vs. iterations, showing ANM's rapid convergence compared to standard BP.

One of the most striking observations in the paper is that ANM training often experiences a "phase-transition-like" discontinuity. As seen in the training curves, after a period of slow progress (the "shoulder"), the error suddenly drops several orders of magnitude in just a few iterations.

Limitations and Future Work

While the results for XOR and parity are impressive, the paper focuses on small "logic-type" problems. A more detailed analysis on real-world applications (like image recognition) is necessary to prove the model's scalability. Additionally, introducing a second learning rate for temperature adds another layer of complexity to hyperparameter tuning, though the authors suggest the benefits far outweigh the cost. Finally, while inspired by biology, the gradient descent method used for temperature optimization remains a mathematical convenience rather than a strict biological claim.

Summary updated

BibTeX
@inproceedings{tawel1988does,
  author = {Raoul Tawel},
  title = {{Does the Neuron "Learn" Like the Synapse?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1988},
}
Figure