Abstract
Unsupervised feature learning typically relies on reconstruction-based objectives, such as those used in Autoencoders and Restricted Boltzmann Machines. This paper proposes the Neuron Learning Machine (NLM), an alternative framework derived from the biological Hebb learning rule. By optimizing a Hebbian objective function with an added stability constraint, the NLM learns abstract, non-redundant representations without requiring a decoder. We evaluate the NLM on MNIST, CIFAR-10, and remote sensing datasets, demonstrating that it consistently outperforms traditional models in classification accuracy and feature quality.
Introduction
The quest for effective unsupervised feature learning has largely been dominated by "task-based" paradigms. Models like Autoencoders (AE) assume that the best features are those that allow for the perfect reconstruction of input data. However, defining these tasks can be computationally expensive and conceptually ambiguous. In contrast, biological systems learn through local synaptic changes. The NLM aims to bridge this gap by implementing the Hebbian principle—"cells that fire together, wire together"—as a direct optimization objective for machine learning, focusing purely on the quality of the learned representation.
The Neuron Learning Machine (NLM)
Unlike traditional architectures, the NLM consists solely of an encoder. The learning process is driven by a mathematically derived objective function that mirrors biological processes.
Modeling the Hebb Rule
The fundamental Hebb rule states that the change in weight ($\Delta W_{ij}$) between two neurons is proportional to the product of their activities: $\Delta W_{ij} = \alpha h_i v_j$. The authors treat this gradient as the derivative of an objective function, leading to a term that maximizes the correlation between input and weights: $v^T Wh$.
Stability and Abstraction
To prevent weights from growing unbounded and to ensure the model learns salient, abstract features rather than simply copying the input, a correlation-based constraint is introduced: $-\lambda \|Wh\|_2^2$. This term mimics biological lateral inhibition, forcing neurons to focus on distinct, non-redundant components of the data. The resulting objective function is:
$$J(W) = v^T Wh - \lambda \|Wh\|_2^2$$
This formulation is also shown to be mathematically equivalent to increasing the lower bound on the mutual information between the input $v$ and the learned features $h$.
Feature Quality and Visualization
A key advantage of the NLM is the quality of the features it extracts. When trained on complex datasets, including satellite imagery and MNIST, the model naturally develops structures reminiscent of biological vision systems.
As seen in the visualizations, NLM filters for remote sensing data (Yellow River and Wuhan) exhibit clear edge-detection properties, which are fundamental for robust computer vision. This suggests that the Hebbian objective effectively captures the underlying structure of the data.
Experimental Results
The NLM was evaluated against standard unsupervised models across several benchmarks, showing superior performance in both shallow and deep configurations.
MNIST and CIFAR-10 Performance
On the MNIST dataset, a single-layer NLM achieved a test error rate of 5.27% with 10,000 training samples, significantly outperforming AEs (8.88%) and RBMs (8.42%). When expanded into deeper or convolutional architectures, the performance gains remained consistent.
As shown in the table above, the stacked NLM reached an error rate of 1.53% on MNIST, outperforming stacked AEs and RBMs. Furthermore, the Convolutional NLM (CNLM) achieved a 21.1% error rate on CIFAR-10, surpassing the Convolutional Autoencoder (21.8%). The model also demonstrated high robustness to its primary hyperparameter, $\lambda$, maintaining stable performance across a wide range of values (0.5 to 1.9).
Conclusion
The Neuron Learning Machine demonstrates that biologically inspired learning rules can serve as powerful alternatives to reconstruction-based unsupervised learning. By focusing purely on the representation through a Hebbian objective, the NLM achieves superior feature quality and classification accuracy with a simpler, encoder-only architecture. Future research will explore more complex biological rules and the application of NLM as a pre-training step for supervised deep learning.
Summary updated