Modeling Hebb Learning Rule for Unsupervised Learning

Jia Liu, Maoguo Gong, Qiguang Miao

Abstract

Unsupervised feature learning typically relies on reconstruction-based objectives, such as those used in Autoencoders and Restricted Boltzmann Machines. This paper proposes the Neuron Learning Machine (NLM), an alternative framework derived from the biological Hebb learning rule. By optimizing a Hebbian objective function with an added stability constraint, the NLM learns abstract, non-redundant representations without requiring a decoder. We evaluate the NLM on MNIST, CIFAR-10, and remote sensing datasets, demonstrating that it consistently outperforms traditional models in classification accuracy and feature quality.

Introduction

The quest for effective unsupervised feature learning has largely been dominated by "task-based" paradigms. Models like Autoencoders (AE) assume that the best features are those that allow for the perfect reconstruction of input data. However, defining these tasks can be computationally expensive and conceptually ambiguous. In contrast, biological systems learn through local synaptic changes. The NLM aims to bridge this gap by implementing the Hebbian principle—"cells that fire together, wire together"—as a direct optimization objective for machine learning, focusing purely on the quality of the learned representation.

The Neuron Learning Machine (NLM)

Unlike traditional architectures, the NLM consists solely of an encoder. The learning process is driven by a mathematically derived objective function that mirrors biological processes.

Modeling the Hebb Rule

The fundamental Hebb rule states that the change in weight ($\Delta W_{ij}$) between two neurons is proportional to the product of their activities: $\Delta W_{ij} = \alpha h_i v_j$. The authors treat this gradient as the derivative of an objective function, leading to a term that maximizes the correlation between input and weights: $v^T Wh$.

Stability and Abstraction

To prevent weights from growing unbounded and to ensure the model learns salient, abstract features rather than simply copying the input, a correlation-based constraint is introduced: $-\lambda \|Wh\|_2^2$. This term mimics biological lateral inhibition, forcing neurons to focus on distinct, non-redundant components of the data. The resulting objective function is:

$$J(W) = v^T Wh - \lambda \|Wh\|_2^2$$

This formulation is also shown to be mathematically equivalent to increasing the lower bound on the mutual information between the input $v$ and the learned features $h$.

Feature Quality and Visualization

A key advantage of the NLM is the quality of the features it extracts. When trained on complex datasets, including satellite imagery and MNIST, the model naturally develops structures reminiscent of biological vision systems.

Comparison of learned features across different models and datasets. NLM demonstrates the ability to learn clear, Gabor-like edge detectors compared to the noisier filters of AE and RBM.

As seen in the visualizations, NLM filters for remote sensing data (Yellow River and Wuhan) exhibit clear edge-detection properties, which are fundamental for robust computer vision. This suggests that the Hebbian objective effectively captures the underlying structure of the data.

Experimental Results

The NLM was evaluated against standard unsupervised models across several benchmarks, showing superior performance in both shallow and deep configurations.

MNIST and CIFAR-10 Performance

On the MNIST dataset, a single-layer NLM achieved a test error rate of 5.27% with 10,000 training samples, significantly outperforming AEs (8.88%) and RBMs (8.42%). When expanded into deeper or convolutional architectures, the performance gains remained consistent.

Classification error rates (%) for stacked and convolutional versions of various models on MNIST and CIFAR-10.

As shown in the table above, the stacked NLM reached an error rate of 1.53% on MNIST, outperforming stacked AEs and RBMs. Furthermore, the Convolutional NLM (CNLM) achieved a 21.1% error rate on CIFAR-10, surpassing the Convolutional Autoencoder (21.8%). The model also demonstrated high robustness to its primary hyperparameter, $\lambda$, maintaining stable performance across a wide range of values (0.5 to 1.9).

Conclusion

The Neuron Learning Machine demonstrates that biologically inspired learning rules can serve as powerful alternatives to reconstruction-based unsupervised learning. By focusing purely on the representation through a Hebbian objective, the NLM achieves superior feature quality and classification accuracy with a simpler, encoder-only architecture. Future research will explore more complex biological rules and the application of NLM as a pre-training step for supervised deep learning.

Summary updated

Abstract

This paper presents to model the Hebb learning rule and proposes a neuron learning machine (NLM). Hebb learning rule describes the plasticity of the connection between presynaptic and postsynaptic neurons and it is unsupervised itself. It formulates the updating gradient of the connecting weight in artificial neural networks. In this paper, we construct an objective function via modeling the Hebb rule. We make a hypothesis to simplify the model and introduce a correlation based constraint according to the hypothesis and stability of solutions. By analysis from the perspectives of maintaining abstract information and increasing the energy based probability of observed data, we find that this biologically inspired model has the capability of learning useful features. NLM can also be stacked to learn hierarchical features and reformulated into convolutional version to extract features from 2-dimensional data. Experiments on single-layer and deep networks demonstrate the effectiveness of NLM in unsupervised feature learning.
BibTeX
@inproceedings{liu2017modeling,
  author = {Jia Liu and Maoguo Gong and Qiguang Miao},
  title = {{Modeling Hebb Learning Rule for Unsupervised Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/322},
}
Figure