Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network

Lixin Fan

Abstract

We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the normalized bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double thresholding on both extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.

Introduction

Deep learning's success is often tempered by its "black box" nature. Heuristic techniques like Batch Normalization (BN) and Rectified Linear Units (ReLU) are indispensable for training deep architectures, yet their fundamental mechanisms remain poorly understood. Conversely, Fuzzy Neural Networks (FNNs) offer a rigorous mathematical foundation but have historically struggled with scalability. This research bridges this gap by introducing the Generalized Hamming Network (GHN), re-interpreting modern deep learning components through the lens of fuzzy logic.

The Generalized Hamming Distance

The core of the GHN is the Generalized Hamming Distance (GHD). For real-valued inputs $x$ and weights $w$, the GHD is defined as:

$$h(x, w) := x + w - 2xw$$

This operation functions as a "fuzzy XOR." By re-evaluating the standard neuron output $w \cdot x + b$, the author identifies a "rectified bias" $b = -0.5(\sum w_i + \sum x_i)$. When this bias is applied, the neuron's output corresponds to the negative GHD.

Visualization of the GHD and its activation functions.

To refine signal propagation, the GHN employs double-thresholding. Unlike standard ReLU, which only thresholds at zero, this scheme suppresses values near the identity element (0.5) where gradients are typically noisy, effectively "cleaning" the signal during backpropagation.

Demystifying Batch Normalization and ReLU

A key insight of this work is that Batch Normalization is not merely a tool for reducing internal covariate shift, but a practical approximation of the GHD's analytical bias. By enforcing the "rectified bias" directly, the need for BN and its associated computational overhead is eliminated.

Comparison of output stability between BN and GHN.

Furthermore, ReLU is re-interpreted as a distance thresholding mechanism. This perspective suggests that ReLU sets a minimum Hamming distance between inputs and weights, a concept that is generalized and improved upon by the GHN's double-thresholding activation.

Experimental Evaluation

The GHN was tested across several benchmarks:

  • MNIST: Achieved 99.0% accuracy with a four-layer GHN, matching the learning speed of BN-equipped networks without optimizing BN parameters.
  • CIFAR-10/100: Reached 89.3% and 60.1% accuracy, respectively. Double-thresholding improved learning efficiency by 1-2 orders of magnitude.
  • Generative Modeling: A GHD-rectified Variational Autoencoder (VAE) reduced reconstruction error by 28% over the baseline.
  • Robustness: GHNs demonstrated stability across learning rates from 0.01 to 0.1, indicating reduced sensitivity to hyperparameter tuning.

Conclusion

The Generalized Hamming Network provides a logical, distance-based framework that demystifies essential deep learning "tricks." By aligning neural computing with fuzzy logic, the GHN achieves state-of-the-art performance while offering better interpretability and faster learning. Future work will focus on a full formulation of deep networks as chains of fuzzy logic inferences.

Summary updated

Abstract

We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the normalized bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double thresholding on both extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.
BibTeX
@inproceedings{fan2017revisit,
  author = {Lixin Fan},
  title = {{Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=B1bnmP-OWH},
}

Loading related content...

Figure