Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network

Lixin Fan

Despite the massive success of deep learning, many of its core components are still treated as "black boxes." Techniques like Batch Normalization (BN) are essential for training deep networks, yet they are often described as mysterious tricks that work by reducing internal covariate shift. Similarly, the Rectified Linear Unit (ReLU) is the industry standard activation function, but its theoretical connection to the underlying logic of neural computing remains somewhat opaque.

This paper seeks to bridge the gap between modern deep learning and the foundational logic of Fuzzy Neural Networks (FNNs), which were popular in the 1990s. By introducing a theoretically justified framework based on the Generalized Hamming Distance (GHD), the authors demystify modern techniques and offer a more interpretable, efficient way to build neural architectures: the Generalized Hamming Network (GHN).

The Generalized Hamming Distance (GHD)

The core of the paper relies on the Generalized Hamming Distance, defined for real numbers $x$ and $w$ as: $$h(x, w) := x + w - 2xw$$

Visualizations of the Generalized Hamming Distance and its transformations.

The authors demonstrate that a standard neuron's operation ($w \cdot x + b$) can be transformed into a measurement of the negative GHD if the bias term $b$ is set analytically to: $$b = -\frac{1}{2} \left( \sum w_l + \sum x_l \right)$$

This mathematical reframing leads to several key methodological innovations:

  • Rectified Bias: Instead of letting the optimizer "learn" the bias through backpropagation, it is calculated directly based on the weights and inputs.
  • Double-Thresholding: The authors re-interpret ReLU as setting a minimal Hamming distance threshold. They improve upon this by introducing a "double-thresholding" scheme that suppresses "fuzzy" elements (values near 0.5) which often carry noisy gradients, while keeping both positive and negative confident regions.
  • Generalized Hamming Network (GHN): A novel network architecture where neuron outputs are strictly defined by the GHD between inputs and weights.

Key Insights: Demystifying Modern Tricks

Batch Normalization is an Approximation

The most striking claim of the paper is that Batch Normalization is essentially a brute-force optimization attempt to reach the "rightful bias" that GHD provides analytically.

Comparison of mean outputs over epochs for networks with Batch Normalization (BN), the proposed analytical bias (XOR), and without BN (WO_BN).

When the GHD bias is enforced, the network's mean outputs remain stable, effectively rendering the sophisticated and computationally expensive Batch Normalization layer unnecessary.

ReLU is Non-Essential for Simple Tasks

For relatively simple tasks like MNIST digit classification, the authors found that non-linear activations like ReLU can be removed entirely without losing accuracy, provided the GHD-based bias is used.

Neural Computing as Fuzzy Logic

The GHD induces a fuzzy XOR logic. This means a deep network can be viewed as a series of stacked logical statements (e.g., "is input $x$ equivalent to weight $w$?"), making the neural "black box" significantly more transparent and interpretable.

Performance and Results

The authors evaluated the GHN across several standard benchmarks, demonstrating that theoretical elegance does not come at the cost of performance:

  • Speed and Robustness: On MNIST, the GHN reached >95% accuracy within just 1,000 mini-batches, performing on par with BN-equipped networks but with fewer parameters.
  • Competitive Accuracy: On CIFAR-10, the GHN achieved 89.3% accuracy, and 60.1% on CIFAR-100. These results are highly competitive with more sophisticated, hand-tuned modern networks.
  • Rapid Convergence: In CIFAR experiments, the double-thresholding scheme reduced the number of iterations required to reach target accuracy by 1 to 2 orders of magnitude by suppressing uncertain "fuzzy regions."
  • Generative Modeling: When applied to a Variational Autoencoder (VAE), the GHD rectification led to a 28% reduction in reconstruction error compared to the baseline model.

Limitations and Future Work

While the GHN shows immense promise, the authors acknowledge a few limitations. On small datasets, such as the Movie Review benchmark, the GHN improved learning speed but quickly hit 100% training accuracy and began to overfit, failing to improve final test accuracy. Additionally, while GHD helped the VAE converge faster, the double-thresholding scheme proved less relevant for generative settings compared to classification tasks.

Ultimately, this paper lays a strong foundational theory. Future work will likely focus on applying these GHD principles to even more complex, state-of-the-art architectures to see if the gains in efficiency, convergence speed, and interpretability hold at massive scales.

Summary updated

Abstract

We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the “normalized” bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double-thresholding on both positive and negative extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.
BibTeX
@inproceedings{fan2017revisit,
  author = {Lixin Fan},
  title = {{Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=B1bnmP-OWH},
}
Figure