Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection

Jing-Xiao Liao, Bojian Hou, Hangcheng Dong, Hao Zhang, Siwei Ma, Jinwei Sun, Shiping Zhang, Fenglei FanarXiv 2022 · DOI · arXiv

Abstract

Inspired by the complexity and diversity of biological neurons, a quadratic neuron is proposed to replace the inner product in the current neuron with a simplified quadratic function. Employing such a novel type of neurons offers a new perspective on developing deep learning. When analyzing quadratic neurons, we find that there exists a function such that a heterogeneous network can approximate it well with a polynomial number of neurons but a purely conventional or quadratic network needs an exponential number of neurons to achieve the same level of error. Encouraged by this inspiring theoretical result on heterogeneous networks, we directly integrate conventional and quadratic neurons in an autoencoder to make a new type of heterogeneous autoencoders. To our best knowledge, it is the first heterogeneous autoencoder that is made of different types of neurons. Next, we apply the proposed heterogeneous autoencoder to unsupervised anomaly detection for tabular data and bearing fault signals. The anomaly detection faces difficulties such as data unknownness, anomaly feature heterogeneity, and feature unnoticeability, which is suitable for the proposed heterogeneous autoencoder. Its high feature representation ability can characterize a variety of anomaly data (heterogeneity), discriminate the anomaly from the normal (unnoticeability), and accurately learn the distribution of normal samples (unknownness). Experiments show that heterogeneous autoencoders perform competitively compared to other state-of-the-art models.

Introduction

Most deep learning innovations focus on high-level architecture—adding more layers or clever shortcuts like ResNets. However, almost all these networks rely on the same fundamental building block: the conventional neuron, which calculates a simple inner product of inputs and weights followed by a nonlinear activation. This "one-size-fits-all" approach limits representational efficiency, especially in Unsupervised Anomaly Detection (AD). Anomalies are often "unnoticeable" (hidden in high-dimensional space), "heterogeneous" (appearing in many different forms), and "unknown" (labels are unavailable during training). A standard autoencoder (AE) might struggle to learn a sufficiently complex distribution of "normal" data to effectively spot these subtle deviations.

The Quadratic Neuron and Heterogeneous Architectures

To address these limitations, we propose a Quadratic Neuron that incorporates power terms and interactions, allowing it to represent nonlinear features like circular or parabolic boundaries much more efficiently than a stack of linear neurons. Mathematically, while a standard neuron is $f(x) = \sigma(w^\top x + b)$, the quadratic neuron is formulated as: $$\sigma((x^\top w^r + b^r)(x^\top w^g + b^g) + (x \odot x)^\top w^b + c)$$

We introduce three primary ways to integrate these neurons into an autoencoder framework:

  • HAE-X: Parallel branches of conventional and quadratic neurons that are fused together.
  • HAE-Y: Uses both types in the encoder but relies on quadratic neurons in the decoder for enhanced information extraction.
  • HAE-I: Interlaces conventional and quadratic layers in a symmetric "sandwich" structure.
Proposed Heterogeneous Autoencoder (HAE) architectures: HAE-X, HAE-Y, and HAE-I.

Stable Training with ReLinear

Quadratic terms can introduce training instability, as small input changes can lead to massive output swings. To mitigate this, we employ the ReLinear strategy. We initialize the quadratic parameters to zero, causing the neuron to act like a conventional one at the start of training. Furthermore, we apply a significantly smaller learning rate to the quadratic components compared to the linear ones to ensure stable convergence.

The ReLinear training strategy applies different learning rates to stabilize quadratic neuron training.

Theoretical Insights and Experimental Results

A key theoretical contribution of this work is the proof of an "Efficiency Gap." We demonstrate that certain functions can be approximated by a heterogeneous network using a polynomial number of neurons, whereas a purely conventional or purely quadratic network would require an exponential number of neurons to achieve the same error level. This synergy allows HAEs to characterize a variety of anomaly data that standard AEs might over-smooth.

Empirically, HAEs were evaluated on 15 tabular benchmarks and industrial bearing fault datasets (XJTU-SY and SBP).

  • Tabular Data: HAE variants (particularly HAE-X) consistently achieved the highest average AUC and Precision, outperforming state-of-the-art methods like DeepSVDD and DAGMM.
  • Industrial Signals: On the XJTU-SY bearing dataset, the HAE-I model achieved an AUC of 0.961, a notable improvement over the standard AE's 0.942.
  • Latent Space: t-SNE visualizations reveal that HAEs create tighter, more distinct clusters for normal data, making anomalies significantly more detectable.

Conclusion

By mixing quadratic and conventional neurons, the Heterogeneous Autoencoder offers a more expressive "vocabulary" for describing normal data distributions. While challenges remain in extending the theory to deeper networks and managing parameter overhead, the HAE demonstrates that neuron-level diversity is a powerful tool for unsupervised anomaly detection.

Summary updated

Abstract

Inspired by the complexity and diversity of biological neurons, a quadratic neuron is proposed to replace the inner product in the current neuron with a simplified quadratic function. Employing such a novel type of neurons offers a new perspective on developing deep learning. When analyzing quadratic neurons, we find that there exists a function such that a heterogeneous network can approximate it well with a polynomial number of neurons but a purely conventional or quadratic network needs an exponential number of neurons to achieve the same level of error. Encouraged by this inspiring theoretical result on heterogeneous networks, we directly integrate conventional and quadratic neurons in an autoencoder to make a new type of heterogeneous autoencoders. To our best knowledge, it is the first heterogeneous autoencoder that is made of different types of neurons. Next, we apply the proposed heterogeneous autoencoder to unsupervised anomaly detection for tabular data and bearing fault signals. The anomaly detection faces difficulties such as data unknownness, anomaly feature heterogeneity, and feature unnoticeability, which is suitable for the proposed heterogeneous autoencoder. Its high feature representation ability can characterize a variety of anomaly data (heterogeneity), discriminate the anomaly from the normal (unnoticeability), and accurately learn the distribution of normal samples (unknownness). Experiments show that heterogeneous autoencoders perform competitively compared to other state-of-the-art models.
BibTeX
@misc{liao2022quadratic,
  author = {Jing-Xiao Liao and Bojian Hou and Hangcheng Dong and Hao Zhang and Siwei Ma and Jinwei Sun and Shiping Zhang and Fenglei Fan},
  title = {{Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection}},
  year = {2022},
  eprint = {2204.01707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.01707},
}
Figure