Automatically Gating Multi-Frequency Patterns through Rectified Continuous Bernoulli Units with Theoretical Principles

Zheng-Fan Wu, Yi-Nan Feng, Hui XueIJCAI 2022 · OpenReview

Abstract

Different nonlinearities are only suitable for responding to different frequency signals. The locally-responding ReLU is incapable of modeling high-frequency features due to the spectral bias, whereas the globally-responding sinusoidal function is intractable to represent low-frequency concepts cheaply owing to the optimization dilemma. Moreover, nearly all the practical tasks are composed of complex multi-frequency patterns, whereas there is little prospect of designing or searching a heterogeneous network containing various types of neurons matching the frequencies, because of their exponentially-increasing combinatorial states. In this paper, our contributions are three-fold: 1) we propose a general Rectified Continuous Bernoulli (ReCB) unit paired with an efficient variational Bayesian learning paradigm, to automatically detect/gate/represent different frequency responses; 2) our numerically-tight theoretical framework proves that ReCB-based networks can achieve the optimal representation ability, which is $O(m^{\eta/(d^2)})$ times better than that of popular neural networks, for a hidden dimension of $m$, an input dimension of $d$, and a Lipschitz constant of $\eta$; 3) we provide comprehensive empirical evidence showing that ReCB-based networks can keenly learn multi-frequency patterns and push the state-of-the-art performance.

Introduction: The Frequency Dilemma

Natural signals, such as images, are composed of multi-frequency patterns ranging from smooth gradients (low frequency) to sharp edges and textures (high frequency). Standard neural network architectures struggle with this diversity due to the inherent limitations of their activation functions:

  1. ReLU and Spectral Bias: Locally-responding activations like ReLU are easy to optimize but suffer from spectral bias, making them inefficient at learning high-frequency details.
  2. Sinusoidal and Optimization Dilemmas: Periodic activations (e.g., SIREN) can represent high frequencies but create highly non-convex loss landscapes with many local minima, making them difficult to train and inefficient for low-frequency components.

Methods: Rectified Continuous Bernoulli (ReCB) Units

The authors introduce the Multi-Frequency Gated Unit, which allows each neuron to dynamically choose between a ReLU-like and a Sinusoidal response. This choice is managed by a gate $z$ governed by a Rectified Continuous Bernoulli (ReCB) distribution.

Key features of the ReCB unit include:

  • Trainable Gating: Unlike discrete binary switches, the ReCB distribution is differentiable, allowing the network to learn the optimal frequency response for each neuron via standard backpropagation and a Variational Bayesian paradigm.
  • Optimization Rectification: The gating mechanism effectively "rectifies" the solution space. By allowing the network to initially rely on stable local responses, it guides the global sinusoidal components toward the correct frequencies, avoiding the typical optimization pitfalls of periodic functions.
Fitting and optimization comparison: (a-c) show fitting results for ReLU, Sin, and ReCB; (d-f) show the corresponding loss landscapes. ReCB successfully fits the signal by navigating a smoother optimization path.

Results and Discussion

The proposed ReCB units were evaluated on synthetic multi-frequency functions and standard benchmarks like CIFAR-10.

Theoretical Advantage

The authors prove that ReCB-based networks achieve an approximation bound $O(m^{\eta/(d^2)})$ times better than pure ReLU networks, where $m$ is the hidden dimension and $d$ is the input dimension. This suggests that "smarter" neurons can be more effective than simply increasing network depth.

Empirical Performance

  • Efficiency: A ReCB-based ResNet-20 achieved a top-1 error of 6.52% on CIFAR-10, outperforming the much deeper ResNet-110 (6.61%).
  • Robustness: ReCB models maintain high accuracy across a wide range of input frequencies, whereas standard models degrade quickly as signal complexity increases.
  • Automatic Specialization: During training, gates naturally converge toward 0 or 1, showing that the network autonomously partitions tasks between high-frequency and low-frequency specialists.

Conclusion

By bridging the gap between local and global activation functions, ReCB units provide a robust and automated solution for modeling multi-frequency data. This approach significantly improves both representation power and optimization stability, paving the way for more efficient and flexible neural architectures.

Summary updated

Abstract

Different nonlinearities are only suitable for responding to different frequency signals. The locally-responding ReLU is incapable of modeling high-frequency features due to the spectral bias, whereas the globally-responding sinusoidal function is intractable to represent low-frequency concepts cheaply owing to the optimization dilemma. Moreover, nearly all the practical tasks are composed of complex multi-frequency patterns, whereas there is little prospect of designing or searching a heterogeneous network containing various types of neurons matching the frequencies, because of their exponentially-increasing combinatorial states. In this paper, our contributions are three-fold: 1) we propose a general Rectified Continuous Bernoulli (ReCB) unit paired with an efficient variational Bayesian learning paradigm, to automatically detect/gate/represent different frequency responses; 2) our numerically-tight theoretical framework proves that ReCB-based networks can achieve the optimal representation ability, which is O(m^{η/(d^2)}) times better than that of popular neural networks, for a hidden dimension of m, an input dimension of d, and a Lipschitz constant of η; 3) we provide comprehensive empirical evidence showing that ReCB-based networks can keenly learn multi-frequency patterns and push the state-of-the-art performance.
BibTeX
@inproceedings{wu2022automatically,
  author = {Zheng-Fan Wu and Yi-Nan Feng and Hui Xue},
  title = {{Automatically Gating Multi-Frequency Patterns through Rectified Continuous Bernoulli Units with Theoretical Principles}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=5OsWkVKyWB},
}
Figure