Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth

Filipe de Avila Belbute-Peres, J Zico Kolter

Sinusoidal representation networks, such as the widely used SIREN architecture, have revolutionized how we handle "implicit" data. By representing images, audio, and 3D shapes as continuous functions within a neural network, they capture fine details that traditional networks miss. However, these networks have historically been fussy, requiring highly specific, non-standard initialization schemes and arbitrary parameter choices to train effectively.

A recent paper from researchers at CMU simplifies this process, providing a theoretical bridge between network parameters and signal processing. The authors introduce Simple Sinusoidal Networks (SSNs), replacing the complex initialization of SIRENs with standard Kaiming initialization. By analyzing these networks through the lens of Neural Tangent Kernels (NTK), they prove that sinusoidal networks act as adjustable low-pass filters, allowing practitioners to tune the network's "bandwidth" based on the target signal's frequencies.

The Challenge with Implicit Neural Representations

Implicit Neural Representations (INRs) use coordinate-based Multilayer Perceptrons (MLPs) to map an input coordinate (like an $x, y$ pixel location) to an output value (like color). While standard ReLU networks struggle to capture fine details due to "spectral bias," sinusoidal activations excel at modeling high-frequency variations.

Despite their advantages, SIRENs are notoriously brittle. They require a very specific uniform initialization to prevent gradients from vanishing or exploding. Furthermore, they rely on a scaling parameter, $\omega$, which is often chosen via trial and error. Until now, there has been a lack of clear theoretical guidance on how to initialize these networks simply and how to choose $\omega$ for a specific task, whether fitting a high-resolution video or solving a complex physics equation.

Methods: Simplifying Initialization and NTK Analysis

The researchers propose two major shifts to stabilize and simplify sinusoidal networks:

  1. Simple Sinusoidal Networks (SSNs): Instead of the complex, multi-stage uniform initialization used in SIRENs, SSNs use the standard Kaiming (He) normal initialization. The authors prove that this standard approach preserves the same variance properties as the original SIREN scheme but is significantly easier to implement in standard deep learning frameworks.
  2. NTK Analysis and Bandwidth Parametrization: The authors derive the Neural Tangent Kernel (NTK) for these networks, which describes how the network evolves during training. They found that for SSNs, the kernel is approximately Gaussian, whereas for SIRENs, it is an oscillatory sinc function.
SSN Neural Tangent Kernel analysis showing Gaussian approximation and bandwidth tuning via omega.

Crucially, both kernels act as low-pass filters. The parameter $\omega$ directly controls the "bandwidth" of this filter, allowing the network to be tuned to the specific frequency content of the target signal.

Key Insights and Practical Heuristics

The theoretical proof that these networks learn via a low-pass filter explains their strong interpolation capabilities. If $\omega$ is set too high, the network attempts to fit high-frequency noise, leading to overfitting. If set too low, it produces a blurry reconstruction, underfitting the signal.

Reconstruction of a multi-frequency signal showing how omega controls the network's low-pass filter bandwidth.
  • The $1/8$ Heuristic: The authors discovered a practical link to the Nyquist frequency. They found that setting $\omega$ to roughly 1/8th of the signal's maximum frequency provides an excellent starting point for optimization.
  • Multi-dimensional Tuning: For signals like video, where time and space operate on different scales, the authors suggest using a vector-valued $\omega$. This allows the network to maintain different bandwidths for different input dimensions.
  • Shift Invariance: The paper confirms that sinusoidal networks are naturally shift-invariant. The network's behavior doesn't fundamentally change if the input coordinates are shifted, which is a vital property for consistent signal processing.

Performance and Results

The authors evaluated SSNs across a wide variety of signal-fitting and physics-based tasks, including images, audio, video, and Physics-Informed Neural Networks (PINNs):

  • Image Fitting: SSNs matched or outperformed SIRENs in almost every category, achieving a 50.04 PSNR on standard grayscale test images compared to SIREN's 49.0.
  • Generalization: In "checkerboard" tests (training on half the pixels and testing on the other half), properly tuned SSNs showed significantly fewer artifacts than SIRENs.
  • PDE Solving: In physics tasks, SSNs were much more accurate at identifying unknown parameters. For the Navier-Stokes equation, the SSN reduced the error in identifying fluid dynamics parameters by nearly half compared to the baseline.
  • Asymptotic Strength: While SIRENs sometimes converge faster in the very early stages of training, SSNs show much stronger asymptotic performance, reaching higher final accuracy with longer training.

Limitations and Future Directions

While the paper vastly simplifies initialization, it notes that for extremely high-frequency signals (like audio), the first layer might still require a custom learning rate to remain stable. Additionally, the $1/8$ heuristic is a powerful guide but may need adjustment for signals with highly non-uniform frequency distributions.

Future work could involve automatically "learning" the optimal $\omega$ during training or extending this bandwidth-centric analysis to other architectures, such as Transformers that utilize periodic positional encodings.

Summary updated

Abstract

Neural networks with sinusoidal activations have been proposed as an alternative to networks with traditional activation functions. Despite their promise, particularly for learning implicit models, their training behavior is not yet fully understood, leading to a number of empirical design choices that are not well justified. In this work, we first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth. Finally, we utilize these insights to inform the sinusoidal network initialization, optimizing their performance for each of a series of tasks, including learning implicit models and solving differential equations.
BibTeX
@inproceedings{belbuteperes2023simple,
  author = {Filipe de Avila Belbute-Peres and J Zico Kolter},
  title = {{Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=yVqC6gCNf4d},
}
Figure