Sinusoidal Neural Representations for High Frequency Signals

157 concepts119 papers1 proposal

Standard neural networks are notoriously "blurry." When used to represent signals like images or 3D shapes, architectures based on Rectified Linear Units (ReLU) exhibit a strong spectral bias, favoring low-frequency components and smoothing over sharp edges and fine textures. As Dupont et al. observe, this bias makes it difficult for small networks to maintain high fidelity when overfitting to complex visual data.

To overcome this, researchers have turned to periodic activation functions—most notably the sine wave. By replacing the piecewise-linear ReLU with a periodic nonlinearity, coordinate-based networks can capture high-frequency spatial details and sharp gradients that are typically lost.

The Intuition of Periodic Activations

In a coordinate-based neural representation, a network maps a continuous coordinate (such as $(x, y)$ for a pixel or $(x, y, z)$ for a 3D point) to a value (such as color or distance to a surface).

When using ReLU activations, the network approximates the target signal as a composition of linear planes. While this works for simple shapes, it struggles with oscillatory patterns or intricate details like the pores on a face or the wrinkles in cloth. Jung et al. note that periodic activations enable multilayer perceptrons to model these intricate surfaces and sharp features more effectively than standard units.

The power of the sine wave lies in its derivatives. The derivative of a sine wave is a cosine wave—another periodic function. This property ensures that the network's gradients remain informative and non-zero across the entire state space. Bansal et al. highlight that these high-fidelity gradients are essential for solving partial differential equations (PDEs) and deriving optimal control commands, where the "slope" of the learned function is just as important as the value itself.

Capturing the High-Frequency Spectrum

The shift to periodic activations directly addresses the spectral bias problem. Chan et al. demonstrate that utilizing periodic activations within generative models allows for the synthesis of high-quality, 3D-consistent images with fine-grained textures. Similarly, in medical imaging, Wu et al. and Shen et al. show that sinusoidal functions allow networks to recover sharp anatomical edges and pathological features from limited or sparsely sampled data.

This efficiency is not just about quality, but also about capacity. Lucey provides theoretical evidence that neural fields using periodic activations are significantly more parameter-efficient than ReLU-based architectures, allowing complex signals to be represented with fewer neurons. This makes them ideal for tasks like image compression, where Dupont et al. show that storing the weights of a tiny sinusoidal network can outperform traditional formats like JPEG at low bit-rates.

Stability in Physics and Geometry

Beyond visual fidelity, periodic activations are uniquely suited for tasks governed by physical laws. Many physical processes depend on spatial or temporal derivatives. For instance, Lifschitz et al. explain that because refractive distortion in water is proportional to the slope of the surface, sine-based networks provide the stable gradients necessary to reconstruct 3D water shapes from 2D images.

In 3D reconstruction, researchers often represent shapes using Signed Distance Fields (SDFs), which must satisfy the Eikonal constraint (the gradient magnitude must be 1). Vasilopoulos et al. argue that sinusoidal activations satisfy this constraint more effectively than standard MLPs, producing smoother and more accurate gradients for robot motion planning. This mathematical stability extends to complex physical simulations; Zhou et al. utilize sinusoidal activations to precisely represent the higher-order functional derivatives required for energy-conserving Hamiltonian mechanics.

Evolution Toward Adaptive Activations

While fixed sine waves are powerful, recent work has explored making these periodic functions more flexible. Liu et al. propose Trainable Matrix Activation Functions, which learn the shape of the nonlinearity during training to better capture oscillatory signals. Other researchers, such as Berzins et al. and Romero, have integrated Gabor wavelets or functions as basis elements, combining the frequency-localization of periodic functions with the spatial-localization of wavelets to capture even more complex, non-smooth signals.

By moving away from the "flat" world of ReLU, these sinusoidal representations allow neural networks to finally see—and represent—the fine-grained, high-frequency details of the physical world.

Key Foundational Work

Pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020) — contributed 2 concept elements to this direction.
Implicit Neural Representations with Periodic Activation Functions (2020) — contributed 2 concept elements to this direction.
Adaptive activation functions accelerate convergence in deep and physics-informed neural networks (2020) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 122 papers
  2. Version 1 · 2026-10-04 · 256 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_periodic_activation_functions_sinusoidal_2c6f12ee,
  title = {periodic activation functions (sinusoidal representation networks)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/periodic-activation-functions-sinusoidal-representation-networks/rdr_2c6f12ee390b43d5b7891463d30c4060},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_periodic_activation_functions_sinusoidal_2c6f12ee_v2,
  title = {periodic activation functions (sinusoidal representation networks)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/periodic-activation-functions-sinusoidal-representation-networks/rdr_2c6f12ee390b43d5b7891463d30c4060/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake