BandRC: Band Shifted Raised Cosine Activated Implicit Neural Representations

Pandula Thennakoon, Avishka Ranasinghe, Mario De Silva, Buwaneka Epakanda, Roshan Godaliyadda, Parakrama Ekanayake, Vijitha R. HeratharXiv 2025 · DOI · arXiv

TL;DR

This paper introduces COSMO-RC, a novel activation function for Implicit Neural Representations (INRs) that addresses spectral bias and improves signal representation by incorporating complex sinusoidal modulation. COSMO-RC significantly outperforms existing methods across various computer vision tasks, including image reconstruction, denoising, and super-resolution, by achieving better spectral support and robustness.

What Problem Does This Paper Address?

Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing continuous signals, offering advantages over traditional discrete methods, especially for high-dimensional data. However, current INR models face several challenges:

  • Spectral Bias: INRs often struggle to capture high-frequency signal content, leading to a phenomenon known as spectral bias.
  • Limited Robustness: They can be sensitive to noise.
  • Difficulty in Capturing Local and Global Features: Jointly representing both fine details and overall structure can be challenging.
  • Poorly Understood Activation Functions: The theoretical underpinnings of why certain activation functions perform better in INRs are not well understood.

This paper aims to address these limitations by proposing a new activation function that provides better spectral support and improves overall signal representation quality.

Methods

The core of this work lies in the theoretical analysis and novel design of an activation function for INRs.

  • Harmonic Distortion Analysis and Chebyshev Polynomials: The authors leverage harmonic distortion analysis and Chebyshev polynomial approximations to understand the spectral behavior of activation functions. They mathematically prove that certain symmetric activation functions (both odd and even) lead to attenuation in their post-activation spectrum, causing a loss of signal information.
  • Complex Sinusoidal Modulation (COSMO): To mitigate this spectral attenuation, they propose modulating activation functions with a complex sinusoidal term. This technique is shown to preserve both odd and even frequency coefficients, ensuring complete spectral support.
  • COSMO-RC Activation Function: Based on this theoretical insight, they introduce COSMO-RC, a novel activation function derived from the Raised Cosine activation. COSMO-RC incorporates the complex sinusoidal modulation and has learnable parameters (bandwidth and frequency shift) for better adaptation to specific tasks.
  • Prior Knowledge Embedder: To address the sensitivity of activation parameters to initialization and task-specific properties, they integrate a prior knowledge embedder (e.g., a ResNet for images) that learns to predict optimal parameters for the COSMO-RC activation layers. This accelerates convergence and improves robustness.
  • Sigmoid Regularizer: A sigmoid projection is used to regularize the output of the prior embedder, further constraining the parameter search space.

Datasets

The paper evaluates COSMO-RC on several standard datasets:

  • Kodak (Franzen) dataset: Used for image reconstruction experiments.
  • DIV2K dataset: Used for image denoising and super-resolution tasks.
  • Celtic spiral knots image: Used for image inpainting.
  • Lucy dataset (Stanford 3D Scanning Repository): Used for occupancy volume representation.
  • Lego dataset: Used for Neural Radiance Fields (NeRF) for novel-view synthesis.

No new datasets were introduced.

Key Insights and Ideas

  • Spectral Attenuation in Symmetric Activations: A key theoretical finding is that odd and even symmetric activation functions inherently lead to spectral attenuation in the post-activation spectrum, limiting their ability to represent high-frequency components.
  • Complex Sinusoidal Modulation as a Solution: Injecting a complex sinusoidal term into activation functions is a principled way to overcome this spectral attenuation, ensuring full spectral support and improving signal representation.
  • COSMO-RC's Superior Spectral Properties: The proposed COSMO-RC activation function, by leveraging this modulation, exhibits better spectral bias characteristics, allowing it to capture high-frequency details more effectively than existing methods like SIREN.
  • Prior Embedder for Robustness and Efficiency: Using a pre-trained network (like ResNet) to predict activation function parameters significantly improves convergence speed and reduces the need for manual hyperparameter tuning, making the model more adaptable and robust.

Key Results

COSMO-RC demonstrates significant improvements across various tasks:

  • Image Reconstruction: Achieved an average PSNR improvement of +5.67 dB over the nearest counterpart on the Kodak dataset, with a peak PSNR of 41.24 dB, significantly outperforming INCODE (35.57 dB).
  • Image Denoising: Showed a +0.46 dB increase in PSNR compared to the nearest SOTA method (INCODE).
  • Super-Resolution: Achieved a +0.64 dB improvement in PSNR for 6x super-resolution compared to the SOTA method.
  • Image Inpainting: Marginally exceeded SOTA methods in single image inpainting, preserving structural continuity and capturing recurring patterns effectively.
  • Occupancy Volume Representation: Outperformed SOTA methods with higher Intersection over Union (IoU) values.
  • Neural Radiance Fields (NeRF): Achieved an average PSNR of 29.50 dB on the Lego test set, surpassing INCODE by +3.45 dB.
  • Efficiency: While COSMO-RC has higher per-iteration training time due to complex operations, it achieves significantly higher accuracy (up to +8.9 dB PSNR gain) for a comparable number of parameters and FLOPs compared to INCODE.

Limitations and Future Work

  • Parameter Sensitivity: The authors note that activation parameters (T and ζ) are sensitive to initialization and should ideally be adjusted for the specific training task. The prior knowledge embedder helps mitigate this, but it's an additional component.
  • Computational Cost: While COSMO-RC offers superior accuracy, its complex sinusoidal activation and prior embedder add elementwise operations, leading to higher per-iteration training times compared to simpler activations.
  • Future Work: The paper suggests that their findings provide a new theoretical perspective that can guide the design of future implicit representations. Further exploration into optimizing the prior embedder and exploring other modulation techniques could be future directions.

Summary updated

BibTeX
@misc{thennakoon2025bandrc,
  author = {Pandula Thennakoon and Avishka Ranasinghe and Mario De Silva and Buwaneka Epakanda and Roshan Godaliyadda and Parakrama Ekanayake and Vijitha R. Herath},
  title = {{BandRC: Band Shifted Raised Cosine Activated Implicit Neural Representations}},
  year = {2025},
  eprint = {2505.11640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.11640},
}
Figure