Compact Circulant Layers with Spectral Priors

Joseph Margaryan, Thomas Hamelryck

TL;DR

This paper introduces Spectral Circulant and BCCB layers, which parameterize neural network weights directly in the frequency domain using the Real Fast Fourier Transform (RFFT). This approach creates extremely compact models (up to 72x fewer parameters) that are inherently "uncertainty-aware" and provide nearly free mathematical guarantees for robustness.

What Problem Does This Paper Address?

Modern deep learning faces a "triple threat" in deployment:

  1. Resource Constraints: Standard networks are memory-hungry, making them hard to fit on edge devices like robots or medical sensors.
  2. Overconfidence: Traditional networks often give "confidently wrong" answers when they encounter data they haven't seen before (distribution shift).
  3. Brittleness: Small perturbations to input data can cause catastrophic failures, and calculating a network's "Lipschitz constant" (a measure of stability) is usually computationally expensive.

While Gaussian Processes (GPs) offer beautiful mathematical solutions for uncertainty, they don't scale well to large datasets. This paper bridges the gap by using circulant matrix structures to bring the benefits of GPs into the world of deep learning efficiently.

Methods

The authors propose parameterizing layers not as standard weight matrices, but as circulant (for 1D) or Block-Circulant-with-Circulant-Blocks (BCCB) (for 2D) operators.

The "secret sauce" is where these parameters live:

  • Spectral Parameterization: Instead of learning spatial filters, the model learns coefficients directly in the RFFT (Real Fast Fourier Transform) domain.
  • Spectral Priors: By placing independent complex Gaussian distributions on these frequency coefficients, the authors induce an exact stationary Gaussian Process prior over the filters.
  • Band-limiting: Because they work in the frequency domain, they can simply "turn off" high-frequency components to compress the model further without leaving the spectral representation.
  • Free Lipschitz Bounds: Because circulant matrices are diagonalized by the Fourier Transform, calculating the layer's spectral norm (and thus its stability) becomes a simple matter of finding the maximum value in the frequency response—a calculation that is essentially "free."

Datasets

The authors evaluated their method across several standard benchmarks:

  • MNIST & Fashion-MNIST: Used for training small Bayesian Neural Networks (BNNs) and testing Out-of-Distribution (OOD) detection.
  • CIFAR-10 & CIFAR-10-C: Used to test variational heads on frozen features and evaluate robustness against corrupted images.
  • Tiny ImageNet: Used to test the scalability of these layers as drop-in replacements for projections in Vision Transformers (ViTs).

Key Insights and Ideas

  • The Geometry of Frequency: Parameterizing in the frequency domain isn't just a mathematical trick; it aligns the trainable parameters with the way we want to regularize them (e.g., smoothing filters by penalizing high frequencies).
  • SVI Made Practical: Stochastic Variational Inference (SVI) is often unstable in deep networks. By using a "Hermitian-aware" low-rank posterior in the frequency domain, the authors make Bayesian deep learning much more tractable and parameter-efficient.
  • Compactness without Sacrifice: You can throw away a huge portion of the "weights" (high-frequency coefficients) and still maintain high accuracy, because natural images are dominated by low-frequency information.
  • Free Diagnostics: Usually, knowing if your model is "robust" requires complex adversarial attacks. Here, the Lipschitz certificates are a built-in geometric diagnostic.

Key Results

  • Extreme Compression: On MNIST, the spectral models achieved competitive accuracy using 7x to 72x fewer parameters than dense baselines.
  • Better Uncertainty: The spectral layers showed significantly better separation between "In-Distribution" and "Out-of-Distribution" data (higher AUROC) compared to standard convolutional layers.
  • ViT Efficiency: Replacing square linear projections in a Vision Transformer with spectral circulant layers improved accuracy and likelihood on CIFAR-10 while using ~1.5x fewer parameters.
  • Robustness: The models provided much tighter Lipschitz certificates and larger "certified radii" (the amount of noise an input can take before the prediction changes) than dense counterparts.

Limitations and Future Work

  • Boundary Artifacts: Circular convolutions assume the edges of an image wrap around to the other side, which can introduce "edge artifacts." While padding can help, it breaks the strict BCCB structure.
  • Scaling to Deep Stacks: While the authors successfully used these in ViT projections and Bayesian heads, scaling structured SVI (Bayesian inference) to very deep, end-to-end spectral stacks remains a challenge.
  • Simple Priors: The current work uses relatively simple radial frequency profiles for priors; the authors suggest that learning richer, data-driven spectral priors is a promising next step.

Summary updated

Abstract

Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.
BibTeX
@misc{margaryan2026compact,
  author = {Joseph Margaryan and Thomas Hamelryck},
  title = {{Compact Circulant Layers with Spectral Priors}},
  year = {2026},
  eprint = {2602.21965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.21965},
}
Figure