When we train, fine-tune, or aggregate neural networks, we typically treat their weight matrices as flat collections of independent numerical coordinates. Operations like weight averaging in federated learning or gradient updates in fine-tuning happen element-wise. This spatial approach assumes that a specific coordinate in one model's weight matrix means exactly the same thing as the same coordinate in another.
However, neural network weights possess an underlying structural topology. If we treat a 2D weight matrix as a continuous spatial signal and pass it through a frequency transform—such as the Fast Fourier Transform (FFT) or the Discrete Cosine Transform (DCT)—a striking hierarchy emerges.
Instead of looking at where a weight is, frequency-domain manipulation looks at how smoothly weights vary across the matrix. This perspective reveals that a model's core, generalizable intelligence is encoded in the low-frequency components of its weights, while task-specific nuances, client-specific variations, and optimization noise reside in the high-frequency components. By filtering and aggregating parameters in the frequency domain, researchers can solve fundamental challenges in federated learning, catastrophic forgetting, and cross-architecture knowledge transfer.
The Spectral Hierarchy of Weights
To understand why this works, we must look at how neural networks learn. A foundational insight into this behavior comes from Nasim Rahaman et al. in On the Spectral Bias of Deep Neural Networks. The authors demonstrated that deep networks exhibit a "spectral bias," learning the low-frequency, smoother components of a target function long before they learn the high-frequency, rapidly oscillating details. Crucially, they showed that these low-frequency components are significantly more robust to parameter perturbations; they remain stable across a wide range of weight configurations, whereas high-frequency patterns require precise coordinate-level alignment and degrade rapidly under noise.
When applied directly to the weight matrices themselves, this principle yields a powerful taxonomy:
- Low-Frequency Components: Represent the macro-structure of the weight matrix. This is the foundational, task-agnostic knowledge that forms the bedrock of the model's capabilities.
- High-Frequency Components: Represent micro-level fluctuations. These capture highly specific data distributions, fine-grained local features, or transient noise introduced during optimization.
Recognizing this hierarchy allows us to treat model manipulation as a signal processing task, using low-pass, high-pass, or band-pass filters to surgically alter a model's knowledge.
Spatial Weight Matrix Frequency Domain (FFT/DCT) Surgical Manipulation
[ w11 w12 w13 ] [ Low Freq: Core Logic ] --> Keep/Average (Generalize)
[ w21 w22 w23 ] ======> [ ]
[ w31 w32 w33 ] [ High Freq: Nuance/Noise] --> Filter/Isolate (Personalize)
Aligning Heterogeneous Models in Federated Learning
In federated learning (FL), a central server aggregates local models trained by distributed clients. When client datasets are highly heterogeneous (non-IID), standard element-wise averaging causes a phenomenon known as "retrogress," where a client's performance plummets immediately after receiving a global update. This happens because coordinate-wise averaging assumes spatial alignment; if two clients learn the same general feature but store it in slightly different weight coordinates, averaging them spatially destroys both.
To bypass this, Zhen Chen et al. proposed a frequency-domain aggregation strategy in Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning and expanded upon it in Personalized Retrogress-Resilient Federated Learning Toward Imbalanced Medical Data. Instead of averaging weights directly, the authors transform convolutional layer weights into the frequency domain using an FFT.
Because generalizable features are anchored in the low-frequency spectrum, the server averages only the low-frequency coefficients across clients while leaving each client's high-frequency coefficients intact. This aligns features along frequency dimensions rather than rigid spatial coordinates, allowing the global model to distribute foundational knowledge without overwriting the specialized, high-frequency local expertise unique to individual hospitals.
Similarly, Yuangang Li et al. leveraged this concept in FedMetaMed: Federated Meta-Learning for Personalized Medication in Distributed Healthcare Systems. By prioritizing the aggregation of low-frequency components during the early stages of federated meta-learning, their framework establishes a stable foundation of general features before attempting to incorporate complex, high-frequency details, leading to superior generalization across distributed healthcare systems.
Preventing Forgetting and Executing Clean Unlearning
The separation of core logic from transient details is equally powerful when a model must learn new tasks sequentially or, conversely, unlearn specific data.
In sequential fine-tuning, models suffer from catastrophic forgetting because new gradient updates overwrite historical parameter distributions. In Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning, Xiao Han et al. introduced DEAL, a framework that treats Low-Rank Adaptation (LoRA) matrices as signals. They apply a wavelet-based heat kernel as a low-pass filter to extract and preserve the low-frequency components where core task knowledge resides. By shielding these essential historical distributions from being overwritten during sequential training, the model retains past knowledge while adapting to new tasks.
The inverse problem occurs in Federated Client Unlearning (FCU), where a model must completely forget a specific client's data to respect privacy rights. In Enable the Right to be Forgotten with Federated Client Unlearning in Medical Imaging, Hao Chen et al. combined contrastive learning with frequency-domain weight filtering. To make a model "forget" a client without destroying its overall utility, the authors surgically protect the low-frequency components that represent shared foundational knowledge. They allow only the high-frequency components—where the specific client's unique data signatures and noise reside—to be modified and erased. This allows the model to unlearn specific data 10–15 times faster than retraining from scratch.
Cross-Scale Transfer and Zero-Shot Denoising
Because low-frequency weight components capture architecture-agnostic foundational knowledge, they can also bridge the gap between models of entirely different scales.
In One-for-All Model Initialization with Frequency-Domain Knowledge, Jianlu Shen et al. introduced FRONT, a framework that extracts a model's core intelligence—or "learngene"—using the Discrete Cosine Transform (DCT). By applying a binary mask to the frequency coefficients, the authors isolate the highly stable low-frequency components of a pre-trained model. Because these components represent macro-level features, they can be seamlessly transferred to downstream models of any size via simple padding or truncation in the frequency domain, accelerating convergence by up to 15×.
The authors extended this paradigm in Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform, confirming that these low-frequency components remain remarkably stable during fine-tuning across different model scales, while high-frequency components fluctuate rapidly to adapt to specific tasks.
Finally, this signal-processing view enables optimization without any training at all. In Unleashing the Potential of Large Language Models through Spectral Modulation, Peng Sun et al. treated the weight matrices of pre-trained Large Language Models (LLMs) as 2D signals. They discovered that over-parameterized LLMs accumulate performance-constraining "noise" in their high-frequency spectrum during standard training. By applying a post-hoc spectral modulation that dampens these high frequencies while preserving the low-frequency core logic, the authors achieved up to a 10% performance boost in a completely zero-shot manner.
Key Foundational Work
Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning (2021) — contributed 3 concept elements to this direction.
On the Spectral Bias of Deep Neural Networks (2018) — contributed 1 concept elements to this direction.