Spectral Probing

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Introduction

Contextualized embeddings from models like BERT encode a wealth of linguistic information across different layers. However, recent research suggests that this information also manifests at different timescales. Some linguistic features, such as part-of-speech (POS) tags, change rapidly from token to token (high frequency), while others, like sentiment or topic, remain stable across entire sequences (low frequency).

Previous attempts to analyze these timescales relied on rigid, handcrafted "fixed-band" filters. These methods are often imprecise and difficult to scale across languages. Spectral Probing addresses this by introducing a fully differentiable method to learn optimal frequency filters for any given NLP task, effectively acting as a learnable equalizer for language model representations.

Methodology

The Spectral Probing framework treats a sequence of embeddings as a signal and applies a learnable frequency filter through four key steps:

  1. Discrete Cosine Transform (DCT): A sequence of embedding values is decomposed into its composite frequency waves. High frequencies represent rapid changes between adjacent tokens, while low frequencies represent slow-moving, sequence-level trends.
  2. Learnable Filters ($\gamma$): Instead of using fixed bands, the model introduces a differentiable parameter vector ($\gamma$) that assigns a weight between 0 and 1 to every possible frequency.
  3. Inverse DCT (IDCT): After the filter is applied in the frequency domain, an Inverse DCT reconstructs a "filtered" version of the embeddings.
  4. Probing: These filtered embeddings are passed to a simple linear probe. Because the filter is differentiable, the model learns exactly which frequencies are most informative for a given task during standard training.
The Spectral Probing architecture: embeddings are transformed via DCT, filtered by a learnable gamma vector, and reconstructed via IDCT before probing.

Analysis of Spectral Profiles

The researchers identified distinct "spectral profiles" for different NLP tasks, revealing a clear dichotomy between token-level and sequence-level information.

  • Token-level tasks (e.g., POS tagging, NER) rely on a broad range of frequencies, including high-frequency components that capture local changes between adjacent tokens.
  • Sequence-level tasks (e.g., Sentiment Analysis, Topic Classification) depend almost exclusively on the lowest frequencies, representing global, sequence-wide trends.
Learned spectral profiles for different NLP tasks. Token-level tasks like POS use high frequencies, while sequence-level tasks like Topic classification are restricted to low frequencies.

These profiles follow a logical linguistic hierarchy. POS tagging utilizes the highest frequencies, while Dependency Relation classification (DEP) shifts slightly lower, and Named Entity Recognition (NER)—which often requires broader context—shifts lower still.

Cross-Lingual Consistency and Performance

A significant finding is the cross-lingual stability of these profiles. The frequency mixture required for a POS tagger in Japanese is nearly identical to that for English (94–98% overlap). This suggests that spectral profiles are inherent to the linguistic task itself, rather than the specific language or model architecture.

Furthermore, filtering out irrelevant frequencies often improves performance. For Topic classification on the 20 Newsgroups dataset, the spectral probe achieved 72.1% accuracy, a massive leap from the 41.3% achieved using unfiltered embeddings. In multilingual settings, automatically filtered ("AUTO") embeddings consistently matched or outperformed original ("ORIG") embeddings, with the most significant gains seen in complex tasks like Natural Language Inference (NLI), where average performance improved from 44.1% to 56.3%.

Conclusion

Spectral Probing provides a granular and efficient lens for understanding how language models distribute information across temporal frequencies. By moving beyond fixed-band filters to a learnable approach, this method not only reveals the "spectral fingerprints" of linguistic tasks but also offers a practical way to enhance model performance by focusing on task-relevant signals. Future work will look to extend these findings to lower-resource languages and different model architectures like GPT or T5.

Summary updated

BibTeX
@inproceedings{mullereberstein2022spectral,
  author = {Max Müller-Eberstein and Rob van der Goot and Barbara Plank},
  title = {{Spectral Probing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.527},
}
Figure