Spectral Probing

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Abstract

Spectral Probing is a novel, learnable framework that identifies the specific "timescales" (frequencies) at which different linguistic features are encoded in language models. By using a differentiable frequency filter, this research demonstrates that tasks like Part-of-Speech (POS) tagging rely on rapid, word-level changes, while sentiment analysis depends on slow, sentence-level shifts—creating a consistent "spectral profile" for tasks across multiple languages.

Introduction

In the field of Natural Language Processing (NLP), "probing" typically investigates where information is stored within a model's layers. Spectral Probing shifts this focus to at what speed information changes. If we view a sentence as a signal, high-frequency information changes rapidly from word to word (e.g., syntactic roles), while low-frequency information changes slowly across the sequence (e.g., overall topic). Previous attempts to isolate these frequencies used manual, coarse filters. This work introduces a granular, automated, and multilingual method to map the "spectral fingerprints" of NLP tasks.

Methodology

The Spectral Probing framework moves from manual frequency bands to a fully learnable, differentiable filter.

The Spectral Probing pipeline: 1) Input signal, 2) DCT decomposition, 3) Learnable filter application, 4) IDCT reconstruction.

The process involves four key steps:

  1. Discrete Cosine Transform (DCT): Sequence embeddings are decomposed into cosine waves of varying frequencies.
  2. Learnable Filter ($\gamma$): A learnable vector $\gamma$ is introduced to automatically amplify or suppress specific frequencies during training.
  3. Inverse DCT (IDCT): The filtered frequencies are reconstructed into a sequence of embeddings.
  4. The Probe: A linear classifier is trained on these filtered embeddings. The entire pipeline is differentiable, allowing the filter and classifier to be optimized jointly.

To accommodate varying sentence lengths, adaptive mean pooling is used to scale a fixed-size filter dynamically.

Experimental Setup

The study evaluated the framework on both monolingual (English) and multilingual datasets.

  • English: Penn Treebank (PTB) for POS tagging and 20 Newsgroups for Topic classification.
  • Multilingual: Seven tasks across six languages (German, English, Spanish, French, Japanese, and Chinese), including Universal Dependencies (POS, DEP), WikiANN (NER), MKQA (QA), and Multilingual Amazon Reviews (Sentiment, Topic).

Results and Discussion

Linguistic Fingerprints

Each task exhibits a distinct "spectral profile." Token-level tasks like POS and NER require a broad spectrum including high frequencies. In contrast, sequence-level tasks like Sentiment and Topic classification rely almost exclusively on low-frequency, slow-changing information.

Spectral profiles across different NLP tasks, showing the transition from high-frequency reliance in syntactic tasks to low-frequency dominance in semantic/sequence-level tasks.

Key Insights

  • Contextual Syntax: Even for "fast" tasks like POS tagging, the learnable filter retains significant low-frequency information, likely to resolve ambiguities using broader context.
  • Universal Timescales: Spectral profiles for specific tasks are remarkably consistent across languages (94-98% overlap between English and Japanese for Sentiment Analysis). This suggests that the "speed" of linguistic information is a fundamental property of the task itself.
  • Performance Gains: Automated spectral probing outperformed manual filters. For Natural Language Inference (NLI), filtering high-frequency "noise" improved cross-lingual accuracy from 44.1% to 56.3%.

Conclusion

Spectral Probing provides a powerful tool for understanding how language models encode different types of information. While currently focused on high-resource languages and BERT-based architectures, future work will explore low-resource settings and diverse model architectures to further validate these universal linguistic timescales.

Summary updated

Abstract

Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages—highlighting their potential as robust, lightweight task descriptors.
BibTeX
@inproceedings{mullereberstein2022spectral,
  author = {Max Müller-Eberstein and Rob van der Goot and Barbara Plank},
  title = {{Spectral Probing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.527},
}
Figure