Abstract
Spectral Probing is a novel, learnable framework that identifies the specific "timescales" (frequencies) at which different linguistic features are encoded in language models. By using a differentiable frequency filter, this research demonstrates that tasks like Part-of-Speech (POS) tagging rely on rapid, word-level changes, while sentiment analysis depends on slow, sentence-level shifts—creating a consistent "spectral profile" for tasks across multiple languages.
Introduction
In the field of Natural Language Processing (NLP), "probing" typically investigates where information is stored within a model's layers. Spectral Probing shifts this focus to at what speed information changes. If we view a sentence as a signal, high-frequency information changes rapidly from word to word (e.g., syntactic roles), while low-frequency information changes slowly across the sequence (e.g., overall topic). Previous attempts to isolate these frequencies used manual, coarse filters. This work introduces a granular, automated, and multilingual method to map the "spectral fingerprints" of NLP tasks.
Methodology
The Spectral Probing framework moves from manual frequency bands to a fully learnable, differentiable filter.
The process involves four key steps:
- Discrete Cosine Transform (DCT): Sequence embeddings are decomposed into cosine waves of varying frequencies.
- Learnable Filter ($\gamma$): A learnable vector $\gamma$ is introduced to automatically amplify or suppress specific frequencies during training.
- Inverse DCT (IDCT): The filtered frequencies are reconstructed into a sequence of embeddings.
- The Probe: A linear classifier is trained on these filtered embeddings. The entire pipeline is differentiable, allowing the filter and classifier to be optimized jointly.
To accommodate varying sentence lengths, adaptive mean pooling is used to scale a fixed-size filter dynamically.
Experimental Setup
The study evaluated the framework on both monolingual (English) and multilingual datasets.
- English: Penn Treebank (PTB) for POS tagging and 20 Newsgroups for Topic classification.
- Multilingual: Seven tasks across six languages (German, English, Spanish, French, Japanese, and Chinese), including Universal Dependencies (POS, DEP), WikiANN (NER), MKQA (QA), and Multilingual Amazon Reviews (Sentiment, Topic).
Results and Discussion
Linguistic Fingerprints
Each task exhibits a distinct "spectral profile." Token-level tasks like POS and NER require a broad spectrum including high frequencies. In contrast, sequence-level tasks like Sentiment and Topic classification rely almost exclusively on low-frequency, slow-changing information.
Key Insights
- Contextual Syntax: Even for "fast" tasks like POS tagging, the learnable filter retains significant low-frequency information, likely to resolve ambiguities using broader context.
- Universal Timescales: Spectral profiles for specific tasks are remarkably consistent across languages (94-98% overlap between English and Japanese for Sentiment Analysis). This suggests that the "speed" of linguistic information is a fundamental property of the task itself.
- Performance Gains: Automated spectral probing outperformed manual filters. For Natural Language Inference (NLI), filtering high-frequency "noise" improved cross-lingual accuracy from 44.1% to 56.3%.
Conclusion
Spectral Probing provides a powerful tool for understanding how language models encode different types of information. While currently focused on high-resource languages and BERT-based architectures, future work will explore low-resource settings and diverse model architectures to further validate these universal linguistic timescales.
Summary updated