Language Through a Prism: A Spectral Approach for Multiscale Language Representations

Alex Tamkin, Dan Jurafsky, Noah D. Goodman

Abstract

Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn more about particular scales. Concretely, we apply spectral filters to the activations of a neuron across an input, producing filtered embeddings that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for training models, which uses spectral filters to constrain different neurons to model structure at different scales. Our proposed BERT + Prism model can better predict masked tokens using long-range context and produces multiscale representations that perform better at utterance- and document-level tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.

Introduction

Language is inherently multiscale, possessing a fractal-like structure where meaning emerges at the level of subwords, words, sentences, and entire documents. While Transformer-based models like BERT have become the standard for processing linguistic data, they often exhibit a "local bias." Because these models are primarily trained on Masked Language Modeling (MLM)—predicting a missing token based on its immediate neighbors—they excel at capturing local syntax but may struggle to represent the broader, "slower" thematic movements of a document. This paper investigates whether we can explicitly disentangle these scales by forcing specific neurons to specialize in specific timescales rather than hoping the model learns to represent diverse information in a single high-dimensional space.

Methods: Spectral Analysis and the Prism Layer

The core innovation of this work is the application of classical signal processing techniques to neural activations. By treating a neuron’s activations across a sequence of tokens as a signal, we can apply the Discrete Cosine Transform (DCT) to move from the token domain to the frequency domain. In this domain, high frequencies represent rapid changes (e.g., word-level fluctuations), while low frequencies represent gradual changes (e.g., document-level themes).

The Prism Layer architecture constrains neurons to specific frequency bands, forcing the model to represent information at multiple scales.

The researchers introduced the Prism Layer, an architectural component that divides neurons into distinct "sectors." During training, each sector is subjected to a different spectral filter (low-pass, band-pass, or high-pass). This constraint forces the model to allocate its representational capacity across all linguistic scales, from jittery word-level details to smooth document-level trends.

Experimental Setup

The approach was evaluated across three distinct linguistic scales using various datasets:

  • Word-level: Part-of-Speech (POS) tagging using the Penn Treebank.
  • Utterance-level: Dialog speech act classification using the Switchboard Dialog Speech Acts corpus.
  • Document-level: Topic classification using the 20 Newsgroups dataset.
  • Training: Models were fine-tuned and trained on the WikiText-103 dataset.

Results and Analysis

A key finding is that linguistic tasks have "natural" frequencies. High-pass filters, which preserve fast changes, are effective for POS tagging but perform poorly for topic classification. Conversely, low-pass filters can actually improve topic classification by removing noisy word-level fluctuations.

Accuracy on Dialog Speech Acts across different spectral bands, showing that mid-to-high frequencies are most relevant for this utterance-level task.

The BERT + Prism model demonstrated significant improvements on "slower" linguistic tasks while remaining competitive on local ones. For instance, topic classification accuracy reached 51.01%, an 18.8% absolute improvement over vanilla BERT (32.21%). Dialog speech act classification also saw a 6.9% improvement. Qualitatively, the model showed a higher log-probability for correct tokens in heavily masked text, indicating a superior ability to utilize long-range context compared to vanilla BERT, which often fails when local cues are absent.

Conclusion

The Prism Layer provides a powerful inductive bias that forces neural networks to attend to the "big picture" without sacrificing local precision. While the current implementation uses "brick-wall" filters, future work could explore smoother attenuation functions. Additionally, while this study focused on NLP, the Prism Layer is domain-agnostic and could potentially be applied to computer vision or video processing to separate local textures from global structures.

Summary updated

Abstract

Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn more about particular scales. Concretely, we apply spectral filters to the activations of a neuron across an input, producing filtered embeddings that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for training models, which uses spectral filters to constrain different neurons to model structure at different scales. Our proposed BERT + Prism model can better predict masked tokens using long-range context and produces multiscale representations that perform better at utterance- and document-level tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.
BibTeX
@inproceedings{tamkin2020language,
  author = {Alex Tamkin and Dan Jurafsky and Noah D. Goodman},
  title = {{Language Through a Prism: A Spectral Approach for Multiscale Language Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=k1hH-3YL4T},
}

Loading related content...

Figure