Language Through a Prism: A Spectral Approach for Multiscale Language Representations

Alex Tamkin, Dan Jurafsky, Noah D. Goodman

Overview

Researchers from Stanford have introduced a "Prism Layer" that uses signal processing techniques—specifically the Discrete Cosine Transform (DCT)—to force different neurons in a language model to focus on different linguistic scales. By isolating frequencies from individual words to entire document topics, this approach significantly boosts performance on long-range tasks like topic classification (+18.8%) while making the model more sensitive to distant context.

The Challenge of Multiscale Language

Language is inherently hierarchical: morphology happens at the subword level, syntax and meaning at the word and sentence levels, and overarching topics at the document level. However, standard Transformer models like BERT often treat these scales uniformly. While BERT can capture these structures, it doesn't always do so efficiently or explicitly.

The authors identify a critical gap: we lack a principled way to disentangle these scales within a model's internal representations. They ask: Can we force a model to "specialize" its neurons so that some focus on the fast-changing details (words) while others focus on the slow-moving trends (topics)?

Methods: Spectral Analysis and the Prism Layer

To solve this, the authors turn to Spectral Analysis, a staple of signal processing.

  1. Discrete Cosine Transform (DCT): They treat a neuron's activations across a sequence of tokens as a digital signal. By applying the DCT, they move from the "token domain" to the "frequency domain." High frequencies represent rapid changes (e.g., part-of-speech tags changing word-to-word), while low frequencies represent gradual shifts (e.g., the overarching topic of a paragraph).
  2. Spectral Filtering: They apply filters to these activations. For example, a low-pass filter "blurs" the activations, removing local word-level noise to reveal document-level signals.
Spectral filtering transforms signals from the input domain to the frequency domain, allowing isolation of low-pass and high-pass frequencies.
  1. The Prism Layer: This is the core architectural contribution. The researchers divide a model's hidden units into five "sectors." Each sector is constrained by a different band-pass filter during training. This forces the model to allocate specific neurons to specific linguistic scales.
The Prism Layer splits neural activations into different frequency bands (low, band, and high pass), forcing neurons to specialize in different linguistic scales.

Key Insights and Results

The researchers evaluated their approach using standard benchmarks like the Penn Treebank (word-level), Switchboard Dialog Speech Acts (utterance-level), and 20 Newsgroups (document-level).

  • Filtering as a Probing Tool: One of the most striking findings is that simply applying a low-pass filter to a pre-trained BERT's activations (without any extra training) makes it significantly better at topic classification. This suggests that local word-level information actually acts as "noise" for global tasks.
  • Specialization via Constraint: By forcing neurons to specialize during training via the Prism Layer, the model learns to represent long-range dependencies more effectively.
  • MLM is Local: The authors found that standard Masked Language Modeling (MLM) is a very "high-frequency" task. BERT's default behavior relies heavily on immediate neighbors to predict a missing word, often ignoring distant context.

The BERT + Prism model showed substantial improvements over a vanilla BERT model:

  • Topic Classification: Accuracy jumped from 32.21% to 51.01% (+18.8% absolute improvement).
  • Dialog Speech Acts: Accuracy increased from 47.09% to 54.02% (+6.9%).
  • Part-of-Speech Tagging: The model maintained high performance (94.41% vs 95.86%), proving that specializing for global tasks doesn't have to break local understanding.
  • Long-range Sensitivity: In a "stress test" where 100 consecutive tokens were masked, the Prism model was significantly better at using distant context to predict the missing words compared to standard BERT.

Limitations and Future Work

  • Filter Design: The authors used "brick-wall" filters (abruptly cutting off frequencies), which can sometimes introduce mathematical artifacts. Smoother attenuation functions might yield even better results.
  • Fixed Bands: The frequency bands were pre-defined. Future work could involve "learnable" filters where the model decides which scales are most important for a given task.
  • Beyond Text: The authors note that this spectral approach is "domain agnostic" and could be highly effective for analyzing multiscale structures in images, audio, or video.

Summary updated

Abstract

Language exhibits structure at a wide range of scales, from subwords to words, sentences, paragraphs, and documents. We propose building models that isolate scale-specific information in deep representations, and develop methods for encouraging models during training to learn more about particular scales of interest. Our method for creating scale-specific neurons in deep NLP models constrains how the activation of a neuron can change across the tokens of an input by interpreting those activations as a digital signal and filtering out parts of its frequency spectrum. This technique enables us to extract scale-specific information from BERT representations: by filtering out different frequencies we can produce new representations that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for use during training, which constrains different neurons of a BERT model to different parts of the frequency spectrum. Our proposed BERT + Prism model is better able to predict masked tokens using long-range context, and produces individual multiscale representations that perform with comparable or improved performance across all three tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.
BibTeX
@inproceedings{tamkin2020language,
  author = {Alex Tamkin and Dan Jurafsky and Noah D. Goodman},
  title = {{Language Through a Prism: A Spectral Approach for Multiscale Language Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=k1hH-3YL4T},
}
Figure