Towards Signal Processing In Large Language Models

Prateek Verma, Mert PilanciarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) based on the Transformer architecture have achieved unprecedented success in natural language processing. However, the internal mechanisms governing information flow between layers remain largely opaque, typically treated as transformations of high-dimensional vectors. Standard architectures rely on self-attention and feed-forward networks to process these vectors but lack explicit modeling of the temporal dynamics—the token-to-token evolution—of activations.

This paper introduces a novel approach that treats intermediate activations as 1-D signals. By applying learnable signal processing filters between decoder blocks, the authors demonstrate that LLMs can achieve significantly faster convergence and improved performance with minimal overhead.

Methodology

The proposed framework reinterprets the sequence of activations for each embedding dimension across the context length as a discrete 1-D signal. For a model with embedding dimension $E$ and context length $L$, the system identifies $E$ distinct signals of length $L$ between consecutive decoder blocks.

Signal Extraction and Filtering

The core of the methodology is the integration of a Time-Frequency (T-F) representation and filtering block as a residual connection.

The proposed architecture integrates T-F filtering as a residual block between decoder layers.

Instead of employing fixed transforms like the Discrete Fourier Transform (DFT), the authors utilize a bank of 1-D causal convolutional filters. This design mimics a Short-Time Fourier Transform (STFT) but allows the model to learn optimal basis functions tailored to the latent space of language.

The 1-D causal filter bank decomposes the signal into multiple components for learnable filtering.

The signal is decomposed into $M$ components, where the model learns scalar weights to amplify or suppress specific frequency characteristics. These filtered components are then recombined and added back to the original signal via a residual connection.

Multi-Scale and Adaptive Mechanisms

To capture information at various temporal resolutions, the architecture employs filters of varying lengths (e.g., 3, 7, 15, and 31). Furthermore, an "adaptive masking" variant was introduced, where a small auxiliary decoder predicts token-specific masks, allowing the filter characteristics to evolve dynamically throughout the sequence.

Experimental Setup

The effectiveness of the proposed method was evaluated on two primary tasks:

  1. Natural Language Processing: Training on the text-8 dataset using character-level tokenization (approx. 0.5 billion tokens).
  2. Audio Classification: A non-causal validation on the FSD-50K dataset, comprising 51,000 audio snippets across 200 categories, to demonstrate the versatility of the signal-processing approach.

Results

The inclusion of learnable signal processing filters yielded substantial improvements in training efficiency and model performance:

  • Convergence Speed: The T-F integrated architecture achieved 40-45% faster convergence in training steps compared to a standard mini-GPT baseline.
  • Predictive Accuracy: On the text-8 test set, the Multi-Scale T-F model reached a Negative Log-Likelihood (NLL) of 1.01, outperforming the baseline's 1.04.
  • Architectural Efficiency: The multi-scale configuration provided a 44% speedup to target performance. The addition of token-varying adaptive weights further increased this speedup by 26% over the fixed multi-scale version.

Discussion and Insights

Embeddings as Trajectories

A key conceptual shift in this work is viewing the sequence of embeddings as a trajectory through high-dimensional space. Signal processing techniques allow the model to "shape" these trajectories, providing a mechanism to control the rate of information change across tokens.

Learned Basis Functions

Analysis of the learned convolutional kernels revealed that they do not converge to standard sinusoidal patterns. Instead, they often resemble "onsets" or impulsive bursts. This suggests that LLMs benefit from detecting specific temporal patterns of change that are not easily captured by classical Fourier analysis.

Limitations and Future Directions

While the results on "mini-GPT" models (8 layers, 128 embedding dimension) are promising, the scalability to billion-parameter models remains to be verified. Additionally, the impact of irregular temporal structures in sub-word tokenization (e.g., BPE) versus character-level tokenization warrants further investigation. Future work may also explore applying these signal processing blocks to alternative architectures such as State Space Models (e.g., Mamba).

Summary updated

BibTeX
@misc{verma2024towards,
  author = {Prateek Verma and Mert Pilanci},
  title = {{Towards Signal Processing In Large Language Models}},
  year = {2024},
  eprint = {2406.10254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10254},
}
Figure