Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions

Stefano Massaroli, Michael Poli, Daniel Y. Fu, Hermann Kumbong, Rom N. Parnichkun, Aman Timalsina, David W. Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, Yoshua Bengio

Abstract

Long Convolution Sequence Models (LCSMs) like Hyena and H3 have emerged as efficient alternatives to Transformers, offering sub-quadratic scaling during training via Fast Fourier Transforms (FFTs). However, these models often face an "inference gap," where auto-regressive generation remains computationally expensive compared to recurrent models. This paper introduces the Laughing Hyena Distillery, a mathematical framework designed to distill long convolution filters into compact State-Space Models (SSMs). By converting convolutions into recurrences, the authors achieve a 10x increase in throughput over Transformers and a 1.5x increase over the original Hyena architecture at the 1.3B parameter scale, maintaining model quality while significantly reducing memory overhead.

Introduction

The quest for scalable sequence modeling has led to the development of "attention-free" architectures. Models such as Hyena leverage long convolutions to capture global dependencies without the quadratic cost of self-attention. While these models excel during training, their inference performance is hampered by the lack of a native recurrent form. Unlike traditional Recurrent Neural Networks (RNNs) or modern SSMs, long convolutions typically require re-processing the entire history or maintaining large caches to generate new tokens, creating a bottleneck in real-time applications.

The Inference Gap in Long Convolutions

In auto-regressive generation, the goal is to produce tokens one by one. For a convolution-based model, calculating the next token naively involves a convolution over the entire prefix. While FFTs make this efficient for fixed-length sequences during training, they do not naturally support the $O(1)$ state updates required for fast inference. This results in an "inference gap" where the model either consumes excessive memory (similar to the KV-cache in Transformers) or suffers from increased latency as the sequence grows.

The Laughing Hyena Distillery

The authors propose a distillation pipeline that transforms pre-trained convolution filters into a recurrent State-Space Model (SSM) representation. This process is not a traditional student-teacher retraining but a mathematical conversion based on system theory.

1. Modal Interpolation

Instead of treating distillation as a standard optimization problem, the framework views it as a rational interpolation task. The distilled model is represented in a modal canonical form, where the impulse response is modeled as a sum of exponentially decaying complex sinusoids. This allows the convolution to be mapped directly to a linear recurrence.

2. Hankel Operator Analysis

To determine the necessary size of the recurrent state (the "memory"), the authors analyze the Hankel matrix of the pre-trained filters. By performing Singular Value Decomposition (SVD) on this matrix, they can mathematically identify the "effective dimension" or McMillan degree of the filter. Surprisingly, even filters spanning thousands of tokens often possess a low-rank structure, allowing them to be captured by a state dimension ($d$) as small as 16 or 32.

3. Multi-head Hyena

To further optimize the architecture, the authors introduce "Multi-head Hyena." Similar to multi-head attention, this approach ties weights across channels into heads. This reduces the number of unique filters requiring distillation and acts as a regularizer, improving the quality of the pre-trained model.

4. Fast Pre-filling

For the initial prompt processing (pre-filling), the authors utilize an FFT-based algorithm to compute the starting state of the recurrence. This ensures that the transition from prompt processing to token generation is seamless and fast.

Experimental Results

The framework was evaluated on large-scale benchmarks, including The Pile and the Associative Recall task, as well as zero-shot evaluations via HELM and LM-Eval-Harness.

Performance and Efficiency

  • Throughput: At the 1.3B parameter scale, the distilled Laughing Hyena model achieves 10x higher peak throughput than a comparable Transformer and 1.5x higher than the original Hyena.
  • Memory Footprint: The model requires 3x less memory than a Transformer when generating 512 tokens, as it replaces the growing KV-cache with a fixed-size recurrent state.
  • Lossless Quality: Benchmarks on tasks like LAMBADA, PIQA, and HellaSwag show that distillation is virtually lossless. Provided the state dimension $d \ge 16$, the distilled recurrent model matches the accuracy of the original long-convolution model.

Scaling and Perplexity

The Multi-head Hyena architecture demonstrated superior scaling properties. At 355M parameters, it achieved a perplexity of 8.9 on The Pile, outperforming both the standard Hyena (9.2) and GPT/Pythia (9.1) models of equivalent size.

Discussion and Limitations

While the Laughing Hyena Distillery provides a robust path to fast inference, the authors note several considerations:

  • State Dimension Sensitivity: Performance is highly sensitive to the state dimension $d$. If $d$ is too low (e.g., $d < 16$), the model's ability to handle complex linguistic tasks like LAMBADA collapses.
  • Numerical Stability: Classical reduction techniques like balanced truncation were found to be unstable for these filters, necessitating the modal interpolation approach.
  • Implementation: Achieving theoretical $O(1)$ gains in practice requires highly optimized hardware kernels for recurrent updates, which remains an active area of optimization.

Conclusion

The Laughing Hyena Distillery bridges the gap between the efficient training of long convolutions and the fast inference of recurrent models. By revealing that long convolutions in language models are inherently low-rank, the authors provide a mathematical foundation for deploying massive, attention-free models with the speed and memory efficiency of a compact recurrence.

Summary updated

BibTeX
@inproceedings{massaroli2023laughing,
  author = {Stefano Massaroli and Michael Poli and Daniel Y. Fu and Hermann Kumbong and Rom N. Parnichkun and Aman Timalsina and David W. Romero and Quinn McIntyre and Beidi Chen and Atri Rudra and Ce Zhang and Christopher Ré and Stefano Ermon and Yoshua Bengio},
  title = {{Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=OWELckerm6},
}
Figure