Liquid Structural State-Space Models

Ramin M. Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, Daniela Rus

Introduction

Sequence modeling for long-range dependencies remains a significant challenge in machine learning. Tasks involving thousands of steps, such as processing long documents or high-frequency audio, require models that are both expressive and computationally efficient. Two prominent approaches have emerged: Liquid Time-Constant (LTC) networks, which offer biological inspiration and input-dependent adaptability but suffer from poor scalability, and Structural State-Space Models (S4), which provide exceptional efficiency through convolutional kernels but lack input-adaptive dynamics.

This paper introduces Liquid-S4, a novel architecture that bridges this gap. By linearizing LTCs into a form compatible with S4's efficient computation, Liquid-S4 achieves state-of-the-art (SOTA) performance on long-range benchmarks while maintaining the adaptive power of liquid networks.

Methodology: The Linearization of Liquid Networks

The core innovation of Liquid-S4 is the mathematical integration of LTC dynamics into the S4 framework. The authors apply a first-order Taylor approximation to the non-linear LTC equations, resulting in a "Linear LTC" where the state transition matrix adapts to the input: $\dot{x}(t) = [A + B u(t)] x(t) + B u(t)$.

Upon discretization, this formulation yields a "Double Kernel" structure. Unlike standard S4, which uses a single convolutional kernel, Liquid-S4 produces:

  1. A standard S4 kernel for long-term memory.
  2. A new "Liquid" kernel that accounts for the auto-correlation of the input signal.

To maintain efficiency, the model utilizes Diagonal Plus Low-Rank (DPLR) decomposition and Cauchy kernels, ensuring $O(L \log L)$ complexity for sequences as long as 16,000+ steps.

Experimental Evaluation

Liquid-S4 was evaluated across diverse benchmarks, including the Long-Range Arena (LRA), medical time-series (BIDMC), and raw speech commands.

Long-Range Arena (LRA) Results

The LRA benchmark tests a model's ability to handle various modalities (text, images, mathematical logic) over long sequences. Liquid-S4 established a new state-of-the-art average performance of 87.32%.

Performance comparison on the Long-Range Arena (LRA) benchmark. Liquid-S4 achieves the highest average score and excels on the challenging Path-X task.

Notably, on the Path-X task—the most difficult in the suite with 16k length sequences—Liquid-S4 achieved 96.66% accuracy.

Additional Benchmarks

  • Speech Recognition: On the Speech Commands dataset, Liquid-S4 reached 96.78% accuracy, surpassing the original S4 while utilizing 30% fewer parameters.
  • Medical Time-Series: On the BIDMC dataset, the model achieved the lowest Root Mean Square Error (RMSE) for predicting vital signs (HR, RR, SpO2) compared to specialized RNN architectures.

Key Insights

  • Adaptive Dynamics: The "liquid" property allows the model to adapt its internal state transitions to the incoming data, enhancing generalization and out-of-distribution performance.
  • Parameter Efficiency: Liquid-S4 often achieves superior results with significantly smaller internal state sizes (e.g., 7 units) compared to traditional models.
  • Kernel Synergy: The interaction between the standard S4 kernel and the Liquid kernel allows for simultaneous long-term memory and specialized feature extraction from temporal correlations.

Conclusion and Future Work

Liquid-S4 represents a significant step forward in efficient sequence modeling. While the addition of the Liquid kernel introduces a slight computational overhead, the gains in accuracy and parameter efficiency are substantial. Future research will explore non-linear variants of the liquid kernel and its application to large-scale multi-modal tasks where temporal causality is paramount.

Summary updated

Abstract

A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structural state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.
BibTeX
@inproceedings{hasani2023liquid,
  author = {Ramin M. Hasani and Mathias Lechner and Tsun-Hsuan Wang and Makram Chahine and Alexander Amini and Daniela Rus},
  title = {{Liquid Structural State-Space Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=g4OTKRKfS7R},
}
Figure