Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling

Mónika Farsang, Ramin Hasani, Radu GrosuES-FoMo-III @ ICML 2025 · OpenReview

TL;DR

This paper introduces LrcSSM, a novel nonlinear recurrent neural network that achieves the speed and parallelizability of linear state-space models (LSSMs) for long sequence processing. By enforcing a diagonal state-transition matrix, LrcSSM enables efficient parallel computation while retaining the expressive power of nonlinear dynamics, offering improved gradient stability and competitive performance on long-range forecasting tasks.

What Problem Does This Paper Address?

Traditional nonlinear recurrent neural networks (RNNs), despite their potential for capturing complex temporal dependencies, have largely been sidelined in favor of linear state-space models (LSSMs). This shift is primarily due to the inherent sequential nature of RNNs, which hinders efficient parallelization and scaling to long sequences. While recent advancements like DEER and ELK have shown promise in parallelizing nonlinear RNNs, they often rely on approximations or struggle with stability. The challenge is to bridge this gap by developing nonlinear models that are both expressive and as efficient and scalable as their linear counterparts for long sequence modeling.

Methods

The core innovation of LrcSSM lies in its adaptation of bio-inspired Liquid-Resistance Liquid-Capacitance (LRC) networks to achieve a diagonal Jacobian. This is achieved by:

  • Modifying LRC equations: The original LRC equations for synaptic conductance and membrane potential are split into state-dependent and input-dependent components. Crucially, cross-state synaptic parameters are zeroed out, and only self-loop parameters are retained for the state-dependent parts.
  • Diagonal Jacobian: This modification inherently results in a diagonal state-transition matrix (A) and a diagonal Jacobian, which is key for parallelization.
  • Parallel Prefix Scan: The diagonal structure allows the entire sequence to be processed in parallel using a single prefix-scan operation, similar to LSSMs.
  • Exact ELK Updates: Unlike previous methods that used approximations (e.g., taking the diagonal of the Jacobian), LrcSSM's diagonal structure allows for exact ELK updates, leading to improved stability and efficiency.

Datasets

The paper evaluates LrcSSM on several long-horizon sequence modeling tasks from the UEA Multivariate Time Series Classification Archive:

  • Long-Horizon:
    • EthanolConcentration
    • MotorImagery
    • EigenWorms
  • Short-Horizon (for ablation studies):
    • Heartbeat
    • SelfRegulationSCP1
    • SelfRegulationSCP2

Key Insights and Ideas

  • Bridging the Gap: LrcSSM successfully demonstrates that nonlinear RNNs can be made as efficient and scalable as linear SSMs for long sequences, challenging the notion that nonlinearity is inherently a bottleneck for parallel processing.
  • Diagonal Jacobian is Key: The paper highlights that forcing the Jacobian to be diagonal is a critical step for achieving both parallelizability and stability in nonlinear recurrent models.
  • Formal Gradient Stability: Unlike some other input-varying systems, LrcSSM offers a formal guarantee of gradient stability, which is crucial for training deep networks on long sequences.
  • Compute-Optimal Scaling: LrcSSM aligns with the compute-optimal scaling law regime, suggesting it's an efficient architecture for modern hardware.

Key Results

  • Competitive Performance: LrcSSM outperforms several state-of-the-art LSSMs, including LRU, S5, S6, and Mamba, on the EthanolConcentration benchmark. It also achieves strong performance on MotorImagery and EigenWorms.
  • Efficiency: LrcSSM processes long sequences as fast as linear state-space layers, with $\mathcal{O}(TD)$ time and memory complexity and $\mathcal{O}(\log T)$ sequential depth.
  • Gradient Stability: The model provides a formal gradient-stability guarantee.
  • Compute Efficiency: It follows the compute-optimal scaling law regime ($\beta \approx 0.42$), outperforming Transformers at equal compute.

Limitations and Future Work

  • Integration Scheme: The current implementation uses an explicit Euler integration scheme. The authors suggest that more sophisticated integration methods (e.g., implicit or hybrid schemes) could be investigated for potentially further improvements, especially since LinOSS-IM (which uses implicit integration) slightly outperforms LrcSSM on some tasks.
  • Input Correlations: For short-horizon tasks, LrcSSM's performance was average, and the authors suspect this might be due to a lack of interesting input correlations in those specific datasets.
  • Complex-Valued Parameters: Experiments with complex-valued parameters for certain components did not yield significant performance gains, but this avenue could be explored further.
  • Additional Time Dimension: The authors note that incorporating an additional time dimension as a hyperparameter, as done in some other models, could be investigated for LrcSSM.

Summary updated

Abstract

We present LrcSSM, a $\textit{nonlinear}$ recurrent model that processes long sequences as fast as today’s linear state-space layers. By forcing the state-transition matrix to be diagonal and learned at every step, the full sequence can be solved in parallel with a single prefix-scan, giving $\mathcal{O}(TD)$ time and memory and only $\mathcal{O}(\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Lastly, for network depth $L$, as the forward and backward passes cost $\Theta(T\,D\,L)$ FLOPs, with its low sequential depth and parameter count $\Theta(D\,L)$, the model follows the compute-optimal scaling law regime ($\beta \approx 0.42$) recently observed for Mamba, outperforming quadratic-attention Transformers at equal compute while avoiding the memory overhead of FFT-based long convolutions. We show that on a series of long-range forecasting tasks, LrcSSM outperforms LRU, S5 and Mamba.
BibTeX
@inproceedings{farsang2025scaling,
  author = {Mónika Farsang and Ramin Hasani and Radu Grosu},
  title = {{Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling}},
  booktitle = {ES-FoMo-III @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=3qRWlWtPTo},
}
Figure