TL;DR
This paper introduces ETSMLP, a novel sequence modeling architecture that integrates Exponential Smoothing (ETS) into a standard Multi-Layer Perceptron (MLP). By leveraging the simplicity of ETS and adding learnable parameters, ETSMLP achieves performance comparable to state-of-the-art models like S4 on long-range sequence benchmarks, while being more computationally efficient.
What Problem Does This Paper Address?
Modeling long-range dependencies in sequential data is a fundamental challenge in machine learning, particularly for applications like natural language processing and time series forecasting. Traditional Transformer models, while powerful, suffer from quadratic computational and memory complexity due to their self-attention mechanism, making them impractical for very long sequences. While models like S4 have shown promise in addressing this by using State Space Models (SSMs), their complex parameterization and initialization schemes (e.g., HiPPO) make them difficult to understand, implement, and analyze. This paper aims to bridge this gap by proposing a simpler, yet effective, SSM-based architecture that is easier to work with and more efficient.
Methods
The core innovation of this paper is the ETSMLP (Exponential Smoothing Multi-Layer Perceptron) model. Instead of starting with complex continuous-time SSMs like S4, ETSMLP begins with a discrete-time SSM: Exponential Smoothing (ETS). The authors propose a Complex Exponential Smoothing (CES) module that enhances ETS by introducing learnable parameters:
- Learnable Damped Factors ($\alpha, \beta$): These parameters control the learning rate and the influence of past observations and current inputs, respectively.
- Complex Parameters: Treating $\alpha$ and $\lambda$ as complex numbers allows the model to capture richer information.
- Exponential Parameterization and Constraint Function: To ensure stable training, the authors propose an exponential transformation for $\lambda$ and a constraint function to keep its values within a feasible range.
- Shortcut Connection (Gating Mechanism): A residual connection with a gating mechanism is added to allow the model to bypass the CES module if needed, enhancing its flexibility.
The CES module is integrated into a standard MLP architecture, typically before the activation function, transforming a channel-only MLP into a sequence learner. The paper also explores a bidirectional version of ETSMLP by combining forward and backward kernels.
Datasets
The paper evaluates ETSMLP on the following datasets:
- Long Range Arena (LRA): A benchmark for long-range sequence modeling tasks, including ListOps, Text, Retrieval, Image, Pathfinder, and Path-X.
- Natural Language Understanding (NLU) Datasets: Including sentence classification (CoLA, SST-2, IMDB), sentence similarity (QQP, MRPC), and natural language inference (MNLI, QNLI).
Key Insights and Ideas
- Simplicity is Powerful: The paper demonstrates that a simple SSM like ETS, when enhanced with learnable parameters and integrated into an MLP, can achieve competitive performance with much more complex models like S4. This suggests that the success of SSMs might not solely depend on intricate parameterizations but also on their fundamental ability to model sequences.
- ETS as a Foundation: By starting with ETS, the authors bypass the complex mathematical derivations and initialization schemes often associated with continuous-time SSMs, making their approach more accessible and easier to implement.
- Learnable Parameters are Crucial: The introduction of learnable damped factors ($\alpha, \beta$) and the use of complex parameters significantly improve performance, highlighting their importance in capturing sequence dynamics.
- Stable Training is Achievable: The proposed exponential parameterization and constraint function are key to enabling stable training of the ETS module, overcoming the gradient explosion issues that can arise with direct parameterization.
- Efficiency Gains: ETSMLP offers significant advantages in terms of computational speed and memory usage compared to Transformers, especially for long sequences, making it a practical alternative for resource-constrained environments.
Key Results
- LRA Benchmark: ETSMLP achieved comparable performance to S4 on average (83.09 vs. 80.48) and significantly outperformed Transformer variants by about 20 points. It also slightly surpassed S4 and DSS on average by 2.61 points.
- NLU Datasets: ETSMLP achieved comparable or superior performance to Transformer encoders on various NLU tasks, demonstrating its versatility.
- Efficiency: ETSMLP consistently achieved higher Words Per Second (WPS) than both Transformer and S4 across various sequence lengths. Its memory usage grew linearly with sequence length, in contrast to the Transformer's quadratic growth.
- Ablation Studies: Removing learnable parameters ($\alpha, \beta, \omega$) or using real instead of complex fields significantly degraded performance, underscoring the importance of these components. Different initialization strategies (Stable vs. Ring) showed consistent performance across rings, but fixed points were less effective.
Limitations and Future Work
- Dataset Size: The current experiments were conducted on datasets with fewer than 100,000 samples. The authors plan to conduct experiments on pretraining, which will require meticulous adjustments to the exponential smoothing and architecture design, potentially drawing inspiration from models like Mamba.
- Constraint Function Design: The empirical design of the constraint function for $\lambda$ is noted as a limitation. The authors believe that a low granularity parameterization could mitigate issues like NaN results during backpropagation when $\lambda$ approaches the precision limits of floating-point numbers.
- Future Work: The authors suggest that their CES module can be integrated into other models to enhance their sequence learning capabilities. They also aim to establish a smoother training process on the parameter space.
Summary updated