For years, sequence modeling was synonymous with Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks. The intuition seemed unshakeable: because time flows sequentially, a model must process data step-by-step, maintaining an internal hidden state to carry information from the past into the future.
However, this recurrent paradigm introduces severe computational bottlenecks. Backpropagation through time (BPTT) is notoriously difficult to train due to vanishing and exploding gradients. More importantly, the step-by-step nature of recurrence prevents parallelization along the time dimension, creating an architectural bottleneck on modern hardware.
To solve this, a powerful alternative direction has emerged: non-recurrent, feedforward temporal modeling. By replacing recurrent hidden loops with learnable memory blocks, structural encodings, or windowed predictions, these architectures capture long-term dependencies while remaining entirely feedforward, stable, and parallelizable.
The Core Intuition: Replacing Feedback Loops with Filters
The foundational shift away from explicit hidden state recurrence dates back to early comparative studies. Bill G. Horne et al. demonstrated in An experimental comparison of recurrent neural networks that feedforward networks utilizing a fixed window of past inputs and outputs can frequently match or outperform networks with explicit hidden state recurrence. This suggested that explicit internal loops are often redundant if historical context can be captured effectively within a feedforward buffer.
Modern architectures realize this intuition through Feedforward Sequential Memory Networks (FSMNs). Instead of summarizing the past into a single hidden vector that updates step-by-step, an FSMN uses memory blocks that function similarly to non-recursive Finite Impulse Response (FIR) filters.
Recurrent Loop (Sequential):
[Input t] ---> [Hidden State t] ---> [Hidden State t+1]
^ |
|---------------------|
Feedforward Memory Block (Parallel):
[Input t-2] \
[Input t-1] -- [Learnable Filter / Weights] ---> [Context Vector t]
[Input t ] /
As described by Shiliang Zhang et al. in Deep-FSMN for Large Vocabulary Continuous Speech Recognition, these memory blocks aggregate information from a fixed window of past and future frames using learnable coefficients. Because the context vector at any time step is a direct, weighted sum of the surrounding window, the network completely bypasses the sequential dependency of RNNs. During training, the entire sequence can be processed simultaneously across GPU cores.
Scaling Context via Depth and Dilations
A naive window-based feedforward network faces a strict trade-off: to see further into the past, you must increase the window size, which causes the parameter count to explode. Non-recurrent temporal modeling circumvents this through hierarchical composition and dilations.
Deep and Pyramidal Structures
By stacking multiple feedforward memory layers and adding skip connections—a framework known as Deep-FSMN (DFSMN)—the effective receptive field grows with network depth. Heng Lu et al. demonstrated in Deep Feed-Forward Sequential Memory Networks for Speech Synthesis that these hierarchical look-back and look-ahead filters allow the network to capture long-term acoustic trajectories smoothly and efficiently.
To optimize this hierarchy, Miao Cao et al. proposed a pyramidal FSMN (pFSMN) in Pyramid Memory Block and Timestep Attention for Speech Emotion Recognition. By using progressively larger memory windows in deeper layers, shallower layers are forced to focus on fine-grained local details, while deeper layers compose these representations to capture abstract, long-range contextual information.
Dilated Memory Blocks
Another way to expand the temporal horizon without adding parameters is through dilations. In MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation, the authors incorporated dilated FSMN blocks into a speech separation architecture. By skipping frames within the memory block filter, the model expands its receptive field exponentially rather than linearly, allowing it to capture broad acoustic patterns while maintaining full parallelization.
Deterministic History: Fixed-Size Ordinally-Forgetting Encoding
While speech and audio tasks naturally map to sliding window filters, language modeling presents a different hurdle: text sequences are highly variable in length, and exact word order matters over long distances.
To adapt feedforward networks for language, Shiliang Zhang et al. introduced the Fixed-Size Ordinally-Forgetting Encoding (FOFE) in The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models. FOFE is a deterministic, recursive algorithm that compresses a variable-length sequence of words into a single, fixed-size vector before it ever reaches the neural network layers.
It achieves this by applying a historical forgetting factor ($\alpha$) to a continuous bag-of-words representation:
$$S_t = \alpha S_{t-1} + e_t$$
Where $e_t$ is the one-hot vector of the current word. Because $\alpha$ continuously discounts past states, words that appeared longer ago contribute less to the current vector values. Crucially, this encoding is completely unique and reversible; the exact word order can be mathematically decoded from the final fixed-size vector.
To capture both immediate local syntax and long-term semantic context, the authors expanded this in Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models by utilizing two distinct forgetting factors simultaneously. This allows a standard feedforward network to process text with high sensitivity to word order, outperforming significantly larger RNNs without ever executing a recurrent hidden step.
Eliminating Cumulative Drift in Predictions
Beyond training speed, feedforward temporal modeling solves a fundamental flaw in how recurrent models generate sequences: error accumulation.
When an RNN predicts a sequence across a long horizon, it typically generates data autoregressively—predicting step $t+1$ by feeding its own potentially flawed prediction from step $t$ back into the network. This causes a compounding drift, where small errors early in the sequence diffuse and degrade the structural integrity of later outputs.
Autoregressive Recurrence (Error Accumulation):
Step 1: Predict P1 -> Feed P1 back
Step 2: Predict P2 (contains Error 1) -> Feed P2 back
Step 3: Predict P3 (contains Error 1 + Error 2)
Simultaneous Feedforward Window:
[Past Context Window] ---> [Feedforward Network] ---> [Full Future Window (P1, P2, P3)]
Judith Bütepage et al. addressed this directly in Deep Representation Learning for Human Motion Prediction and Classification. By designing a feedforward temporal encoder that maps a past context window to an entire future sequence window simultaneously, the model avoids iterative feedback loops entirely. This global window-to-window mapping prevents error diffusion, allowing the model to maintain structural realism over long-term temporal horizons.
Efficiency at the Edge
Because non-recurrent architectures rely on standard matrix multiplications and linear projections rather than sequential state updates, they are highly hardware-efficient and easily compressed for edge deployment.
The authors of Compact Feedforward Sequential Memory Networks for Small-footprint Keyword Spotting introduced the Compact FSMN (cFSMN). By inserting low-rank linear projection layers before the memory blocks, they decoupled the model's hidden dimension from the memory block size. This architectural choice was further validated by the authors of Audio Tagging with Compact Feedforward Sequential Memory Network and Audio-to-Audio Ratio Based Data Augmentation, who achieved state-of-the-art audio tagging performance with up to 30 times fewer parameters than standard convolutional and recurrent architectures.
Whether applied to acoustic modeling, language processing, or specialized tasks like marine mammal classification (the authors), feedforward temporal modeling proves that explicit recurrence is not a requirement for understanding time. By restructuring how context is captured—via sliding filters, structural encodings, and parallel windows—these models achieve the performance of recurrent networks with the speed and stability of feedforward architectures.
Key Foundational Work
The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models (2015) — contributed 2 concept elements to this direction.
An experimental comparison of recurrent neural networks (1994) — contributed 1 concept elements to this direction.