In the landscape of language modeling, a persistent trade-off exists between Feedforward Neural Networks (FNNs) and Recurrent Neural Networks (RNNs/LSTMs). While FNNs are computationally efficient and easily parallelizable, they traditionally struggle with long-distance dependencies due to their fixed input windows. Conversely, LSTMs capture long-term context but are notoriously expensive to train and difficult to scale.
This research introduces Dual-FOFE, an evolution of the Fixed-size Ordinally Forgetting Encoding (FOFE) method. By utilizing multiple forgetting factors simultaneously, Dual-FOFE allows simple feedforward architectures to achieve performance competitive with complex LSTMs at a fraction of the computational cost.
The Dual-FOFE Architecture
The original FOFE mechanism encodes a variable-length sequence into a single fixed-size vector recursively: $z_t = \alpha \cdot z_{t-1} + e_t$, where $e_t$ represents the one-hot vector of the current word. This creates a unique "fingerprint" for any sequence. However, the choice of the forgetting factor ($\alpha$) creates a dilemma: a small $\alpha$ excels at capturing local word order, while a large $\alpha$ is necessary to preserve long-term context.
Dual-FOFE resolves this by generating two separate FOFE codes for the same sequence using different $\alpha$ values (e.g., 0.5 and 0.9). These encodings are concatenated and fed into a standard FNN. This specialization allows one branch of the network to focus on precise local positioning while the other maintains a broader thematic memory of the past text.
Beyond the dual encoding, the authors explore higher-order FOFE, where the network receives not just the current state $z_t$, but also previous states like $z_{t-1}$ and $z_{t-2}$. This further enhances the model's ability to disambiguate nearby word positions without increasing the complexity of the underlying recurrent mechanism.
Experimental Evaluation
The authors evaluated Dual-FOFE on two major benchmarks: the enwiki9 dataset (1 billion bytes of Wikipedia) and the massive Google Billion Words (GBW) corpus. To manage the large vocabularies involved (up to 800,000 words), they employed Noise Contrastive Estimation (NCE), bypassing the computational bottleneck of a full softmax layer.
On the enwiki9 dataset, Dual-FOFE achieved a 10% relative improvement in perplexity over the original single-factor FOFE model. The results on the Google Billion Words task were even more significant, demonstrating that Dual-FOFE can outperform established LSTM baselines.
As shown in the results table, the Dual-FOFE model achieved a perplexity of 39.0, surpassing several large LSTM models that scored between 43.7 and 48.2. Most notably, Dual-FOFE reached this level of performance using only 1 GPU, whereas the LSTM models it outperformed required 40 GPUs for training. This represents a massive leap in hardware efficiency for large-scale language modeling.
Conclusion and Future Directions
Dual-FOFE demonstrates that with an intelligent encoding strategy, simple feedforward networks can match or exceed the performance of recurrent architectures. By specializing different branches of the input to handle different temporal scales, the model avoids the "Goldilocks" problem of choosing a single compromise hyperparameter.
While the current work uses fixed hyperparameters for $\alpha$, future research could explore making these factors learnable parameters. Additionally, while Dual-FOFE proves to be a powerful alternative to LSTMs, further comparison against modern Transformer architectures—which also leverage parallelizable feedforward structures—would provide a more complete picture of its place in the current NLP ecosystem.
Summary updated