This research introduces a novel encoding method called Fixed-size Ordinally-Forgetting Encoding (FOFE), designed to bridge the gap between simple feedforward neural networks (FNNs) and complex recurrent neural networks (RNNs) for language processing. FOFE allows feedforward networks to capture long-term dependencies in text without the training difficulties typically associated with recurrent architectures.
The Challenge: Memory vs. Complexity
In Neural Network Language Models (NNLMs), researchers traditionally face a trade-off between two architectures:
- Feedforward Neural Networks (FNNs): These are computationally efficient and easy to train but suffer from a "short memory," typically limited to a fixed window of previous words (e.g., n-grams).
- Recurrent Neural Networks (RNNs): These possess an internal state that can theoretically track dependencies across an entire document. However, they are notoriously difficult to train due to the vanishing gradient problem and the sequential nature of Back-propagation Through Time (BPTT).
FOFE aims to provide FNNs with an "infinite" memory similar to RNNs while maintaining the architectural simplicity and training speed of feedforward models.
Methodology: Fixed-size Ordinally-Forgetting Encoding
The core innovation is the FOFE mechanism, which maps variable-length word sequences into unique, fixed-size vectors. Unlike standard "Bag of Words" approaches, FOFE preserves word order by applying a forgetting factor ($\alpha$).
Given a sequence of words represented as one-hot vectors $e_t$, the FOFE encoding $z_t$ is calculated recursively:
$$z_t = \alpha \cdot z_{t-1} + e_t$$
Where $\alpha$ is a constant between 0 and 1. Because older words are multiplied by $\alpha$ more times than newer words, their influence "fades" over time, yet their relative position is mathematically preserved. This allows a single fixed-size vector to represent the entire history of a sentence. The authors demonstrate that if $\alpha \le 0.5$, the FOFE code is strictly unique; for $0.5 < \alpha < 1$, it remains "almost unique," with collisions being practically non-existent in natural language data.
Experimental Setup and Results
The authors evaluated FOFE on two major benchmarks: the Penn Treebank (PTB) (1M words) and the Large Text Compression Benchmark (LTCB) (160M words).
One of the primary advantages of FOFE-FNNLM is its efficiency on modern hardware. Because the model is feedforward, the encoding process can be flattened into matrix multiplications, avoiding the sequential bottlenecks of RNNs.
As shown in the results for the LTC dataset, the FOFE-FNNLM consistently outperformed both standard FNNs and RNNs. Specifically, a 2nd-order FOFE-FNNLM achieved a test perplexity of 107, surpassing the best RNN-LM (112) and standard n-gram FNNs. On the PTB corpus, the model achieved a perplexity of 108, outperforming high-performing LSTM models (117) and standard RNNs (123).
Conclusion
FOFE provides a powerful alternative to recurrent architectures by enabling feedforward networks to handle variable-length context effectively. By avoiding BPTT, FOFE-based models are significantly easier and faster to train while achieving state-of-the-art results in language modeling. Future work may explore applying FOFE to other NLP tasks such as machine translation and question answering, where capturing long-range dependencies is equally critical.
Summary updated