Introduction
Neural Network Language Models (NNLMs) offer superior accuracy compared to traditional n-gram models but suffer from significant computational overhead at runtime. In latency-sensitive applications like Machine Translation (MT) or Speech Recognition (ASR), where tens of thousands of lookups are required per sentence, the slow matrix-vector multiplications of deep networks become a bottleneck. While a "pre-computation trick" exists for 1-layer networks—replacing calculations with table lookups—it has historically failed to scale to deeper, more accurate architectures because intermediate layer outputs change dynamically.
The Lateral Network Architecture
To bridge the gap between the speed of shallow models and the accuracy of deep models, this research introduces the Lateral Network. Unlike standard "stacked" architectures where layers are sequential, the lateral architecture places hidden layers in parallel.
In this design, both hidden layers receive input directly from the fixed word embedding layer. Their outputs are combined using an element-wise function—such as addition ($C_{add}$), multiplication ($C_{mul}$), or maximum ($C_{max}$)—before proceeding to the output layer. Because each lateral layer connects directly to static embeddings, the dot products for the entire vocabulary can be pre-calculated once after training.
Pre-computation and Runtime Efficiency
The primary advantage of the lateral structure is its compatibility with pre-computation. At test time, the model bypasses expensive matrix-vector multiplications. Instead, it performs a handful of simple vector lookups and additions. This transforms the computational complexity from $O(H \cdot V)$ (where $H$ is hidden size and $V$ is vocabulary size) to a constant-time table lookup.
While this approach increases memory usage—a 2-lateral network requires approximately 136 MB of RAM compared to 25 MB for a non-precomputed version—it remains significantly more efficient than traditional n-gram models, which often require several gigabytes of storage.
Experimental Results
The models were evaluated on language modeling (120M words from the New York Times) and English-German machine translation (600M words).
Accuracy and Perplexity
The lateral architecture demonstrated superior accuracy over stacked counterparts. A 3-layer lateral model using multiplicative combination ($C_{mul}$) achieved a perplexity of 71.1, outperforming both the 1-layer baseline (77.9) and the 3-layer stacked model (74.8). The success of $C_{mul}$ suggests that multiplication effectively captures complex, non-linear interactions between input features.
Runtime Performance
The most striking result is the gain in execution speed. By utilizing the pre-computation trick, the lateral network achieves performance levels comparable to highly optimized traditional models like KenLM.
As shown in the performance data, a pre-computed 2-lateral multiplicative model achieves 305,000 lookups per second (LPS) on a single CPU core. This represents a massive improvement over a standard 2-stacked network, which manages only 24,000 LPS—a speedup of over 12x.
Conclusion
The lateral network architecture successfully enables deep learning accuracy with the runtime efficiency of shallow models. By restructuring hidden layers to allow for pre-computation, researchers can deploy high-performance NNLMs in real-time environments without the typical latency penalties. Future work may explore scaling these architectures further or applying lateral structures to other embedding-based models like the Deep Structured Semantic Model (DSSM).
Summary updated