Introduction
In machine learning, "fusion" (combining multiple models) typically yields the best results when the individual models are diverse—meaning they make different kinds of mistakes. In language modeling, researchers often combine traditional N-gram models with Neural Network Language Models (NNLMs) because their fundamental differences provide a significant "diversity gain."
However, attempting to ensemble multiple NNLMs together usually hits a wall. Even with different random initializations, NNLMs tend to be highly correlated (often with a correlation coefficient as high as 0.988), essentially learning the same internal representations. This paper addresses a core question: Can we explicitly force a neural network to learn diverse representations during training to maximize the benefits of model fusion?
The Diverse Embedding Neural Network (DENN) is a novel architecture designed to answer this by forcing models to learn multiple, distinct ways of representing word histories. By using a specialized loss function that penalizes similarity between embedding sub-spaces, DENN achieves significantly lower perplexity than standard neural language models and traditional ensembles.
Methods: The DENN Architecture
The authors introduce the Diverse Embedding Neural Network (DENNLM). Instead of projecting a word history into a single high-dimensional embedding space, the DENN projects it onto multiple, lower-dimensional sub-spaces.
The core innovations of this approach include:
- Architectural Sparsity: A DENN is mathematically equivalent to a standard NNLM but uses block-diagonal weight matrices. This structure means the model actually has fewer parameters than a dense NNLM of a comparable size, making it more efficient.
- Representational Diversity Score: To ensure the sub-spaces are genuinely different, the authors define a diversity metric based on the correlation of pairwise cosine angles between words across different embeddings. If two embedding spaces "see" the relationship between words similarly, they are penalized.
- Augmented Loss Function: The training objective is modified to include three parts:
- A mixture loss to ensure the combined (fused) prediction is accurate.
- An individual discriminative loss to ensure each sub-network is capable of predicting the next word on its own.
- A diversity term that explicitly penalizes similarity between the different embedding representations.
Datasets
The researchers evaluated their model on the Penn Treebank (UPenn) dataset, a standard benchmark in natural language processing. The training set consists of approximately 1 million N-grams with a vocabulary of 10,000 words. While small by today’s Large Language Model standards, it remains a well-studied benchmark for understanding the fundamental behavior of language model architectures.
Key Insights
- Diversity as an Objective: The most significant insight is that diversity shouldn't be left to chance (e.g., random initialization). By making diversity an explicit part of the loss function, the model is forced to explore different features of the linguistic data.
- The "Representational" View: Diversity is best enforced at the embedding layer. By ensuring that different sub-networks have different "geometric views" of the vocabulary, subsequent layers are naturally forced to process that information differently.
- Efficiency through Structure: Because the DENN uses block-structured matrices, it achieves better performance than a standard NNLM while using fewer parameters. This suggests that "dense" isn't always "better" if the density leads to redundant learning.
- Correlation vs. Gain: The paper demonstrates a strong negative correlation (-0.97) between the similarity of model outputs and the gain achieved through fusion. The less correlated the models, the higher the performance jump when they are combined.
Key Results
The primary metric used to evaluate the models is perplexity (where lower is better).
As shown in the results, explicit diversity training yields substantial improvements:
- Standard NNLM (1 NN): 137.32
- Randomly Initialized Ensemble (4 NN-RandInit): 134.85
- 4-component DENN: 122.69 (a ~13.6% improvement over the single NNLM).
- 8-component DENN: 116.41 (an ~18.0% improvement).
Furthermore, when interpolated with a traditional 4-gram model, the 8-component DENN reached a perplexity of 109.32, significantly outperforming the standard NNLM + 4-gram combination (116.33).
Limitations and Future Work
- Computational Complexity: Calculating the diversity score across the entire vocabulary is computationally expensive. The authors mitigated this by sampling 500 random words per minibatch, which provided a good approximation without massive overhead.
- Scope of Architecture: The study focused primarily on feed-forward neural networks. While the authors suggest these principles apply to Recurrent Neural Networks (RNNs) and LSTMs, extensive experimental data for those architectures was not provided in this specific paper.
- Scaling: Because the paper uses the relatively small Penn Treebank, future work is needed to determine if these diversity constraints remain effective or become too difficult to optimize as vocabulary sizes scale to hundreds of thousands or millions of words.
Summary updated