Experimenting with Power Divergences for Language Modeling

Matthieu Labeau, Shay B. Cohen

Introduction

Standard neural language models (LMs) are typically trained using Maximum Likelihood Estimation (MLE), which is equivalent to minimizing the Kullback-Leibler (KL) divergence between the empirical data distribution and the model's predicted distribution. However, natural language is characterized by extreme frequency imbalances, famously described by Zipf’s law: a handful of common words (e.g., "the", "of") appear with massive frequency, while a "long tail" of rare words appears very infrequently.

MLE treats discrepancies across all tokens uniformly. Consequently, models often become highly proficient at predicting common function words but struggle significantly with rare, content-rich words that carry the most semantic weight. This paper investigates replacing the standard KL divergence with a broader family of power divergences—specifically $\alpha, \beta, \text{ and } \gamma$ divergences—to better navigate this frequency imbalance and allow researchers to prioritize learning on specific parts of the vocabulary.

Methodology: The Power Divergence Family

The authors explore three generalizations of the KL divergence, each offering unique properties for language modeling:

  1. $\alpha$-divergences: These control how the model distributes probability mass. Values of $\alpha > 1$ encourage better generalization across the distribution, whereas $\alpha < 1$ tends to make the model focus more on specific training instances.
  2. $\beta$-divergences: Frequently used in robust statistics to mitigate the influence of outliers. In the context of language, low-frequency tokens can be viewed as "outliers." Setting $\beta < 1$ can effectively increase the relative weight of these rare tokens during the training process.
  3. $\gamma$-divergences: A scale-invariant variant of the $\beta$-divergence. Its scale-invariance is particularly advantageous for training unnormalized models, as it does not require the output distribution to sum to one at every iteration, reducing computational overhead.

Beyond exact softmax training, the researchers derived objective functions for Approximated Softmax (AS) and Noise Contrastive Estimation (NCE), ensuring these power divergences remain compatible with computationally efficient sampling methods.

Experimental Setup

The proposed methods were evaluated on two benchmark datasets:

  • Penn Treebank (PTB): Tested on both the standard 10k vocabulary and a "full" version (~39k words) to specifically measure performance on the long tail.
  • Wikitext-2 (WT2): A larger dataset (~33k vocabulary) that preserves more natural linguistic complexity than the pruned PTB.

Key Results

The core finding is that power divergences allow for a controlled trade-off between common and rare word performance. By adjusting parameters like $\gamma$, it is possible to "gear" the model to prioritize the tail of the distribution.

Validation cross-entropy for low-frequency tokens vs. complete vocabulary across different power divergences.

As shown in the figure above, using parameters such as $\alpha=1.1$ or $\gamma=1.1$ significantly reduces the validation cross-entropy for low-frequency word bins compared to the standard KL baseline. While this sometimes comes at a slight cost to the overall vocabulary cross-entropy, the gains on rare words are substantial.

Furthermore, the study revealed a surprising synergy with sampling-based methods. In NCE training on Wikitext-2, an $\alpha = 1.1$ objective improved validation perplexity by approximately 1.1 points over the standard MLE-based NCE. This suggests that the inherent robustness of power divergences helps mitigate the noise introduced by sampling approximations.

Discussion and Limitations

A significant hurdle in adopting these divergences is the "Perplexity Bias." Perplexity, the standard metric for LMs, is mathematically tied to MLE. A model optimized for a power divergence might show "worse" perplexity even if it captures the underlying distribution of rare words more accurately. To address this, the authors advocate for "counterpart" metrics that evaluate models on their specific optimization objectives.

Additionally, hyperparameter sensitivity remains a challenge. The optimal values for $\alpha, \beta, \text{ or } \gamma$ vary depending on vocabulary size and the chosen sampling method, often necessitating extensive grid searches.

Conclusion

This research demonstrates that moving beyond the standard KL divergence offers a powerful mechanism for addressing the "rare word problem" in language modeling. By leveraging power divergences, models can be tuned to better represent the long tail of natural language, particularly when using efficient sampling-based training. Future work should investigate whether these intrinsic improvements translate to better performance in downstream applications like machine translation or summarization, where rare words often hold the key to meaning.

Summary updated

BibTeX
@inproceedings{labeau2019experimenting,
  author = {Matthieu Labeau and Shay B. Cohen},
  title = {{Experimenting with Power Divergences for Language Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1421},
}
Figure