Modified Dirichlet Distribution: Allowing Negative Parameters to Induce Stronger Sparsity

Kewei TuEMNLP 2016 · OpenReview

Introduction

In probabilistic modeling, particularly for Natural Language Processing (NLP), inducing sparsity is a critical objective. We often seek models that explain data using a minimal set of active components, such as topics in a document or rules in a grammar. The standard tool for this task is the Dirichlet distribution. When its parameters ($\alpha$) are set between 0 and 1, the distribution concentrates probability mass on a few dimensions, favoring sparse outcomes.

However, the Dirichlet distribution faces a fundamental "sparsity ceiling." Mathematically, its parameters must remain positive to ensure the normalization factor (the Beta function) does not diverge. In large-scale unsupervised learning, the likelihood of the data can easily overwhelm a weak positive prior, preventing the model from discovering the underlying sparse structure. This paper introduces the Modified Dirichlet (mDir) distribution, a simple yet powerful extension that allows for negative parameters to induce significantly stronger sparsity.

The Modified Dirichlet (mDir)

The mDir distribution overcomes the limitations of the standard Dirichlet through two primary modifications:

  1. The $\epsilon$ Constraint: To prevent the distribution from diverging when parameters are zero or negative, a small positive constant $\epsilon$ is introduced. Every element $x_i$ in the probability vector is constrained such that $x_i \ge \epsilon$.
  2. Negative Parameters: With the values bounded away from zero, the parameters $\alpha_i$ can safely take on negative values. A highly negative $\alpha$ acts as an aggressive sparsity prior, pushing variables toward the $\epsilon$ floor much more forcefully than a standard Dirichlet prior ever could.

Despite these changes, the mDir retains the crucial property of conjugacy to the multinomial distribution, making it a "drop-in" replacement for existing probabilistic models. Furthermore, the authors provide an efficient $O(n \log n)$ algorithm for finding the mode of the distribution, ensuring computational feasibility for high-dimensional tasks.

Experimental Results

The paper evaluates the mDir distribution on synthetic Gaussian mixtures and the benchmark Wall Street Journal (WSJ) corpus for unsupervised dependency parsing using the Dependency Model with Valence (DMV).

Unsupervised Dependency Parsing

On the DMV model, mDir achieved a directed parsing accuracy of approximately 0.63. This result is significant because it matches the performance of far more complex state-of-the-art models. The mDir achieves this simply by applying a superior prior to a basic model, demonstrating that better handling of sparsity can be as effective as complex feature engineering.

Sensitivity analysis of directed parsing accuracy with respect to the sparsity parameter $\alpha$ (left) and smoothing parameter $\epsilon$ (right). Negative $\alpha$ values significantly improve performance over standard Dirichlet priors.

Hyperparameter Sensitivity

The performance of mDir is governed by the interplay between $\alpha$ and $\epsilon$. While negative $\alpha$ drives sparsity, $\epsilon$ serves as a smoothing factor. This smoothing ensures that no component is ever truly pruned to zero, allowing the model to "recover" components during iterative learning if they were prematurely suppressed.

Parsing accuracy as a function of the smoothing parameter $\epsilon$. The performance exhibits an "inverted U-shape," where intermediate values balance uniformity and local optima avoidance.

As shown in the figures, the choice of $\epsilon$ is a balancing act. If $\epsilon$ is too large, the model becomes overly uniform; if it is too small, the model may get trapped in poor local optima early in training. The optimal performance typically occurs at intermediate values (e.g., $10^{-4}$), where the model maintains enough flexibility to discover the correct structure.

Conclusion

The Modified Dirichlet distribution provides a mathematically sound and computationally efficient way to break the sparsity ceiling of standard priors. By allowing negative parameters, it enables researchers to tune the strength of sparsity to match the scale of their data. Its success in unsupervised dependency parsing suggests it is a potent tool for any domain where discovering latent sparse structure is paramount.

Summary updated

BibTeX
@inproceedings{tu2016modified,
  author = {Kewei Tu},
  title = {{Modified Dirichlet Distribution: Allowing Negative Parameters to Induce Stronger Sparsity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  url = {https://openreview.net/forum?id=BybdqGM_-H},
}
Figure