Introduction
In probabilistic modeling, particularly for Natural Language Processing (NLP), inducing sparsity is a critical objective. We often seek models that explain data using a minimal set of active components, such as topics in a document or rules in a grammar. The standard tool for this task is the Dirichlet distribution. When its parameters ($\alpha$) are set between 0 and 1, the distribution concentrates probability mass on a few dimensions, favoring sparse outcomes.
However, the Dirichlet distribution faces a fundamental "sparsity ceiling." Mathematically, its parameters must remain positive to ensure the normalization factor (the Beta function) does not diverge. In large-scale unsupervised learning, the likelihood of the data can easily overwhelm a weak positive prior, preventing the model from discovering the underlying sparse structure. This paper introduces the Modified Dirichlet (mDir) distribution, a simple yet powerful extension that allows for negative parameters to induce significantly stronger sparsity.
The Modified Dirichlet (mDir)
The mDir distribution overcomes the limitations of the standard Dirichlet through two primary modifications:
- The $\epsilon$ Constraint: To prevent the distribution from diverging when parameters are zero or negative, a small positive constant $\epsilon$ is introduced. Every element $x_i$ in the probability vector is constrained such that $x_i \ge \epsilon$.
- Negative Parameters: With the values bounded away from zero, the parameters $\alpha_i$ can safely take on negative values. A highly negative $\alpha$ acts as an aggressive sparsity prior, pushing variables toward the $\epsilon$ floor much more forcefully than a standard Dirichlet prior ever could.
Despite these changes, the mDir retains the crucial property of conjugacy to the multinomial distribution, making it a "drop-in" replacement for existing probabilistic models. Furthermore, the authors provide an efficient $O(n \log n)$ algorithm for finding the mode of the distribution, ensuring computational feasibility for high-dimensional tasks.
Experimental Results
The paper evaluates the mDir distribution on synthetic Gaussian mixtures and the benchmark Wall Street Journal (WSJ) corpus for unsupervised dependency parsing using the Dependency Model with Valence (DMV).
Unsupervised Dependency Parsing
On the DMV model, mDir achieved a directed parsing accuracy of approximately 0.63. This result is significant because it matches the performance of far more complex state-of-the-art models. The mDir achieves this simply by applying a superior prior to a basic model, demonstrating that better handling of sparsity can be as effective as complex feature engineering.
Hyperparameter Sensitivity
The performance of mDir is governed by the interplay between $\alpha$ and $\epsilon$. While negative $\alpha$ drives sparsity, $\epsilon$ serves as a smoothing factor. This smoothing ensures that no component is ever truly pruned to zero, allowing the model to "recover" components during iterative learning if they were prematurely suppressed.
As shown in the figures, the choice of $\epsilon$ is a balancing act. If $\epsilon$ is too large, the model becomes overly uniform; if it is too small, the model may get trapped in poor local optima early in training. The optimal performance typically occurs at intermediate values (e.g., $10^{-4}$), where the model maintains enough flexibility to discover the correct structure.
Conclusion
The Modified Dirichlet distribution provides a mathematically sound and computationally efficient way to break the sparsity ceiling of standard priors. By allowing negative parameters, it enables researchers to tune the strength of sparsity to match the scale of their data. Its success in unsupervised dependency parsing suggests it is a potent tool for any domain where discovering latent sparse structure is paramount.
Summary updated