In machine learning, modeling a complex probability distribution $p(x)$ often runs into a fundamental mathematical wall: the partition function. While we can easily design a neural network to output an unnormalized energy or score $\tilde{p}(x)$, turning this into a valid probability distribution requires dividing by a normalizing constant $Z = \sum_x \tilde{p}(x)$ (or $\int \tilde{p}(x)dx$). For high-dimensional data, calculating $Z$ is completely intractable because it requires summing or integrating over an exponentially large space.
In continuous domains, score matching elegantly sidesteps this problem. Instead of modeling the probability density directly, we model its score function—the gradient of the log-density with respect to the data, $\nabla_x \log p(x)$. Because $\nabla_x \log p(x) = \nabla_x \log \tilde{p}(x) - \nabla_x \log Z$, and the derivative of the constant $\log Z$ is zero, the intractable partition function completely vanishes.
But what happens when our data is discrete, such as text tokens, DNA sequences, or binary graphs? In these spaces, gradients do not exist. Moving from the word "cat" to "dog" is a discrete jump, not a continuous slide. Without gradients, standard score matching breaks down.
To solve this, a powerful framework has emerged: discrete score matching. By replacing continuous gradients with localized probability ratios, this approach brings the benefits of partition-function-free learning and diffusion models to discrete worlds.
The Core Insight: Probability Ratios as Discrete Scores
To eliminate the partition function without gradients, we must find an alternative mathematical operator that isolates the unnormalized density from its normalizing constant. The solution lies in looking at the ratio of probabilities between a state $x$ and its neighboring states $x'$.
Consider the ratio of the probability of a sequence $x$ to a slightly perturbed version $x'$:
$$\frac{p(x')}{p(x)} = \frac{\tilde{p}(x') / Z}{\tilde{p}(x) / Z} = \frac{\tilde{p}(x')}{\tilde{p}(x)}$$
Just like the continuous derivative, this ratio causes the partition function $Z$ to cancel out perfectly. We can compute and match this ratio using only our unnormalized model.
The foundational theory for this perspective dates back to Lyu (2009), who demonstrated that score matching could be generalized to discrete spaces by replacing the gradient operator with general linear operators that preserve local density relationships. More recently, Meng et al. (2022) formalized this via Concrete Score Matching (CSM). They defined a "Concrete score" based on the relative rate of change in probability between a data point and its defined neighbors. Crucially, they proved a beautiful geometric connection: when you embed a discrete space into a Euclidean framework and shrink the neighborhood distance toward zero, this discrete probability ratio converges exactly to the standard continuous gradient of the log-density.
Overcoming Instability with Bregman Divergences
While minimizing the difference between a model's predicted probability ratios and the true data ratios sounds straightforward, optimizing it in practice is notoriously difficult.
If you train a neural network using a standard mean squared error (L2) loss to match these ratios, the training process frequently diverges. Ratios are inherently asymmetric and unconstrained; they can explode toward infinity or drop close to zero. Even worse, an unconstrained neural network might predict a negative ratio, which is physically impossible for probabilities.
To resolve this numerical instability, Lou et al. (2023) and Lou et al. (2024) introduced Score Entropy Discrete Diffusion (SEDD), which utilizes a loss function derived from Bregman divergence (specifically, generalized I-divergence).
The magic of the Bregman divergence loss is that it introduces a natural log-barrier structure. If the neural network attempts to predict a ratio that approaches zero or goes negative, the log-barrier penalizes the model with an infinite loss. This mathematical constraint forces the network's predictions to remain strictly positive and stable throughout training. As shown by Zhang et al. (2025) and Srikanth et al. (2025), measuring error via Bregman divergence aligns perfectly with the intrinsic geometry of discrete state transitions, providing rigorous convergence guarantees that mirror those of continuous models.
Alternatively, Pham et al. (2025) proposed a different solution to the variance problem: instead of estimating the raw probability ratios directly, they re-engineered the objective as an L2 regression over a conditional expectation of the forward process. This formulation bypasses ratio estimation entirely, eliminating high-variance updates while maintaining a stable training trajectory.
Powering Discrete Diffusion Models
The most impactful application of discrete score matching is in generative modeling, specifically discrete diffusion.
In continuous diffusion models, data is gradually degraded by adding Gaussian noise, described by a Stochastic Differential Equation (SDE). To reverse this process and generate data, a network learns to predict the continuous score function at every time step.
In discrete diffusion, we replace Gaussian noise with a Continuous Time Markov Chain (CTMC) or a stochastic jump process. Instead of drifting smoothly, a data point (like a sentence) sits in a state and then abruptly "jumps" to another state by swapping a token. For example, Winkler et al. (2024) established a formal bridge between these domains by utilizing the Ehrenfest process as a discrete analog to the continuous Ornstein-Uhlenbeck process.
To generate data, we must run this jump process in reverse. The transition rate for the reverse process depends directly on the ratio of the marginal probabilities of the states:
$$\text{Reverse Rate} \propto \frac{p(x')}{p(x)}$$
By training a neural network to estimate these categorical probability ratios using ratio matching objectives (Sun et al., 2023), we gain a mathematically sound way to guide the reverse jump process.
To scale this to massive vocabularies (like dictionaries of 30,000+ tokens), calculating ratios for every possible token transition is impossible. Instead, the training objective focuses on local perturbations—evaluating transitions only within a small, randomized subset of neighboring states (Lou et al., 2023). This localization makes discrete diffusion highly efficient, allowing models to match or exceed the performance of traditional autoregressive models on text generation and infilling tasks.
Furthermore, Zhang et al. (2025) unified these perspectives under the Target Concrete Score Matching (TCSM) framework. By shifting the training focus from denoising intermediate noisy states to directly matching the concrete score of the clean target data, they unlocked the "post-training" phase for discrete diffusion—enabling downstream fine-tuning, preference optimization, and distillation.
Mathematical Roots and Broader Horizons
Discrete score matching is not merely a heuristic adaptation of continuous equations; it is deeply rooted in Stein's method for probability approximations.
In continuous spaces, the Stein operator relies on derivatives. To characterize discrete distributions, Goldstein et al. (2013) and Ley et al. (2013) developed discrete Stein operators. These operators act on functions by measuring how the probability mass shifts when discrete states are swapped, serving as an integration-by-parts analogue for discrete settings.
This mathematical foundation allows researchers to use discrete scores for tasks far beyond generative modeling:
- Goodness-of-Fit Testing: Yang et al. (2018) utilized discrete Stein operators to construct the Kernelized Discrete Stein Discrepancy (KDSD). This allows researchers to test whether a dataset matches a complex discrete model without ever computing the model's partition function.
- Reinforcement Learning and Fine-Tuning: Zekri et al. (2025) introduced Score Entropy Policy Optimization (SEPO), which uses transition probability ratios to calculate differentiable policy gradients, enabling stable alignment of discrete diffusion models using black-box rewards.
- Information Estimation: Foresti et al. (2026) leveraged discrete score estimation to calculate Mutual Information and entropy directly on high-dimensional text and DNA sequences, bypassing the unstable embedding tricks required by traditional continuous estimators.
By reframing the score function from a vector of spatial gradients to a network of local probability ratios, discrete score matching provides a rigorous, stable, and unified language for analyzing and generating structured data.
Key Foundational Work
Interpretation and Generalization of Score Matching (2009) — contributed 3 concept elements to this direction.
Learning Bayesian Networks: A Unification for Discrete and Gaussian Domains (1995) — contributed 1 concept elements to this direction.