Energy-Tweedie: Score meets Score, Energy meets Energy

Abstract

This paper establishes a fundamental mathematical bridge between denoising theory (Tweedie’s formula) and probabilistic forecasting (scoring rules). By deriving a novel identity that connects the Stein score of a noisy distribution to the gradient of the Energy Score, the author generalizes generative modeling beyond the standard Gaussian paradigm. This framework enables the development of "heavy-tailed" diffusion models and provides a principled method for estimating noise parameters directly from posterior samples.

Introduction

In generative modeling, Tweedie’s formula serves as a cornerstone, particularly within the context of diffusion models. It provides an elegant relationship between the "clean" version of a data point and its version corrupted by Gaussian noise, linking the posterior mean directly to the gradient of the log-marginal—the Stein score.

However, the classical formulation of Tweedie’s identity is strictly "Gaussian-centric." When data is corrupted by heavy-tailed or anisotropic distributions, this relationship breaks down. Furthermore, a conceptual divide has long existed between Energy Models (distributions defined by potential functions) and Scoring Rules (metrics used to evaluate probabilistic forecasts). This paper seeks to unify these concepts, providing a mathematical bridge that accommodates a much broader class of noise distributions.

The Energy-Score Identity

The primary contribution of this work is the derivation of the Energy-Score identity. The theoretical development proceeds through several key stages:

1. Generalizing Tweedie: The author extends Tweedie’s identity to the Elliptical distribution family. In these models, the log-density is proportional to a potential function $V$ of a Mahalanobis distance, allowing for a more flexible representation of noise.
2. The Path-Derivative Approach: Utilizing path-derivative notation common in Variational Inference, the paper demonstrates that the Stein score of a noisy marginal is essentially the negative gradient of the expected potential of the noise.
3. Theorem 4.1: The core proof shows that for data noised with a Generalized Gaussian distribution, the Stein score is exactly proportional to the gradient of a parameter-matched Energy Score (a strictly proper scoring rule).

This identity is significant because it harmonizes two distinct meanings of "energy" (energy-based models vs. the energy scoring rule) and two meanings of "score" (Stein score vs. scoring rules).

Methodology and Estimation

To make this theory actionable, the author proposes a Monte-Carlo estimator for the score that requires only samples from a denoising posterior. This allows for "off-the-shelf" diffusion-style sampling even when the noising distribution is non-Gaussian.

One of the most compelling aspects of this generalized framework is the shift from "mean-seeking" to "median-seeking" behavior. Classical Gaussian denoising targets the mean of the posterior. By employing heavy-tailed noise (such as the Multivariate Laplace), the score field becomes median-seeking. This allows the model to target cluster centers more directly during the early stages of the generative process, rather than collapsing onto a general manifold or ring structure.

Experimental Validation: Eight Gaussians

The paper validates these concepts using the Eight Gaussians dataset, a standard 2D benchmark consisting of eight clusters arranged in a circle. This synthetic environment allows for clear visualization of how different score fields behave.

Key findings from the experiments include:

• Generalized Diffusion: While standard Gaussian diffusion focuses on the overall ring structure initially, the Generalized Gaussian version ($\beta = 1.4$) targets the specific mixture centers more aggressively.
• Accuracy: Quantitative analysis confirms that the proposed identity-based score estimator achieves high cosine similarity with the ground-truth score field, even at low noise levels ($\sigma$).
• Parameter Calibration: The identity allows for the learning of noise parameters ($\beta, \lambda, \Sigma$) by minimizing the discrepancy between different score estimators, effectively "learning" the noise distribution of a corrupted dataset.

Discussion and Future Work

While the Energy-Tweedie framework offers a robust generalization of denoising, it introduces new theoretical and computational challenges. The author notes that for non-Gaussian cases, the sampling procedure cannot be strictly classified as a "reverse-SDE" in the traditional sense, though it remains a valid generative process. Additionally, the Monte-Carlo approximation of the score is computationally more intensive than the single-point estimates used in standard diffusion models.

Future research will focus on scaling these heavy-tailed energy scores to high-dimensional data, such as image synthesis on ImageNet, and further exploring the robustness of these models against outliers in complex datasets.