Abstract
Distributional approximation is a fundamental problem in machine learning with numerous applications across all fields of science and engineering and beyond. The key challenge in most approximation methods is the need to tackle the intractable normalization constant pertaining to the parametrized distributions used to model the data. In this paper, we present a novel Stein operator on Lie groups leading to a kernel Stein discrepancy (KSD) which is a normalization-free loss function. We present several theoretical results characterizing the properties of this new KSD on Lie groups and its minimizers namely, the minimum KSD estimator (MKSDE). Proof of several properties of MKSDE are presented, including strong consistency, CLT and a closed form of the MKSDE for the von Mises-Fisher distribution on SO(N). Finally, we present experimental evidence depicting advantages of minimizing KSD over maximum likelihood estimation.
Introduction
In machine learning and statistics, fitting probability distributions to data is a core task. While Maximum Likelihood Estimation (MLE) is the standard approach, it faces significant hurdles when applied to non-Euclidean spaces like Lie groups. Lie groups, such as the special orthogonal group $SO(3)$ (representing 3D rotations) or the special Euclidean group $SE(3)$ (representing rigid body motions), are essential for modeling physical systems, robotics, and computer vision.
The primary difficulty lies in the "normalization constant"—the value that ensures a probability density integrates to one. On Lie groups, these constants often involve complex integrals or hypergeometric functions that are computationally intractable. Consequently, researchers using MLE must rely on numerical approximations that introduce a trade-off between computational cost and estimation accuracy. This paper introduces a framework to bypass these constants entirely using a novel Stein operator.
Methodology: The Stein Operator and KSD
The authors propose a Kernel Stein Discrepancy (KSD) specifically designed for the geometry of Lie groups. The core of this approach is a new Stein operator that leverages the algebraic structure of these spaces, specifically using left-invariant bases. Unlike previous manifold-based KSDs, this method avoids local coordinate charts, which can become unstable at the boundaries of a space.
By utilizing the "Stein identity," the expectation of this operator under the target distribution is zero. This property allows for the construction of a discrepancy measure—a loss function—that depends only on the gradient of the log-density. Because the normalization constant is a scalar that disappears during differentiation, the resulting loss function is "normalization-free."
From this foundation, the authors derive the Minimum Kernel Stein Discrepancy Estimator (MKSDE). They provide theoretical proofs for its strong consistency (ensuring it converges to the true parameters as data increases) and demonstrate that it follows a Central Limit Theorem. For standard distributions like the von Mises-Fisher (vMF) and Riemannian normal distributions on $SO(N)$, the authors even provide closed-form solutions for the estimator, significantly reducing computational overhead.
Experimental Evaluation
The proposed MKSDE was evaluated against MLE using both synthetic and real-world datasets:
- Synthetic Data: Testing on samples from von Mises-Fisher and Cayley distributions showed that MKSDE is highly robust. While MLE performance degrades as distributions become more "concentrated" (due to the failure of numerical approximations for hypergeometric functions), MKSDE remains stable and accurate.
- ModelNet10 Orientation Task: Using a database of CAD models, the authors simulated 3D object orientation estimation. MKSDE consistently achieved lower geodesic distances between the estimated mode and the true orientation compared to MLE across various object categories like chairs and sofas.
Key Insights and Results
One of the most significant findings is the practical failure of MLE in highly concentrated scenarios. In $SO(3)$ rotation estimation, MKSDE consistently achieved lower Frobenius distances to the ground truth. The geometric invariance of the method—ensured by the use of left-invariant vector fields—means the KSD is intrinsic to the group's geometry and independent of specific coordinate mappings.
Furthermore, the framework enables "one-shot" goodness-of-fit testing. Researchers can determine if a dataset matches an entire family of distributions (e.g., "is this data distributed according to a von Mises-Fisher model?") without needing to specify or estimate the parameters first.
Conclusion and Future Work
This work provides a robust, theoretically grounded alternative to MLE for distributions on Lie groups. By eliminating the need for normalization constants, MKSDE offers a more stable and often more accurate estimation method, particularly for concentrated distributions where traditional methods struggle.
Future research directions include exploring more "geometry-aware" kernels, as the choice of kernel remains a critical hyperparameter. Additionally, while the current focus is on compact groups like $SO(N)$, extending these practical applications to non-compact groups like the affine group or $SL(N)$ presents an exciting avenue for further exploration.
Summary updated