Representation Learning for Equivariant Inference with Guarantees

Daniel Ordoñez-Apraez, Vladimir Kostić, Alek Fröhlich, Vivien Brandt, Karim Lounici, Massimiliano Pontil

TL;DR

The paper introduces Equivariant Neural Conditional Probability (eNCP), a novel framework that bridges geometric deep learning and operator theory to learn symmetry-respecting data representations. By exploiting structural symmetries, eNCP simultaneously tackles regression, conditional probability estimation, and uncertainty quantification while providing the first-of-its-kind non-asymptotic statistical learning guarantees.

What Problem Does This Paper Address?

Modeling how a target variable $y$ changes based on an observed variable $x$—formally known as estimating the conditional probability distribution $P(y|x)$—is a cornerstone of machine learning. It underpins critical tasks like risk assessment in finance, medical diagnosis, and robust robot control. However, doing this in high-dimensional spaces suffers from the curse of dimensionality unless guided by strong inductive biases.

While Geometric Deep Learning (GDL) has successfully leveraged physical and geometric symmetries (like rotation or reflection invariance) to build highly efficient neural network architectures, the field has largely focused on empirical success and application-specific designs. Until now, there has been a significant gap in our theoretical understanding: we lacked rigorous statistical learning guarantees showing exactly how and why enforcing these symmetries mathematically guarantees better generalization and sample efficiency.

Methods

The authors address this by connecting two previously isolated fields: spectral contrastive learning (which models conditional expectations using linear operators) and GDL.

Instead of directly regressing a target, eNCP approximates the spectral decomposition of the conditional expectation operator. Grounded in group representation theory, the framework forces the learned representations to be disentangled across independent symmetry quotient groups (known as isotypic subspaces).

As shown below, this structural constraint forces the underlying matrix operator to become block-diagonal:

Figure 2. Left: NCP’s bilinear NN architecture. Right: eNCP’s G-equivariant bilinear NN architecture, featuring ϕθ and ψθ as G-equivariant NNs and Eθ as a G-equivariant block-diagonal matrix. Each block is equivariant to a quotient group G(k) ⊆G and is constrained to have singular spaces of dimension at least dk—the minimal dimension for a representation of the action of G(k).

To train this architecture, the authors introduce a disentangled contrastive training loss derived from U-statistics, which independently regularizes the orthonormality of each symmetric subspace. Once trained, the single learned model can be used for standard regression or to reconstruct the full conditional Cumulative Distribution Function (cCDF) for uncertainty quantification.

Datasets

The framework was evaluated on both synthetic benchmarks and challenging real-world robotics tasks:

  1. Symmetric Conditional Gaussian Mixture Models (cGMM): A synthetic dataset extended by the authors to parametrically enforce symmetries under various finite groups (such as cyclic groups $C_2, C_6$, dihedral group $D_6$, and the icosahedral group $I_h$).
  2. Quadruped Robot Locomotion (Solo and AlienGo): Real-world and simulated robotics datasets where the goal is to predict highly non-linear, symmetry-governed physical properties—such as Center of Mass (CoM) momenta, Ground Reaction Forces (GRF), instantaneous work, and kinetic energy—from noisy proprioceptive sensor readings during locomotion over rough terrain.

Key Insights and Ideas

  • Symmetry Multiplies Sample Size: The core theoretical breakthrough shows that enforcing equivariant disentangled representations mathematically boosts the effective sample size from $N$ to $diso \cdot N$ (where $diso$ is the sum of the dimensions of the group's irreducible representations). This provides a clear explanation for the massive sample efficiency gains seen in practice.
  • Task-Agnostic Representations: Because eNCP learns the underlying conditional expectation operator rather than a direct mapping, the exact same learned features can be frozen and reused for downstream regression, quantile estimation, or rare-event modeling without retraining the backbone network.
  • Mitigating Rare-Event Bottlenecks: By introducing a "symmetry index," the authors prove that exploiting symmetry helps mitigate the difficulty of estimating rare events, especially when data is asymmetric locally but obeys global symmetry orbits.

Key Results

  • Superior Sample Efficiency: In quadruped regression tasks, eNCP achieved lower Mean Squared Error (MSE) with significantly fewer training samples compared to standard Multi-Layer Perceptrons (MLPs), Equivariant MLPs (eMLPs), and symmetry-agnostic NCP models.
  • Drastic Parameter Reduction: Due to weight sharing enforced by $G$-equivariance, the encoder networks in eNCP required up to 60x fewer trainable parameters (for the $I_h$ group) than their symmetry-agnostic counterparts, while converging roughly 2.5x faster.
  • Well-Calibrated Uncertainty Quantification: In predicting confidence intervals (CIs) for robot work and kinetic energy, eNCP was the only model capable of maintaining reliable, well-calibrated coverage during both nominal locomotion and sudden, transient physical disturbances.
Figure 1. Left: Test set sample efficiency for G-equivariant regression (MSE vs. training samples) when predicting the G-equivariant linear and angular momentum of a quadruped robot’s center of mass (CoM) from noisy joint positions and velocities. Right: Uncertainty quantification via G-equivariant prediction of 90% Confidence Intervals (CIs) (light-red area) for the robot’s instantaneous work Ut and kinetic energy Tt during locomotion over rough terrain for our method (eNCP) and competitors. The figure shows a trajectory with a strong initial disturbance, where blue markers denote samples within the predicted CI and red markers denote those outside. Note that only eNCP is able to predict well-calibrated CI intervals that cover both the disturbance and recovery phases.

Limitations and Future Work

The primary limitation of eNCP is its reliance on fully specified and perfectly known symmetry priors, which is a standard but restrictive assumption in geometric deep learning. In real-world scenarios, symmetries are often imperfect, partially broken, or entirely unknown.

The authors suggest that promising future directions include extending the framework to accommodate partial or uncertain symmetry information and developing statistical tests to automatically discover latent symmetries directly from data.

Summary updated

Abstract

In many real-world applications of regression, conditional probability estimation, and uncertainty quantification, exploiting symmetries rooted in physics or geometry can dramatically improve generalization and sample efficiency. While geometric deep learning has made empirical advances by incorporating symmetry and geometry priors, less attention has been given to statistical learning guarantees. In this paper, we introduce an equivariant representation learning framework that simultaneously addresses regression, conditional probability estimation, and uncertainty quantification while providing first-of-its-kind non-asymptotic statistical learning guarantees. Grounded in operator and group representation theory, our framework approximates the spectral decomposition of the conditional expectation operator, building representations that are both equivariant and disentangled along independent symmetry quotient groups. Empirical evaluations on synthetic datasets and real-world robotics applications confirm the potential of our approach, matching or outperforming existing equivariant baselines in regression while providing well-calibrated uncertainty estimates.
BibTeX
@inproceedings{ordonezapraez2026representation,
  author = {Daniel Ordoñez-Apraez and Vladimir Kostić and Alek Fröhlich and Vivien Brandt and Karim Lounici and Massimiliano Pontil},
  title = {{Representation Learning for Equivariant Inference with Guarantees}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=dWCG53oQWc},
}
Figure