Reward-oriented Causal Representation Learning

Zirui Yan, Emre Acartürk, Ali TajerNeurIPS 2025 · OpenReview

Causal Representation Learning (CRL) typically aims to perfectly uncover hidden, low-dimensional causal factors from high-dimensional observations, such as extracting the underlying physics from raw image pixels. Traditionally, researchers have focused on "identifiability"—the ability to perfectly reconstruct these latent variables and their entire causal graph.

However, perfect recovery is often unnecessary and computationally wasteful if the ultimate goal is a specific downstream task (e.g., a robot arm reaching a target). In these scenarios, an agent only needs to understand the causal relationships relevant to maximizing its reward. This paper introduces Reward-oriented Causal Representation Learning (RO-CRL), a framework that shifts the focus from perfect latent recovery to learning the causal structure only at the coarsest level sufficient for optimizing a task.

The RO-CRL Framework

The RO-CRL framework models the environment using linear Structural Equation Models (SEMs) for the latent variables, which are then mapped to the observation space via a linear transformation. To navigate the trade-off between exploring the causal structure and exploiting it for rewards, the authors propose an adaptive exploration algorithm.

The pipeline consists of two main phases:

  1. Forced Exploration: An initial phase of atomic interventions to establish baseline statistics.
  2. Adaptive Exploration: A continuous loop that iteratively refines its understanding of the environment and selects actions.
The RO-CRL pipeline integrates latent recovery with UCB-based intervention selection.

During the adaptive phase, the algorithm performs CRL Inference by estimating the latent variables and graph using differences in precision matrices between observational and interventional data. It then uses an Upper Confidence Bound (UCB) Selection rule to pick the next intervention, prioritizing actions that might yield high rewards or resolve critical structural uncertainties. An Under-sampling Check ensures that single-node interventions are sampled frequently enough to maintain accurate structural estimates.

Key Insights

  • "Coarseness" is a Virtue: Learning the latent structure only to the degree necessary for the task significantly improves efficiency compared to exhaustive mapping.
  • Precision Matrix Differences as a Signal: The algorithm effectively uses the difference between precision matrices of observational and interventional data to "see" the latent causal structure through high-dimensional noise.
  • Unifying CRL and Causal Bandits: The paper demonstrates that standard Causal Bandits (CB) are a special case of RO-CRL where the transformation is the identity matrix. Notably, the RO-CRL algorithm actually improves state-of-the-art regret bounds for several standard CB settings.
  • Finite-Sample Guarantees: Unlike many previous works that rely on asymptotic (infinite data) assumptions, this paper provides concrete finite-sample PAC recovery guarantees.

Theoretical Guarantees and Regret Bounds

The authors provide a comprehensive theoretical analysis of regret—the difference between the reward obtained by the algorithm and the reward an optimal "oracle" would have achieved.

Summary of recovery guarantees and regret bounds for RO-CRL.

For an $n$-dimensional latent space and a $d$-dimensional observation space over a horizon $T$, the algorithm's regret scales as $\tilde{O}(d^{1/3}n^{1/3}u^{2/3}T^{2/3} + u\sqrt{T})$, where $u$ is a parameter reflecting total uncertainty in the graph estimates.

Crucially, the authors prove an almost-matching lower bound of $\Omega(d^{1/3}n^{1/3}p^{2/3}T^{2/3} + p\sqrt{T})$ (where $p$ counts the number of causal paths), demonstrating that their upper bound is nearly tight. In empirical simulations on synthetic datasets, the regret curve flattens over time, confirming that the algorithm successfully identifies near-optimal interventions as it learns the system.

Limitations and Future Directions

While theoretically robust, the current framework relies heavily on linearity constraints for the causal system, the transformation, and the reward function. Real-world systems are often highly non-linear. Additionally, calculating the UCB can become computationally expensive for complex graphs, which may require approximations or specific graph assumptions (like non-negative weights) for practical implementation. Extending this reward-oriented framework to non-linear SEMs and evaluating it on complex real-world datasets (such as video or genomic data) represent the most promising next steps for this line of research.

Summary updated

Abstract

Causal representation learning (CRL) is the process of disentangling the *latent* low-dimensional causally-related generating factors underlying high-dimensional observable data. Extensive recent studies have characterized CRL identifiability and *perfect* recovery of the latent variables and their attendant causal graph. This paper introduces the notion of *reward-oriented* CRL, the purpose of which is to move away from perfectly learning the latent representation and instead learning it to the extent needed for optimizing a desired downstream task (reward). In reward-oriented CRL, perfectly learning the latent representation can be excessive; instead, it must be learned at the *coarsest* level sufficient for optimizing the desired task. Reward-oriented CRL is formalized as the optimization of a desired function of the observable data over the space of all possible interventions and focuses on linear causal and transformation models. To sequentially identify the optimal subset of interventions, an adaptive exploration algorithm is designed that learns the latent causal graph and the variables needed to identify the best intervention. It is shown that for an $n$-dimensional latent space and a $d$-dimensional observation space, over a horizon $T$ the algorithm's regret scales as $\tilde O(d^{\frac{1}{3}}n^{\frac{1}{3}}u^{\frac{2}{3}}T^{\frac{2}{3}} + u\sqrt{T})$, where $u$ measures total uncertainty in the graph estimates. Furthermore, an almost-matching lower bound is shown to scale as $\Omega(d^{\frac{1}{3}}n^{\frac{1}{3}}p^{\frac{2}{3}}T^{\frac{2}{3}} + p\sqrt{T})$, in which $u$ is replaced by $p$ that counts the number of causal paths in the graph.
BibTeX
@inproceedings{yan2025reward,
  author = {Zirui Yan and Emre Acartürk and Ali Tajer},
  title = {{Reward-oriented Causal Representation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=I1hgynwxGZ},
}
Figure