The Lauritzen-Chen Likelihood For Graphical Models

Ilya Shpitser

Introduction

Graphical models, including Markov Random Fields (MRFs) and Bayesian Networks (BNs), serve as the foundation for probabilistic reasoning across diverse fields such as computer vision, bioinformatics, and causal inference. Despite their ubiquity, constructing these models for complex, non-categorical data remains a significant challenge. Practitioners often encounter "incongenial" models—where local specifications contradict the global joint distribution—or "redundant" models, where overlapping parameters prevent unique identification.

In this work, Ilya Shpitser introduces the Lauritzen-Chen (L-C) Likelihood. This mathematical framework provides a principled approach to building graphical models that are guaranteed to be both congenial and non-redundant, while maintaining consistency across equivalent graph structures.

The Challenges of Model Specification

When defining an MRF, researchers typically utilize clique potentials—functions describing interactions among groups of variables. However, these potentials often function as "black boxes." It is difficult to ensure that an interaction defined for a set of variables $\{A, B\}$ is mathematically compatible with an interaction defined for $\{B, C\}$. This is known as the congeniality problem. Furthermore, these specifications frequently overlap, leading to redundancy, which complicates parameter identification and estimation.

Bayesian Networks (Directed Acyclic Graphs or DAGs) face a different hurdle. Multiple DAGs can represent the same set of conditional independence assumptions, forming a "Markov equivalence class." Ideally, a likelihood function should be invariant across all graphs within the same class. However, standard parametric approaches often introduce implicit assumptions based on edge directionality, causing the likelihood to vary depending on the specific DAG chosen. This inconsistency undermines score-based model selection and causal discovery.

The Lauritzen-Chen Factorization

The L-C Likelihood bridges two previously distinct mathematical concepts:

  1. Lauritzen’s Clique Factorization (1996): A method for decomposing distributions based on the cliques of an undirected graph.
  2. Chen’s Odds Ratio Decomposition (2007): A technique to decompose joint distributions into univariate conditionals and interaction terms (generalized odds ratios) relative to "reference values."

The resulting factorization represents the joint likelihood as a product of univariate conditional distributions, $p(v_k | v^*_{-k})$, and generalized interaction terms, $\phi_C(v_C)$. The univariate conditionals represent "main effects," while the interaction terms capture higher-order dependencies.

A critical property of this factorization is variational independence. This means that the parameters of one term can be modified without necessitating adjustments to others to maintain a valid joint distribution. This independence allows for greater flexibility, such as using semi-parametric methods (e.g., kernel regressions) for main effects while keeping interaction terms parametric.

Key Theoretical Results

The paper establishes several foundational results that validate the L-C framework:

  • Equivalence of Decompositions: Theorem 2 and Corollary 1 formally link Chen’s odds-ratio terms with Lauritzen’s clique potentials, providing a practical mechanism for constructing the L-C likelihood.
  • Non-Redundancy in CMRFs: Corollary 2 extends this framework to Conditional Markov Random Fields (CMRFs), offering a non-redundant factorization suitable for structured prediction tasks like image labeling.
  • Equivalence Class Consistency: Proposition 2 demonstrates that for any DAG, an L-C likelihood based on its "essential graph" (the graph representing its equivalence class) is equivalent to the standard DAG factorization. Crucially, it remains consistent regardless of the specific edge orientations within that class.

Implications and Future Work

The L-C framework offers a "best of both worlds" approach for researchers. It provides the mathematical rigor needed to avoid redundancy and inconsistency while allowing for the integration of flexible, semi-parametric modeling techniques. By applying the L-C likelihood to the essential graph, the author ensures that the likelihood is invariant to edge orientations that do not change the model's underlying logic.

However, practical implementation faces hurdles. Like most MRF-based approaches, the L-C likelihood involves a normalizing constant (partition function) $Z$. Calculating $Z$ is computationally expensive (NP-hard), requiring the adaptation of approximation methods like variational inference or sum-product algorithms. Additionally, while the paper provides the theoretical foundation, the development of efficient, high-dimensional estimators and user-friendly software remains an area for future research.

Conclusion

The Lauritzen-Chen Likelihood represents a significant step forward in the formalization of graphical models. By unifying clique factorization with odds-ratio decomposition, Shpitser provides a robust framework that ensures model consistency and non-redundancy, paving the way for more reliable causal discovery and probabilistic modeling in complex data environments.

Summary updated

Abstract

Graphical models such as Markov random fields (MRFs) that are associated with undirected graphs, and Bayesian networks (BNs) that are associated with directed acyclic graphs, have proven to be a very popular approach for reasoning under uncertainty, prediction problems and causal inference. Parametric MRF likelihoods are well-studied for Gaussian and categorical data. However, in more complicated parametric and semi-parametric settings, likelihoods specified via clique potential functions are generally not known to be congenial {(jointly well-specified)} or non-redundant. Congenial and non-redundant DAG likelihoods are far simpler to specify in both parametric and semi-parametric settings by modeling Markov factors in the DAG factorization. However, DAG likelihoods specified in this way are not guaranteed to coincide in distinct DAGs within the same Markov equivalence class. This complicates likelihoods based model selection procedures for DAGs by ``sneaking in'' potentially unwarranted assumptions about edge orientations. In this paper we link a density function decomposition due to Chen with the clique factorization of MRFs described by Lauritzen to provide a general likelihood for MRF models. The proposed likelihood is composed of variationally independent, and non-redundant closed form functionals of the observed data distribution, and is sufficiently general to apply to arbitrary parametric and semi-parametric models. We use an extension of our developments to give a general likelihood for DAG models that is guaranteed to coincide for all members of a Markov equivalence class. Our results have direct applications for model selection and semi-parametric inference.
BibTeX
@inproceedings{shpitser2023lauritzen,
  author = {Ilya Shpitser},
  title = {{The Lauritzen-Chen Likelihood For Graphical Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
}
Figure