When designing probabilistic models to understand the world, we often frame the problem causally: a set of hidden, independent factors generates the data we observe. For example, "Rain" and "a broken water pipe" are two entirely independent latent causes that can both lead to the same observed effect: "Wet Pavement."
Intuitively, if you walk outside and see wet pavement, either cause is possible. But if you look up and notice it is pouring rain, your belief that a pipe burst drops significantly. The rain has explained away the wet pavement.
While this reasoning is entirely natural for humans, it introduces a profound mathematical challenge in machine learning. This phenomenon—the explaining-away effect—causes latent variables that are completely independent in our prior beliefs to become highly dependent once we condition them on an observation.
Prior (Independent Causes):
[ Rain ] [ Broken Pipe ]
Posterior (Coupled via Observation):
[ Rain ] <=========> [ Broken Pipe ]
\ /
v v
[ Wet Pavement ]
Understanding and mitigating this conditional dependency is a central thread in the evolution of deep generative models, bridging the gap between classical Bayesian networks and modern deep variational frameworks.
Why Explaining Away Makes Inference Intractable
In a directed graphical model, the joint probability of latent variables $h$ and observed data $v$ factorizes according to a directed acyclic graph. Because the prior $p(h)$ is typically chosen to be fully factorial (i.e., $p(h) = \prod_i p(h_i)$), generating data is computationally trivial: you simply sample each latent factor independently and pass them through a generative network to get $v$.
However, as Gehler et al. note, the act of observing $v$ couples these latent variables in the posterior distribution $p(h|v)$. Because multiple independent factors can trigger the same observed outcome, knowing the state of one factor alters the probabilities of all other potential causes. As the number of hidden components or the depth of the network grows, this entangling effect cascades. Gan et al. and Taylor et al. demonstrate that this coupling makes exact calculation of the posterior distribution exponentially complex, rendering exact inference and learning mathematically intractable in deep directed architectures.
This intractability poses a severe bottleneck for many-to-one causal relationships. For instance, Yan et al. highlight how this exact issue complicates the training of Noisy-OR models, where calculating the exact posterior of multiple independent causes given a shared effect becomes a massive computational hurdle when data is limited.
The Failure of Mean-Field Approximations
To bypass this intractability, modern frameworks like Variational Autoencoders (VAEs) often rely on Amortized Variational Inference. Instead of calculating the true posterior $p(h|v)$, they train an inference network $q(h|v)$ to approximate it.
The simplest and most common choice for this approximation is the mean-field assumption, which forces the variational posterior to be fully independent: $$q(h|v) = \prod_i q(h_i|v)$$
While mathematically convenient, this assumption directly contradicts the physics of directed models. As Rolfe and Su et al. argue, forcing an independent approximation onto a posterior that is inherently coupled by the explaining-away effect creates an expressivity bottleneck. Because the independent posterior cannot capture the true correlations induced by the data, the model fails to learn complex, highly correlated underlying data distributions.
Furthermore, Ver Steeg et al. point out that this conditional dependence ruins parallel coordinate updates during inference: because the optimal value for one latent variable depends explicitly on the current values of all other variables influencing the same observation, standard block-free parallel updates can become unstable or fail to converge.
Strategy 1: Structuring the Variational Posterior
If independent approximations fail because they ignore explaining away, one solution is to build inference networks that explicitly accommodate these induced dependencies.
Faithful Inversion
How do we know what edges to include in an inference network? Webb et al. introduced an algorithmic approach called NaMI (Natural Minimal I-map generator). By systematically inverting the graphical structure of a generative model, the authors demonstrate that an inference network can only reach the true posterior if its architecture explicitly includes edges between independent causes that share an observed child. Explicitly modeling these explaining-away dependencies ensures that the amortized inference network is "faithful" to the true posterior.
Generative Model (Prior): Faithful Inference Network:
[ h1 ] [ h2 ] [ h1 ] <====> [ h2 ]
\ / ^ ^
v v \ /
[ v ] [ v ]
Higher-Order and Automated Structures
Other researchers have focused on directly enriching the mathematical framework of the approximation:
- Second-Order Mean Field Theory: Kappen et al. adapted the Thouless-Anderson-Palmer (TAP) approximation from statistical physics to directed graphs. By incorporating second-order interactions, their method captures pairwise explaining-away correlations without the massive overhead of full covariance matrices.
- Automatic Structured Variational Inference (ASVI): Ambrogioni et al. developed a method that automatically derives structured variational families from a model's forward pass. By using a convex-update rule, ASVI naturally captures the posterior correlations created by shared observed children while maintaining the same time and space complexity as standard mean-field methods.
Strategy 2: Bypassing the Effect via Undirected Models
An entirely different school of thought avoids fixing the explaining-away problem by choosing architectures where the problem cannot exist in the first place: undirected graphical models.
In an undirected model, such as a Restricted Boltzmann Machine (RBM), the relationship between hidden units $h$ and visible units $v$ is symmetric, defined by an energy function rather than a causal direction. Crucially, because there are no directed "v-structures" (two parents pointing to one child), hidden units remain conditionally independent of each other given the observed data.
$$p(h|v) = \prod_i p(h_i|v)$$
This structural property completely eliminates the explaining-away effect during inference. As Hinton et al. explain, eliminating this statistical coupling allows a system to perform the rapid, independent updates of hidden states necessary for real-time perception and sequence modeling.
This advantage led to a rich lineage of undirected and hybrid architectures designed to bypass the explaining-away bottleneck:
- The Rate Adapting Poisson (RAP) Model: Gehler et al. utilized an undirected framework for information retrieval and object recognition, enabling lightning-fast inference via simple matrix multiplication because the latent topics do not entangle when data is observed.
- Energy-Based Models (EBMs): For overcomplete representations, Welling et al. bypassed the slow, iterative optimization techniques required to resolve explaining-away coupling in directed models by treating features as deterministic functions of inputs within an energy-based framework.
- Temporal and Product Models: To handle high-dimensional sequences without inference bottlenecks, Sutskever et al. introduced the Temporal Restricted Boltzmann Machine (TRBM), and Taylor et al. revitalized Products of Hidden Markov Models (PoHMMs). Both approaches leverage undirected connections within layers to keep hidden states conditionally independent given the observations.
The Hybrid Trade-off
The choice between directed and undirected models represents a fundamental trade-off in machine learning. Directed models offer easy, ancestral data generation but suffer from intractable inference due to explaining away. Undirected models offer trivial inference but suffer from intractable learning and generation due to the difficulty of estimating the global partition function.
To balance these forces, Hinton and Hinton et al. proposed hybrid architectures, such as Deep Belief Networks. These models use directed connections in the deeper layers to capture a causal generative hierarchy, but terminate in an undirected associative memory (an RBM) at the top layer, allowing the system to enjoy the representational benefits of both paradigms.
However, researchers must tread carefully when mixing these paradigms. As Nguyen et al. demonstrated with Supervised RBMs, introducing even a single directed connection from latent variables to an outcome label immediately reintroduces the explaining-away effect, causing previously independent hidden units to become conditionally dependent and significantly driving up the computational cost of inference.
Similarly, in multi-output systems like Gaussian processes, Bruinsma et al. show that hidden processes typically become entangled once data is observed. To combat this, they introduce the Orthogonal Instantaneous Linear Mixing Model (OILMM), which uses an orthogonal mixing matrix to mathematically decouple the latent processes, restoring parallelization and linear scaling.
Summary
The explaining-away effect is an inescapable mathematical consequence of modeling the world with directed, independent causes. When we observe an effect, the potential causes become intrinsically coupled.
The history of deep generative modeling can be viewed as a continuous effort to handle this coupling: either by developing sophisticated, structured variational distributions that can faithfully mirror these induced dependencies, or by designing undirected and hybrid architectures that structurally design them out of the equation entirely.
Key Foundational Work
Spiking Boltzmann Machines (1999) — contributed 1 concept elements to this direction.
Second Order Approximations for Probability Models (2000) — contributed 1 concept elements to this direction.