Half-VAE: An Encoder-Free VAE to Bypass Explicit Inverse Mapping

Yuan-Hao Wei, Yan-Jie Sun, Chen ZhangarXiv 2024 · DOI · arXiv

Abstract

Inference and inverse problems are fundamentally linked, involving the deduction of unknown causes from observed data. While Variational Autoencoders (VAEs) are powerful tools for approximating complex posterior distributions, they typically rely on an encoder to perform inverse mapping. This study introduces the "Half-VAE," a streamlined architecture that eliminates the encoder entirely. By treating latent variables as direct trainable parameters optimized through the objective function, the Half-VAE bypasses the need for explicit inverse mapping. We demonstrate that with a suitable prior, such as a Gaussian Mixture Model (GMM), these latent parameters can converge to mutually independent components, effectively solving Independent Component Analysis (ICA) problems, even in complex scenarios.

Introduction: The Challenge of Inverse Mapping

In machine learning, inverse problems—such as Independent Component Analysis (ICA)—require identifying hidden sources (latent variables) from observed mixtures. Standard VAE frameworks address this through an encoder-decoder structure: the encoder learns a functional mapping from observations to latent space, while the decoder learns the generative process.

However, this approach encounters significant hurdles in "underdetermined" systems, where the number of hidden sources exceeds the number of observations. In such cases, a unique mathematical inverse mapping ($f^{-1}$) may not exist. If the mapping is ill-defined or non-existent, a neural network encoder cannot reliably learn it. This research explores whether the "unmixing" (encoding) step can be bypassed entirely by focusing on direct optimization of the latent space.

The Half-VAE Framework

The proposed "Half-VAE" architecture simplifies the traditional VAE by removing the encoding component. Instead of utilizing a neural network to predict latent variables ($Z$) from input data ($X$), the latent variables themselves—specifically their means ($Z_\mu$) and variances ($Z_\sigma$)—are treated as trainable weights.

The Half-VAE architecture treats latent variables as trainable parameters, eliminating the need for an explicit encoder network.

In this setup, the latent variables are optimized directly via backpropagation using a modified Evidence Lower Bound (ELBO). This objective function balances reconstruction accuracy (via the decoder) with a regularization term (KL divergence) that encourages the latent variables to match a prior distribution.

Adaptive Priors

A critical component of the Half-VAE's success is the use of an adaptive Gaussian Mixture Model (GMM) prior. Unlike a standard Normal prior, which is often too restrictive for the non-Gaussian signals typical in ICA, the GMM parameters are also trainable. This allows the model to dynamically adapt its expectations of the independent components' distributions during the training process.

Experimental Validation

The efficacy of the Half-VAE was tested using simulated ICA datasets consisting of three independent signals mixed linearly. The model's task was to recover these original signals without any prior knowledge of the mixing matrix or the source distributions.

The Half-VAE was benchmarked against two traditional VAE configurations: a full VAE with a GMM prior and a "Vanilla VAE" using a standard normal prior.

Key Results and Convergence

The Half-VAE demonstrated superior performance in recovering hidden signals. Quantitatively, the Half-VAE (GMM) achieved a mean Root Mean Square Error (RMSE) of 0.1846, significantly outperforming the full VAE with GMM (0.2528) and the Vanilla VAE (0.6892).

Evolution of latent variables over 1,000 epochs, showing the convergence of trainable parameters $Z_\mu$ toward the ground truth signals.

Qualitative analysis confirms these findings. As shown in the training progression, the latent parameters $Z_\mu$ start as near-constant or noisy initializations and gradually converge to accurately reflect the ground truth independent components over 1,000 epochs. Furthermore, the model exhibited high confidence, with narrow 95% confidence intervals for the inferred components.

Discussion

The primary insight of this work is that while a functional inverse mapping might be mathematically impossible to define, the latent values themselves remain reachable through direct optimization. By shifting from "amortized inference" (using one network for all inputs) to "direct optimization" of latent parameters, the Half-VAE sidesteps the requirement for a functional mapping.

Limitations and Future Directions

Despite its success, the Half-VAE faces several challenges:

  1. Initialization Sensitivity: Convergence speed is highly dependent on the initial random state of the parameters.
  2. Scalability: Since latent variables are parameters tied to specific data points, scaling to massive datasets may require more memory than amortized approaches.
  3. Underdetermined Limits: While it bypasses explicit mapping, solving highly underdetermined ICA still requires robust constraints.

Future research will focus on developing smarter initialization strategies and exploring alternative prior distributions to handle a wider variety of signal characteristics.

Summary updated

Abstract

Inference and inverse problems are closely related concepts, both fundamentally involving the deduction of unknown causes or parameters from observed data. Bayesian inference, a powerful class of methods, is often employed to solve a variety of problems, including those related to causal inference. Variational inference, a subset of Bayesian inference, is primarily used to efficiently approximate complex posterior distributions. Variational Autoencoders (VAEs), which combine variational inference with deep learning, have become widely applied across various domains. This study explores the potential of VAEs for solving inverse problems, such as Independent Component Analysis (ICA), without relying on an explicit inverse mapping process. Unlike other VAE-based ICA methods, this approach discards the encoder in the VAE architecture, directly setting the latent variables as trainable parameters. In other words, the latent variables are no longer outputs of the encoder but are instead optimized directly through the objective function to converge to appropriate values. We find that, with a suitable prior setup, the latent variables, represented by trainable parameters, can exhibit mutually independent properties as the parameters converge, all without the need for an encoding process. This approach, referred to as the Half-VAE, bypasses the inverse mapping process by eliminating the encoder. This study demonstrates the feasibility of using the Half-VAE to solve ICA without the need for an explicit inverse mapping process.
BibTeX
@misc{wei2024half,
  author = {Yuan-Hao Wei and Yan-Jie Sun and Chen Zhang},
  title = {{Half-VAE: An Encoder-Free VAE to Bypass Explicit Inverse Mapping}},
  year = {2024},
  eprint = {2409.04140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04140},
}
Figure