TL;DR
Generative models in computer vision have struggled with inference due to complex, high-dimensional latent spaces. This paper introduces the "informed sampler," a novel Markov Chain Monte Carlo (MCMC) method that leverages existing discriminative computer vision techniques to guide the sampling process. This approach significantly improves inference efficiency and robustness in challenging generative models, particularly those involving rendering engines and multi-modal posteriors.
What Problem Does This Paper Address?
Computer vision has long been fascinated by the idea of generative models that can accurately capture the physical process of image formation. The dream is to model a scene using latent variables (like object positions, lighting, etc.) and then use a rendering engine to produce an image. With such a model, Bayesian inference could, in principle, allow us to infer scene properties from observed images. However, this vision has largely remained unfulfilled. The primary hurdle is the difficulty of performing posterior inference in these complex, high-dimensional models. This difficulty has led the field to favor efficient but data-hungry discriminative models. This paper aims to bridge this gap by developing a general and efficient inference technique that can be applied to accurate generative models, effectively reviving the promise of Bayesian inference in computer vision.
Methods
The core contribution is the informed sampler, a Markov Chain Monte Carlo (MCMC) method that enhances standard sampling by incorporating discriminative information. The key idea is to use a mixture kernel for the proposal distribution, combining a traditional local proposal ($T_L$) with a new informed global proposal ($T_G$). This global proposal is learned discriminatively from the observed image ($\hat{I}$) and is conditioned on it.
Here's how it works:
- Discriminative Learning of $T_G$: The paper proposes learning $T_G(\cdot|\hat{I})$ offline. This involves generating synthetic data from the generative model, computing feature representations (e.g., Histograms of Oriented Gradients - HoG) of the rendered images, and then using techniques like k-means clustering and Kernel Density Estimation (KDE) to learn a conditional density estimate for the latent variables given these features.
- Mixture Kernel: At test time, the sampler uses a mixture of the local proposal (exploring the immediate neighborhood of the current state) and the learned global proposal (making larger jumps guided by the image). The mixture coefficient ($\alpha$) controls the balance between these two.
- "Inverse Graphics": The overall approach can be framed as "inverse graphics," where the goal is to infer the scene parameters ($\theta$) that would generate a given image ($\hat{I}$), using a rendering engine $G$ as the forward model ($p(\hat{I}|\theta)$).
The paper also discusses baseline methods for comparison, including standard Metropolis-Hastings (MH), Metropolis-Hastings within Gibbs (MHWG), Parallel Tempering (PT), and a Regenerative MCMC method (REG-MH).
Datasets
The paper does not explicitly mention using pre-existing public datasets for training or evaluation. Instead, it generates synthetic data for its experiments using the described generative models (rendering engines). The "datasets" are thus created on-the-fly for each experiment by rendering images from known ground truth parameters and then using these rendered images and their corresponding parameters for training the informed proposal distributions and for evaluating the sampling methods.
Key Insights and Ideas
- Bridging Generative and Discriminative: The most significant insight is that discriminative methods, which have excelled in computer vision, can be effectively "plugged into" generative models to overcome their inference limitations. This is not about replacing generative models but augmenting them.
- Leveraging "Stochastic Inverses": The learned global proposal $T_G$ acts as a "stochastic inverse" of the generative model, providing informed guesses about the latent variables based on the observed image.
- Generalizability: The proposed method is designed to be general. The learning of the global proposal using KDEs on clustered features is a relatively simple and reusable technique, allowing it to be applied across different generative models and computer vision tasks without extensive re-engineering.
- Multi-modality is Key: The paper highlights that many computer vision problems inherently lead to multi-modal posterior distributions (e.g., due to symmetries in the scene). Standard samplers struggle with this, whereas the informed sampler's global proposals are crucial for navigating these modes.
- The Power of "Informed" Jumps: The informed sampler demonstrates that intelligently guided proposals, even if not perfect, can dramatically improve mixing and convergence compared to random or purely local exploration.
Key Results
The informed sampler (INF-MH and INF-INDMH) consistently outperformed baseline methods across three challenging experiments:
- Estimating Camera Extrinsics: In a multi-modal problem involving camera pose estimation in a cubical room, INF-MH achieved significantly better convergence (lower PSRF) and visited more modes than MH, PT, and REG-MH. It also showed better auto-correlation, indicating more independent samples.
- Occluding Tiles: For a higher-dimensional problem with occlusions, standard samplers like MH and MHWG failed to converge. The informed sampler, particularly when adapted to block-wise updates (INF-BMHWG), showed much better convergence, higher acceptance rates, and lower reconstruction errors.
- Estimating Body Shape: In a 3D body shape reconstruction task from depth images, INF-MH demonstrated faster convergence and lower RMSE compared to baselines. It also provided a useful characterization of posterior uncertainty, which is beneficial for applications like body measurements.
Across experiments, the informed sampler consistently achieved higher acceptance rates and lower PSRF values, indicating more efficient and reliable inference.
Limitations and Future Work
- Generative Model Accuracy: The current approach assumes an accurate generative model. Relaxing this assumption to handle approximate generative models is a significant area for future work.
- Feature Representation: While the paper uses simple features (HoG, depth histograms), more sophisticated discriminative features could further improve performance.
- Trans-dimensional Models: The current framework is for fixed-dimensional latent spaces. Extending it to trans-dimensional models (e.g., using reversible jump MCMC) is a direction for future research.
- Learning $T_G$: The paper explored KDEs and random forests for learning $T_G$. Investigating other conditional density estimation techniques or more advanced regression approaches could yield further improvements.
- High-Dimensionality: While the block-wise approach for occluding tiles showed promise, scaling to very high-dimensional problems remains a challenge. Future work could explore more sophisticated ways to factorize distributions and learn block-specific informed proposals.
- Real-world Data: The experiments primarily used synthetic data. Applying the method to real-world, noisy, and potentially incomplete data would be a crucial next step.
Summary updated