Deep Residual Mixture Models

Perttu Hämäläinen, Arno SolinarXiv 2020 · OpenReview

Abstract

We propose Deep Residual Mixture Models (DRMMs), a novel deep generative model architecture. Compared to other deep models, DRMMs allow more flexible conditional sampling: The model can be trained once with all variables, and then used for sampling with arbitrary combinations of conditioning variables, Gaussian priors, and (in)equality constraints. This provides new opportunities for interactive and exploratory machine learning, where one should minimize the user waiting for retraining a model. We demonstrate DRMMs in constrained multi-limb inverse kinematics and controllable generation of animations.

Introduction

In the landscape of deep generative modeling, there is often a trade-off between scalability and flexibility. Standard architectures like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) are typically "hard-wired" for specific inputs and outputs; changing conditioning variables or adding constraints post-training often requires retraining the entire network. Conversely, classical Gaussian Mixture Models (GMMs) offer extreme flexibility for conditional sampling and constraint application but struggle to scale to the high-dimensional, complex manifolds found in real-world data.

Deep Residual Mixture Models (DRMMs) bridge this gap. By combining the hierarchical structure of deep learning with the mathematical tractability of mixture models, DRMMs scale like deep networks while remaining as flexible as a GMM.

Methods

The DRMM architecture is built on the principle of probabilistic residual learning. The model consists of a stack of layers, where each layer functions as a mixture model that attempts to reconstruct its input.

The DRMM layer architecture. Each layer $i$ computes component membership $h^{(i)}$, reconstructs the input $\hat{X}^{(i)}$, and passes the residual $R^{(i)}$ to the next layer.

As shown in the architecture diagram, each layer passes the residual—the difference between the input and the reconstruction—to the subsequent layer. This approach is conceptually similar to Residual Vector Quantization (RVQ). Because each layer adds its own mixture components to the previous ones, a model with $L$ layers and $K$ components per layer can represent up to $K^L$ distinct density modes. This exponential capacity allows relatively small models to capture incredibly complex distributions.

Training is conducted via a three-stage curriculum to avoid local optima, starting with shallow models and gradually introducing categorical latents and full joint likelihood optimization.

Results and Key Insights

The authors evaluated DRMMs on synthetic 2D data, humanoid inverse kinematics (IK), and high-dimensional motion capture datasets.

Visualization of DRMMs on synthetic data. Increasing the number of layers allows the model to capture complex, self-similar structures like spirals and Sierpinski triangles with exponential efficiency.

A key finding is the model's ability to exploit self-similarity. As illustrated in the synthetic data experiments, the residual structure is particularly effective at modeling fractal properties, with each layer "zooming in" on the details left by the previous one. Quantitative analysis using F1 scores confirmed that deeper models consistently outperform shallower ones for a fixed parameter count, validating the benefit of the deep architecture.

In practical applications, DRMMs demonstrated a "train once, predict anything" capability:

  • Inverse Kinematics: The model successfully solved multi-limb IK problems by applying post-training constraints (e.g., target positions and environmental boundaries) in closed-form using Gaussian integrals.
  • Animation Synthesis: The model generated natural-looking character animations by conditioning on goal states, without being explicitly programmed for specific paths.

Limitations and Future Work

The primary limitation currently lies in residual alignment. The model uses simple geometric shifts to calculate residuals. The authors suggest that per-component scaling or rotation could improve alignment, though this would increase the mathematical complexity of applying constraints. Future work may also explore integrating DRMMs as flexible latent space models within other architectures like CNNs or Transformers.

Summary updated

Abstract

We propose Deep Residual Mixture Models (DRMMs), a novel deep generative model architecture. Compared to other deep models, DRMMs allow more flexible conditional sampling: The model can be trained once with all variables, and then used for sampling with arbitrary combinations of conditioning variables, Gaussian priors, and (in)equality constraints. This provides new opportunities for interactive and exploratory machine learning, where one should minimize the user waiting for retraining a model. We demonstrate DRMMs in constrained multi-limb inverse kinematics and controllable generation of animations.
BibTeX
@misc{hamalainen2020deep,
  author = {Perttu Hämäläinen and Arno Solin},
  title = {{Deep Residual Mixture Models}},
  year = {2020},
  eprint = {2006.12063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.12063},
}
Figure