Example-Based Image Synthesis of Articulated Figures

Trevor DarrellNeurIPS 1998 · OpenReview

Introduction

Image-based view synthesis aims to render realistic objects without the need to build exhaustive 3D models. While this approach works well for rigid objects or smooth deformations (like facial expressions), it traditionally fails when applied to articulated figures, such as human arms.

This paper introduces a novel method for synthesizing realistic images of articulated objects by breaking down complex, non-linear appearance mappings into a collection of "safe," piecewise-smooth interpolation networks. By clustering examples into sets where interpolation is guaranteed to be well-behaved, the system avoids the physically impossible "ghosting" artifacts that plague traditional global interpolation methods.

The Challenge of Articulated Figures

Articulated objects present two major challenges for standard learning networks:

  1. Discontinuities: Physical constraints or obstacles might force an object to suddenly jump from one configuration to another. For example, an arm might have to move from an "elbow-up" to an "elbow-down" position to avoid a table.
  2. One-to-Many Mappings: In parameter space (like hand coordinates), a single point might correspond to multiple valid physical appearances. A hand can be in the exact same location with the elbow pointing up or down.
Synthetic 2DOF arm configurations showing overlapping poses and complex mappings.

Traditional interpolation methods, such as Radial Basis Functions (RBFs) or Principal Component Analysis (PCA), attempt to find a single smooth manifold for all data. When faced with these articulated "singularities," they produce physically invalid images—blurry messes that attempt to average two entirely different poses.

Methodology: Piecewise Smooth Interpolation

To overcome these limitations, the author proposes representing appearance as a set of piecewise smooth functions rather than one global model.

  • Coupled Representation: The system separates "shape" (a dense correspondence map or flow) from "texture" (the actual pixel data). It warps textures based on interpolated shapes to create the final image.
  • Set Growing Algorithm: To find "safe" regions for interpolation, the method uses a clustering technique. It starts with seed points and grows a set by adding neighbors only if the new point is well-approximated by the existing set, and all interior points within the resulting convex hull remain well-interpolated.
  • Restricted Interpolation: During synthesis, the system only interpolates if the desired parameters fall within the convex hull of a known "safe" set. If a point falls outside all sets, it snaps to the nearest valid hull boundary to prevent erroneous extrapolation.
  • Temporal Consistency: For video generation, the system stays within a chosen set as long as possible. If it must switch sets, it only moves to "adjacent" sets where the appearances are physically similar.

Key Insights

The most clever insight of this work is using the convex hull in parameter space as a mathematical safety net. If the system stays inside the hull of a "consistent" cluster, it is highly likely to produce a physically plausible image.

Furthermore, instead of trying to "solve" the complexity of articulated motion with a massive, complex neural network, the author simplifies the problem by partitioning the data into manageable, locally linear chunks. By allowing these sets to overlap, the system can represent multiple appearances for the same parameter and use temporal context to choose the correct one.

Results and Real-World Application

The paper evaluates the method on both synthetic and real-world datasets. In tests with a simulated 2DOF planar arm, a standard RBF network produced distorted, "broken" arms near workspace boundaries. In contrast, the proposed "example sets" method produced perfectly crisp, physically valid configurations.

Real human arm sequence demonstrating an 'elbow up' configuration.

On a real-world dataset consisting of a 33-frame human arm sequence, the method successfully discovered the different manifolds for "elbow up" and "elbow down" configurations. The method also proved efficient: it reduces the number of examples needed by identifying which points are redundant (those inside the hull that are already well-approximated) and which are essential (those defining the convex hull).

Limitations and Future Work

While highly effective, the method has a few notable limitations:

  • Correspondence Dependency: The approach relies heavily on having accurate dense correspondences (knowing which pixel on one arm matches a pixel on another). In the real-arm example, this required a semi-automated tracker.
  • Sparsity: If training data is too sparse in appearance space, the "set growing" algorithm might fail to find effective groupings. The author suggests using temporal trajectories from video to help bridge these gaps.
  • Scaling: While effective for a 2DOF arm, the paper does not discuss how the convex hull calculations or the number of sets would scale for high-degree-of-freedom characters, such as a full human body.

Summary updated

Abstract

We present a method for learning complex appearance mappings. such as occur with images of articulated objects. Traditional interpolation networks fail on this case since appearance is not necessarily a smooth function nor a linear manifold for articulated objects. We define an ap(cid:173) pearance mapping from examples by constructing a set of independently smooth interpolation networks; these networks can cover overlapping re(cid:173) gions of parameter space. A set growing procedure is used to find ex(cid:173) ample clusters which are well-approximated within their convex hull; interpolation then proceeds only within these sets of examples. With this method physically valid images are produced even in regions of param(cid:173) eter space where nearby examples have different appearances. We show results generating both simulated and real arm images.
BibTeX
@inproceedings{darrell1998example,
  author = {Trevor Darrell},
  title = {{Example-Based Image Synthesis of Articulated Figures}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=rJVTK_W_WS},
}
Figure