Visualizing Movement Control Optimization Landscapes

Perttu Hämäläinen, Juuso Toikka, C. Karen LiuIEEE TVCG 2019 · DOI · OpenReview

TL;DR

This paper introduces a novel visualization technique for understanding the complex optimization landscapes of character movement control. By adapting methods from neural network research, they reveal why optimizing movement is challenging and demonstrate how common techniques like early termination and specific action parameterizations significantly improve optimization efficiency.

What Problem Does This Paper Address?

Creating realistic and dynamic character movements in animation and robotics often involves solving complex optimization problems. These problems aim to find the best sequence of actions (like joint torques or target poses) to achieve desired goals (e.g., walking, jumping) while minimizing costs (e.g., energy expenditure, deviation from a target). However, the objective functions governing these movements are often not simple, closed-form equations. Instead, they are derived from complex physics simulations, making them high-dimensional, non-convex, and potentially multimodal. This lack of understanding about the "shape" of the optimization problem makes it difficult to diagnose why optimization fails, why it's slow, or why certain approaches work better than others. The authors aim to shed light on these challenges by visualizing these high-dimensional optimization landscapes.

Methods

The core contribution is the application of a 2D random slice visualization technique, inspired by recent work in visualizing neural network loss landscapes. This method involves:

  • Random Slice Visualization: Evaluating the high-dimensional objective function along a 2D plane (a subspace) defined by two random orthogonal basis vectors. This allows for a visual representation of the landscape's shape, including its convexity, conditioning, and multimodality.
  • Testbed Problems: The visualizations are applied to two main test problems:
  • Inverted Pendulum: A simple, well-understood system where analytical solutions and Hessians can be computed for deeper analysis. This is used to study the effects of trajectory length, action space choice, reward vs. cost functions, and early termination.
  • 3D Humanoid Locomotion: A more complex, realistic scenario using the Unity Machine Learning Agents framework, demonstrating the generalizability of the approach to high-dimensional, non-differentiable physics simulations.
  • Analysis of Optimization Properties: The visualizations are used to investigate how factors like trajectory length, action parameterization (e.g., torques vs. target angles), reward formulation (costs vs. exponentiated rewards), and early termination affect the optimization landscape.

Datasets

The paper does not explicitly mention or introduce new datasets for training. Instead, it uses:

  • Simulated Environments: The primary "data" comes from simulations of an inverted pendulum and a 3D humanoid locomotion task. These simulations serve as the objective functions that are then visualized.
  • OpenAI Gym MuJoCo Tasks: For broader validation, the authors replicate experiments from Section 9.4 using four standard OpenAI Gym MuJoCo environments (Walker2d-v2, Hopper-v2, HalfCheetah-v2, Humanoid-v2).

Key Insights and Ideas

  • Visualizing the "Why": The visualizations provide intuitive explanations for why certain optimization techniques work well. For instance, they show how parameterizing actions as target angles (rather than raw torques) leads to a much better-conditioned optimization landscape.
  • Trajectory Optimization's Curse of Dimensionality: As the planning horizon (trajectory length) increases, trajectory optimization becomes significantly more ill-conditioned and non-separable, making it much harder to solve.
  • Policy Optimization Scales Better: Neural network-based policy optimization, despite having many more parameters, scales much better with longer planning horizons, exhibiting landscapes that remain relatively well-behaved.
  • Early Termination is a Double-Edged Sword: While early termination can remove false optima and improve landscape convexity, it can also introduce new problems if not handled carefully, especially when combined with negative rewards.
  • Splines as a Preconditioner: Using pose splines for action parameterization can significantly improve the conditioning of trajectory optimization, effectively treating the spline control points as higher-level actions.
  • Visualizations Predict Performance: The authors demonstrate that the insights gained from visualizing the landscapes correlate well with actual optimization performance, as measured by metrics like convergence speed and final reward.

Key Results

  • Inverted Pendulum:
    • Trajectory optimization becomes increasingly ill-conditioned and non-separable with longer trajectories (Fig. 5).
  • Policy optimization landscapes become more multimodal with longer episodes (Fig. 6).
  • Parameterizing actions as target angles (instead of torques) leads to significantly better-conditioned landscapes, especially for longer horizons (Fig. 7 vs. Fig. 5).
  • Early termination can remove false optima, making landscapes more convex, particularly when combined with reward functions (Fig. 9 & 10).
  • Humanoid Locomotion:
    • Trajectory optimization landscapes become increasingly multimodal and ill-conditioned with longer planning horizons (Fig. 13).
  • Spline parameterization improves trajectory optimization landscapes, making them smoother and less noisy (Fig. 14 vs. Fig. 13).
  • Policy optimization landscapes remain largely unchanged and well-behaved with increasing planning horizons (Fig. 16).
  • Scalability Comparison (Fig. 17): For long planning horizons (e.g., 10 seconds), policy optimization (PPO) significantly outperforms trajectory optimization (LM-MA-ES, LM-MA-ES with splines) in terms of reward achieved per simulated timestep.

Limitations and Future Work

  • Optimistic Visualizations: The 2D random slice visualizations are generally optimistic regarding ill-conditioning and multimodality. While they have high specificity (if a problem looks bad, it is bad), their sensitivity is limited (a problem that looks good might still be difficult).
  • Future Work:
    • Investigating more sensitive visualization techniques, such as using basis vectors derived from Hessian eigenvectors.
  • Developing methods to visualize problems with higher intrinsic dimensionality more accurately.
  • Exploring parameterization of actions as target states for complex agents with unactuated roots, potentially using general-purpose neural network controllers.
  • Extending visualizations to analyze spacetime optimization.

Summary updated

Abstract

A large body of animation research focuses on optimization of movement control, either as action sequences or policy parameters. However, as closed-form expressions of the objective functions are often not available, our understanding of the optimization problems is limited. Building on recent work on analyzing neural network training, we contribute novel visualizations of high-dimensional control optimization landscapes; this yields insights into why control optimization is hard and why common practices like early termination and spline-based action parameterizations make optimization easier. For example, our experiments show how trajectory optimization can become increasingly ill-conditioned with longer trajectories, but parameterizing control as partial target states---e.g., target angles converted to torques using a PD-controller---can act as an efficient preconditioner. Both our visualizations and quantitative empirical data also indicate that neural network policy optimization scales better than trajectory optimization for long planning horizons. Our work advances the understanding of movement optimization and our visualizations should also provide value in educational use.
BibTeX
@article{hamalainen2019visualizing,
  author = {Perttu Hämäläinen and Juuso Toikka and C. Karen Liu},
  title = {{Visualizing Movement Control Optimization Landscapes}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2019},
  doi = {10.1109/tvcg.2020.3018187},
}
Figure