Learning nonlinear level sets for dimensionality reduction in function\n approximation

Guannan Zhang, Jiaxin Zhang, Jacob HinklearXiv 2019 · DOI · arXiv

Abstract

We developed a Nonlinear Level-set Learning (NLL) method for dimensionality reduction in high-dimensional function approximation with small data. This work is motivated by a variety of design tasks in real-world engineering applications, where practitioners would replace their computationally intensive physical models (e.g., high-resolution fluid simulators) with fast-to-evaluate predictive machine learning models, so as to accelerate the engineering design processes. There are two major challenges in constructing such predictive models: (a) high-dimensional inputs and (b) small training data. Existing methods like sliced inverse regression and active subspace approaches reduce the input dimension by learning a linear coordinate transformation; our main contribution is to extend the transformation approach to a nonlinear regime. Specifically, we exploit reversible networks (RevNets) to learn nonlinear level sets of a high-dimensional function and parameterize its level sets in low-dimensional spaces.

Introduction: The Challenge of High-Dimensional Engineering Models

In engineering and physics, practitioners often rely on "black-box" simulators—such as fluid dynamics models or structural stress tests—to guide design processes. These models are typically high-dimensional, involving dozens of input parameters, and are incredibly slow to run. To accelerate design, engineers attempt to build "surrogate models" (fast-to-evaluate approximations). However, they face two significant hurdles:

  1. The Curse of Dimensionality: As the number of input dimensions grows, the volume of data required to train an accurate model increases exponentially.
  2. Data Scarcity: Because individual simulations are computationally expensive, training datasets are often limited to a few hundred points.

Existing techniques, such as Sliced Inverse Regression (SIR) or Active Subspaces (AS), attempt to mitigate these issues by identifying a few "important" linear directions in the input space. However, if the function's sensitivities follow a curved or nonlinear path, these linear methods fail to simplify the problem effectively, leading to poor approximations and overfitting.

Methodology: Nonlinear Level-set Learning (NLL)

The core innovation of this work is Nonlinear Level-set Learning (NLL). The method seeks a bijective (one-to-one and reversible) nonlinear transformation that maps original high-dimensional inputs $x$ to a new coordinate system $z$. In this transformed space, the function is designed to be sensitive to only a few "active" coordinates, while the remaining "inactive" coordinates represent directions along which the function value remains constant.

Visualization of level sets and gradients in NLL.

Key components of the NLL framework include:

  • Reversible Residual Networks (RevNets): Unlike standard neural networks, RevNets are perfectly invertible. This ensures no information is lost during coordinate transformation and allows for seamless mapping back to the original input space.
  • Gradient-Based Loss Function: The authors utilize the gradient of the target function to encourage "inactive" transformed coordinates to align with the function's level sets (surfaces of constant value). By moving along these sets, the function value does not change, effectively neutralizing those dimensions.
  • Jacobian Regularization: To ensure stability and volume preservation, a penalty term keeps the determinant of the Jacobian matrix close to one, preventing the transformation from becoming singular.

Experimental Results and Real-World Application

The NLL method was rigorously evaluated across synthetic and real-world engineering benchmarks:

  1. Synthetic Functions: In 20D tests, NLL reduced the sensitivity of "inactive" dimensions by two orders of magnitude, outperforming linear methods (SIR/AS) which only achieved one order of magnitude reduction. With just 500 training samples, NLL achieved a relative error (RMSE) of 8-13%, whereas standard neural networks and linear reduction methods failed to generalize, with errors between 30-65%.
  2. Composite Material Design: A critical application involved an 8D problem from NASA concerning the buckling-resistance of rocket inter-stage shells. The goal was to predict the "knockdown factor" based on the ply angles of eight composite layers.
Application of NLL to rocket inter-stage shell design.

In this "small data" regime (500 samples), NLL significantly outperformed all baselines, providing a highly accurate surrogate for a Finite Element Method (FEM) simulation that typically takes 10 hours per run.

Conclusion and Future Directions

NLL demonstrates that aligning coordinates with nonlinear level sets is a powerful strategy for dimensionality reduction. By using RevNets, the method preserves the topology of the input space while collapsing complexity more effectively than linear alternatives.

Current limitations include the requirement for function gradients, which may not always be available in black-box scenarios. Future research will explore gradient-free versions of NLL, potentially utilizing contour regression or manifold tangent learning to estimate necessary directions from function values alone. Additionally, addressing the computational overhead of Jacobian calculations and the non-uniqueness of the learned transformations remain open areas for improvement.

Summary updated

Abstract

We developed a Nonlinear Level-set Learning (NLL) method for dimensionality\nreduction in high-dimensional function approximation with small data. This work\nis motivated by a variety of design tasks in real-world engineering\napplications, where practitioners would replace their computationally intensive\nphysical models (e.g., high-resolution fluid simulators) with fast-to-evaluate\npredictive machine learning models, so as to accelerate the engineering design\nprocesses. There are two major challenges in constructing such predictive\nmodels: (a) high-dimensional inputs (e.g., many independent design parameters)\nand (b) small training data, generated by running extremely time-consuming\nsimulations. Thus, reducing the input dimension is critical to alleviate the\nover-fitting issue caused by data insufficiency. Existing methods, including\nsliced inverse regression and active subspace approaches, reduce the input\ndimension by learning a linear coordinate transformation; our main contribution\nis to extend the transformation approach to a nonlinear regime. Specifically,\nwe exploit reversible networks (RevNets) to learn nonlinear level sets of a\nhigh-dimensional function and parameterize its level sets in low-dimensional\nspaces. A new loss function was designed to utilize samples of the target\nfunctions' gradient to encourage the transformed function to be sensitive to\nonly a few transformed coordinates. The NLL approach is demonstrated by\napplying it to three 2D functions and two 20D functions for showing the\nimproved approximation accuracy with the use of nonlinear transformation, as\nwell as to an 8D composite material design problem for optimizing the\nbuckling-resistance performance of composite shells of rocket inter-stages.\n
BibTeX
@misc{zhang2019learning,
  author = {Guannan Zhang and Jiaxin Zhang and Jacob Hinkle},
  title = {{Learning nonlinear level sets for dimensionality reduction in function\textbackslash\{\}n approximation}},
  year = {2019},
  eprint = {1902.10652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.10652},
}
Figure