Deep Multi-Fidelity Active Learning of High-Dimensional Outputs

Shibo Li, Robert Kirby, Shandian ZheOpenReview 2022 · OpenReview

Abstract

Many applications, such as in physical simulation and engineering design, demand we estimate functions with high-dimensional outputs. The training examples can be collected with different fidelities to allow a cost/accuracy trade-off. In this paper, we consider the active learning task that identifies both the fidelity and input to query new training examples so as to achieve the best benefit-cost ratio. To this end, we propose DMFAL, a Deep Multi-Fidelity Active Learning approach. We first develop a deep neural network-based multi-fidelity model for learning with high-dimensional outputs, which can flexibly, efficiently capture all kinds of complex relationships across the outputs and fidelities to improve prediction. We then propose a mutual information-based acquisition function that extends the predictive entropy principle. To overcome the computational challenges caused by large output dimensions, we use multi-variate Delta's method and moment-matching to estimate the output posterior, and Weinstein-Aronszajn identity to calculate and optimize the acquisition function. The computation is tractable, reliable and efficient. We show the advantage of our method in several applications of computational physics and engineering design.

Introduction

In fields like computational physics and engineering design, researchers often need to map simple inputs (such as design parameters) to complex, high-dimensional outputs (such as 3D stress fields or fluid velocity on a mesh). Generating these outputs via numerical solvers (e.g., PDE solvers) is computationally expensive and time-consuming. To mitigate this, multi-fidelity data is often employed: low-fidelity simulations provide fast but coarse approximations, while high-fidelity simulations offer accuracy at a much higher cost.

The core challenge lies in Active Learning (AL): determining which input to test next and at which fidelity to maximize information gain per unit cost. Traditional AL methods struggle with high-dimensional outputs because calculating uncertainty (entropy) for vectors with thousands of dimensions is computationally prohibitive. This paper introduces DMFAL (Deep Multi-Fidelity Active Learning), a framework designed to build efficient surrogate models for expensive physical simulations by intelligently balancing fidelity and cost.

Methodology

DMFAL introduces three primary innovations to address the limitations of existing multi-fidelity active learning:

1. Chained Neural Network Architecture

The model treats fidelities as a sequence. Each fidelity is represented by its own neural network that produces a low-dimensional "latent" output. This latent representation, combined with the original input, serves as the input for the subsequent fidelity's network. This architecture allows the model to learn complex, non-linear correlations between different simulation accuracies while avoiding the noise and resolution issues inherent in raw high-dimensional data.

2. Mutual Information Acquisition

DMFAL utilizes a mutual information-based acquisition function. Rather than simply targeting areas of high uncertainty, it selects the input and fidelity that maximize the mutual information between the query and the target (highest) fidelity, normalized by the query cost. This ensures the most "bang for the buck" in data collection.

3. Efficient High-Dimensional Uncertainty Estimation

To overcome the "curse of dimensionality," the authors employ the multivariate delta method to linearize the network's uncertainty and the Weinstein-Aronszajn identity. This mathematical approach allows the calculation of the determinant of a massive $d \times d$ matrix (where $d$ is the output dimension) by operating on a much smaller $k \times k$ latent matrix. This makes the computation of predictive entropy analytically tractable and significantly faster than traditional Monte Carlo methods.

Experimental Evaluation

The authors evaluated DMFAL on several challenging problems in physics and engineering, including:

  • Partial Differential Equations (PDEs): Burgers’ equation, Poisson’s equation, and the Heat equation.
  • Topology Optimization: Predicting the optimal structure of an L-shaped beam under stress.
  • Computational Fluid Dynamics (CFD): Predicting velocity fields in fluid flow (Navier-Stokes) with output dimensions up to 112,500.
Comparison of DMFAL against baseline methods across six benchmark tasks.

Key Results

  • Superior Efficiency: As shown in the figure above, DMFAL consistently outperforms standard active learning baselines (such as MF-BALD or Predictive Variance) and random sampling across all tasks, achieving lower normalized Root Mean Square Error (nRMSE) for the same total cost.
  • Significant Speedups: Once trained, the DMFAL surrogate models provided a 42x speedup for topology optimization and a 466x speedup for fluid dynamics compared to traditional numerical solvers.
  • Scalability: The method successfully scaled to output dimensions exceeding 100,000, a regime where traditional Gaussian Process-based models typically fail.

Conclusion and Future Work

DMFAL demonstrates a robust approach to active learning for high-dimensional, multi-fidelity data. By leveraging latent information transfer and analytic shortcuts for uncertainty estimation, it achieves significant computational savings.

However, some limitations remain. The current framework assumes a strict hierarchy of fidelities, which may not apply to multi-source data without a clear ranking. Additionally, the reliance on the delta method's linearization might be less effective for highly discontinuous functions. Future research could explore generalizing acquisition functions like BALD to broader multi-fidelity settings and relaxing the hierarchical assumptions.

Summary updated

Abstract

Many applications, such as in physical simulation and engineering design, demand we estimate functions with high-dimensional outputs. The training examples can be collected with different fidelities to allow a cost/accuracy trade-off. In this paper, we consider the active learning task that identifies both the fidelity and input to query new training examples so as to achieve the best benefit-cost ratio. To this end, we propose DMFAL, a Deep Multi-Fidelity Active Learning approach. We first develop a deep neural network-based multi-fidelity model for learning with high-dimensional outputs, which can flexibly, efficiently capture all kinds of complex relationships across the outputs and fidelities to improve prediction. We then propose a mutual information-based acquisition function that extends the predictive entropy principle. To overcome the computational challenges caused by large output dimensions, we use multi-variate Delta's method and moment-matching to estimate the output posterior, and Weinstein-Aronszajn identity to calculate and optimize the acquisition function. The computation is tractable, reliable and efficient. We show the advantage of our method in several applications of computational physics and engineering design.
BibTeX
@misc{li2022deep,
  author = {Shibo Li and Robert Kirby and Shandian Zhe},
  title = {{Deep Multi-Fidelity Active Learning of High-Dimensional Outputs}},
  year = {2022},
  url = {https://openreview.net/forum?id=WZ-bYJBfbZw},
  howpublished = {OpenReview},
}
Figure