Regularized Multi-Output Gaussian Convolution Process With Domain Adaptation

Xinming Wang, Chao Wang, Xuan Song, Levi Kirby, Jianguo WuIEEE TPAMI 2023 · DOI

TL;DR

This paper introduces a novel framework called Regularized Multi-output Gaussian Convolution Process with Domain Adaptation (MGCP-R) to improve transfer learning for multi-output Gaussian processes (MGPs). It tackles two key challenges: negative transfer (where learning from some outputs harms the target) and input domain inconsistency (where source and target data have different feature spaces). MGCP-R achieves this by adaptively selecting informative sources and aligning input domains, leading to more accurate predictions.

What Problem Does This Paper Address?

Multi-output Gaussian Processes (MGPs) are powerful tools for modeling multiple related outputs simultaneously, offering benefits like improved generalization and uncertainty quantification. However, applying MGPs to transfer learning – where we want to leverage data from "source" outputs to improve predictions for a "target" output – faces significant hurdles.

Firstly, negative transfer can occur. If a source output is not truly related to the target, or even negatively correlated, attempting to learn from it can actually degrade the target's prediction accuracy. This is particularly problematic in large datasets where identifying truly relevant sources is difficult.

Secondly, input domain inconsistency is a common issue. Imagine trying to predict the density of a ceramic product. Data from a "dry pressing" manufacturing process might use temperature and pressure as inputs, while "additive manufacturing" might use solids loading and temperature. While temperature is shared, other inputs differ, making direct transfer of learned models challenging. Existing domain adaptation methods are often not directly applicable to the non-convex optimization problems inherent in MGPs.

This paper aims to address both negative transfer and input domain inconsistency within a unified MGP framework.

Methods

The authors propose a Regularized Multi-output Gaussian Convolution Process (MGCP-R) framework. Here's a breakdown of their key techniques:

  • Convolution Process (CP) for Non-separable Covariance: Instead of traditional separable covariance structures that can be restrictive, they use a convolution process. This allows for more flexible, non-separable covariance functions, enabling better modeling of complex relationships between outputs. This is a standard technique in advanced MGP literature but is applied here in a specific way.
  • Regularization for Source Selection: To combat negative transfer, they introduce a regularization term to the log-likelihood function. This penalty term adaptively shrinks the influence of less informative or negatively correlated source outputs towards zero. This effectively acts as a mechanism for automatically selecting the most beneficial sources for transfer learning.
  • Domain Adaptation through Marginalization and Expansion (DAME): To handle input domain inconsistency, they propose a novel method called DAME.
  • Marginalization: For sources with inconsistent input domains, they first identify shared input features and then "marginalize out" the unique features. This projects the source data onto the shared feature space.
  • Kernel Regression for Induced Data: A kernel regression model is trained on the marginalized source data to capture its underlying trend. This is used to generate "induced" data points in the shared domain.
  • Expansion: These induced data points are then "expanded" by adding noise to simulate the unique features of the target domain. This creates a "pseudo-dataset" that aligns with the target's input domain while preserving the marginal distribution of the original source. This pseudo-dataset can then be used for transfer learning.

Datasets

The paper uses the following datasets for evaluation:

  • Simulated Datasets:
    • 1D Example: A synthetic dataset with one-dimensional input, designed to test negative transfer.
  • Simulation Case II: A synthetic dataset with two-dimensional input, specifically designed to test domain adaptation with inconsistent input domains.
  • Simulation Case III: Synthetic datasets with varying numbers of sources and higher input dimensions to test scalability.
  • Real-World Dataset:
    • Ceramic Manufacturing Data: Data on ceramic product density collected from two different manufacturing processes (dry pressing and additive manufacturing). This dataset is used to demonstrate the framework's effectiveness in a practical scenario with domain inconsistency.

Key Insights and Ideas

  • Unified Approach to Transfer Learning Challenges: The most significant insight is the development of a single framework that simultaneously addresses both negative transfer and input domain inconsistency, two major roadblocks in MGP-based transfer learning.
  • Regularization as Implicit Source Selection: The regularization approach elegantly solves the problem of identifying relevant sources. By penalizing the influence of irrelevant sources, the model automatically "selects" the most informative ones, leading to more robust predictions.
  • DAME: Bridging Domain Gaps: The DAME method provides a principled way to adapt source data to the target domain. By marginalizing and then expanding, it creates a useful pseudo-dataset without losing essential information from the original sources. This is crucial for leveraging data from disparate sources.
  • Convolution Process for Flexibility: The use of convolution processes for covariance modeling allows for greater flexibility compared to simpler, separable models, which is essential for capturing complex inter-output relationships.

Key Results

  • Improved Prediction Accuracy: MGCP-R consistently outperformed baseline methods (non-regularized MGCP, MGCP-RF, BGCP-R, GCP) across various simulation scenarios and the real-world ceramic manufacturing case. For instance, in the ceramic manufacturing case, MGCP-R achieved a mean MAE of 0.286, significantly better than other methods.
  • Effective Source Selection: The regularization mechanism successfully shrunk the influence of irrelevant sources to near zero, as evidenced by parameter estimates and improved prediction accuracy compared to methods that included all sources.
  • Robustness to Domain Inconsistency: The DAME method enabled successful transfer learning even when source and target domains differed significantly, as demonstrated in Simulation Case II and the ceramic manufacturing example.
  • Computational Efficiency: While MGCP-RF (a full-covariance model) was computationally expensive, MGCP-R offered a good balance of performance and computational feasibility, with regularization reducing optimization time.

Limitations and Future Work

  • Computational Complexity with High Dimensions: While MGCP-R is more efficient than full-covariance models, the computational complexity can still be a challenge with a very large number of sources or high-dimensional inputs. The authors suggest that computationally-efficient algorithms are needed for such scenarios.
  • Correlated Noise: The current framework assumes independent noise. The authors suggest that considering correlated noise, particularly in time series analysis, could further improve prediction accuracy and model flexibility.
  • Joint Optimization of Modeling and Adaptation: The current approach treats MGCP modeling and domain adaptation as separate tasks. The authors propose that jointly optimizing these two aspects within a unified framework would be a valuable direction for future research.
  • Application to Classification: The current work focuses on regression. Extending the framework to classification problems, where posterior distributions are not analytically tractable, presents an interesting avenue for future work.

Summary updated

BibTeX
@article{wang2023regularized,
  author = {Xinming Wang and Chao Wang and Xuan Song and Levi Kirby and Jianguo Wu},
  title = {{Regularized Multi-Output Gaussian Convolution Process With Domain Adaptation}},
  journal = {IEEE TPAMI},
  year = {2023},
  doi = {10.1109/tpami.2022.3205036},
}
Figure