Transfer Learning via 𝓁1 Regularization

Masaaki Takada, Hironori FujisawaarXiv 2020 · OpenReview

TL;DR

This paper introduces "Transfer Lasso," a novel method for adapting machine learning models to changing environments by incorporating $\ell_1$ regularization on parameter differences between source and target domains. This approach promotes sparsity in both the model parameters and the changes to those parameters, allowing for selective knowledge transfer and improved stability-plasticity balance.

What Problem Does This Paper Address?

Many real-world applications, such as user preference modeling or sensor networks, operate in non-stationary environments where the underlying data-generating process (concept drift) changes over time. Traditional machine learning models, which assume stationary data, struggle to adapt effectively in such scenarios. While existing methods for concept drift detection and adaptation exist, they often fail to capture gradual changes or can be theoretically challenging to support. Parameter-based transfer learning aims to leverage knowledge from past (source) domains to improve performance on current (target) domains, but existing methods often update all parameters even for minor data shifts, lacking interpretability and efficient adaptation. This paper addresses the challenge of building models that can effectively learn from new data while retaining relevant knowledge from previous data, even when the environment is constantly evolving.

Methods

The core contribution is the "Transfer Lasso" method, which extends the standard Lasso (least absolute shrinkage and selection operator) by adding a new regularization term. This new term penalizes the $\ell_1$ norm of the difference between the current model parameters ($\beta$) and a previously learned set of parameters ($\beta_{init}$), which serves as an initial estimate from a source domain or previous time step. The objective function for Transfer Lasso is:

$$ \min_{\beta \in \mathbb{R}^p} \frac{1}{2n} \sum_{i=1}^n (y_i - X_i^T \beta)^2 + \lambda \alpha ||\beta||_1 + \lambda (1-\alpha) ||\beta - \beta_{init}||_1 $$

Here, $\lambda$ is the regularization parameter for the standard Lasso term (controlling model complexity), and $\alpha$ is a new hyper-parameter that balances the influence of the two regularization terms. When $\alpha=1$, Transfer Lasso reduces to standard Lasso. When $\alpha=0$, it essentially uses $\beta_{init}$ as a base learner and only adapts based on the residuals. This formulation encourages sparsity in both the parameter vector $\beta$ (via the $||\beta||_1$ term) and the difference vector $||\beta - \beta_{init}||_1$, meaning only a subset of parameters are updated when adapting to new data. This selective updating allows the model to retain knowledge from the source domain for stable parameters while adapting only those that are significantly affected by concept drift. The paper also provides a coordinate descent algorithm for efficiently solving this optimization problem.

Datasets

The paper uses simulated data for concept drift and transfer learning scenarios. For real-world evaluation, it utilizes the Newsgroup message data. This dataset is described as containing 1500 examples and 913 boolean bag-of-words features. The experiment simulates concept drift by changing user interests from "space and baseball" to "medicine" over time. The data is divided into 30 batches for sequential training and testing.

Key Insights and Ideas

  • Sparsity in Changes: The most significant insight is the introduction of $\ell_1$ regularization on parameter differences. This encourages sparse updates, meaning only a few parameters are adjusted when adapting to new data. This is crucial for balancing stability (retaining learned knowledge) and plasticity (adapting to new information).
  • Implicit Stationarity Detection: The method naturally handles stationary environments. If the data remains consistent with the initial estimate, the $||\beta - \beta_{init}||_1$ term will drive $\beta$ towards $\beta_{init}$, and the model will remain largely unchanged, effectively acting as an implicit stationarity detector.
  • Theoretical Guarantees: The paper provides theoretical guarantees on estimation error bounds and convergence rates, showing that Transfer Lasso can achieve smaller estimation errors than standard Lasso under certain conditions, especially when the initial estimate is good.
  • Interpretability: The sparsity in parameter changes makes the adaptation process more interpretable. One can identify which specific features' relationships are changing over time.

Key Results

  • Concept Drift Simulation: In both abrupt and gradual concept drift scenarios, Transfer Lasso consistently outperformed "Lasso (single)" (training on only the current batch) and often balanced the performance of "Lasso (all)" (training on all historical data) by reducing estimation errors after drift occurred, while maintaining good performance during stationary periods.
  • Transfer Learning Simulation: Transfer Lasso outperformed Lasso (all) and Lasso (single) in terms of $\ell_2$-error across various "transfer rates" (probability of switching features). It also demonstrated superior accuracy in feature screening.
  • Newsgroup Data: Transfer Lasso achieved competitive or superior AUC scores compared to Lasso (all) and Lasso (single) throughout the experiment, even during stationary periods. It also exhibited stable coefficient behaviors, with some coefficients remaining unchanged, indicating effective knowledge retention.

Limitations and Future Work

  • Hyper-parameter Tuning: The method introduces an additional hyper-parameter ($\alpha$) that needs to be tuned, typically via cross-validation, which can add computational overhead.
  • Initial Estimate Quality: The performance of Transfer Lasso is dependent on the quality of the initial estimate ($\beta_{init}$). While the paper shows it can handle incorrect initial estimates, performance might degrade.
  • Future Work: The authors suggest that the framework is applicable to various parametric models, including GLMs, GAMs, and deep learning models. Further exploration of its application to these more complex models and different types of non-stationary environments would be valuable. The paper also mentions that the theoretical analysis is primarily for linear models, and extending these guarantees to non-linear settings would be an important direction.

Summary updated

Abstract

Machine learning algorithms typically require abundant data under a stationary environment. However, environments are nonstationary in many real-world applications. Critical issues lie in how to effectively adapt models under an ever-changing environment. We propose a method for transferring knowledge from a source domain to a target domain via $\ell_1$ regularization. We incorporate $\ell_1$ regularization of differences between source parameters and target parameters, in addition to an ordinary $\ell_1$ regularization. Hence, our method yields sparsity for both the estimates themselves and changes of the estimates. The proposed method has a tight estimation error bound under a stationary environment, and the estimate remains unchanged from the source estimate under small residuals. Moreover, the estimate is consistent with the underlying function, even when the source estimate is mistaken due to nonstationarity. Empirical results demonstrate that the proposed method effectively balances stability and plasticity.
BibTeX
@misc{takada2020transfer,
  author = {Masaaki Takada and Hironori Fujisawa},
  title = {{Transfer Learning via 𝓁1 Regularization}},
  year = {2020},
  eprint = {2006.14845},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.14845},
}
Figure