TL;DR
This paper introduces a novel framework for transfer learning that reformulates hierarchical Bayesian models as undirected graphical models. This allows for convex optimization of posterior point estimates, making it more efficient and scalable for high-dimensional problems, and introduces "degree of transfer" coefficients to control how much information is shared between related tasks.
What Problem Does This Paper Address?
Transfer learning aims to leverage knowledge from related tasks to improve performance on a target task, especially when data for the target task is scarce. Hierarchical Bayesian models offer a principled way to do this by defining a prior distribution over parameters of related tasks. However, standard hierarchical Bayesian inference often involves complex, non-convex optimization problems for point estimation (like Maximum A Posteriori - MAP), and requires proper priors that can be restrictive, especially in high-dimensional settings. This paper addresses the challenge of efficiently and flexibly performing posterior point estimation in hierarchical transfer learning scenarios, particularly for high-dimensional data.
Methods
The core contribution is the reformulation of hierarchical Bayesian models into undirected graphical models (Markov Random Fields). Instead of relying on directed probabilistic models and proper priors, the authors use similarity measures between parent and child parameters and define an objective function that encourages these parameters to be close. This objective is designed to be convex for many common learning scenarios (e.g., when using convex divergence functions like L1, L2, or KL-divergence and concave data likelihoods). This convexity allows for efficient optimization using standard gradient-based techniques to find MAP estimates.
A key innovation is the introduction of "degree of transfer" (DOT) coefficients ($\lambda$). These coefficients act as weights on the similarity measures between parent and child parameters, allowing for flexible control over how much information is transferred. For instance, certain parameters might be strongly related and thus heavily influenced by their parent, while others might be more independent. The paper proposes two methods for estimating these DOT coefficients:
- Undirected Empirical Bayes Estimation: This method uses bootstrapping to estimate the variance of the difference between parent and child parameters across sampled datasets, and then uses this variance to set the DOT coefficients.
- Hyperprior-Based Estimation: This approach places a prior (specifically, an inverse-Gamma prior) on the DOT coefficients themselves, allowing them to be optimized jointly with the model parameters.
Datasets
The paper evaluates its approach on two real-world tasks:
- Mammal Shape Modeling: Using a dataset of 60 landmarks from hand-outlined images of mammals (Elephants, Bison, Rhinos, Giraffes, and Llamas). The data was reduced to 20 dimensions using PCA.
- Newsgroups Posting Classification: Using the Newsgroup dataset, which consists of 18,827 newsgroup postings from 20 distinct newsgroups. For experiments, a hierarchy over 15 classes was created, grouping them into abstract classes (Religion, Politics, Vehicles, Sports, and Computers).
Key Insights and Ideas
- Undirected Reformulation for Convexity: By framing hierarchical transfer learning as an undirected graphical model, the problem of finding MAP estimates becomes convex for many common divergence measures and data likelihoods. This is a significant advantage over traditional hierarchical Bayesian methods that often lead to non-convex optimization.
- "Degree of Transfer" Coefficients: The introduction of DOT coefficients ($\lambda$) provides fine-grained control over information sharing. This is crucial because not all parameters of related tasks are equally similar; some can be transferred more effectively than others.
- Flexibility in Priors: The framework allows for improper priors (similarity measures), removing the restrictive constraints of proper priors often required in traditional hierarchical Bayesian models, especially for high-dimensional data.
- Scalability: The approach is designed to scale well with large amounts of data, as the primary computational cost for the data objective involves computing sufficient statistics once.
Key Results
The proposed Undirected Hierarchical Bayes (HB) approach consistently outperformed baselines in both experimental settings:
- Mammal Shape Modeling: The Hyperprior and Bootstrap DOT estimation methods showed significant improvements in test log-likelihood over the regularized CV Reg baseline (independent learning), typically in the range of 5-10 bits per instance. The CV Const variant (without explicit DOT estimation but with cross-validated global weight) also showed benefits. The paper notes that a standard shrinkage approach performed poorly.
- Newsgroups Posting Classification: The Undirected HB approach achieved a better classification rate than the recursive Shrinkage approach, regularized CV Reg, and non-regularized Likelihood baselines. The Hyperprior training time was comparable to Shrinkage.
Limitations and Future Work
- Learning Hierarchy Structure: The paper acknowledges that while their framework allows for learning parameters within a given hierarchy, learning the structure of the hierarchy itself (i.e., identifying edges and which parameters transfer together) is a promising future direction, potentially by adapting methods from probabilistic abstraction hierarchies.
- Multiple Parents: The current framework assumes a single parent for each node. The authors suggest exploring hierarchies where a class can have multiple parents, which is common in real-world scenarios (e.g., WordNet, GO hierarchy).
- Convexity Guarantees: While the framework allows for convex optimization, the guarantee of convexity depends on the choice of divergence function and data objective. The paper notes that for Gaussian density estimation, the objective is not jointly concave in mean and precision, requiring iterative optimization phases.
Summary updated