Abstract
In this paper, we show how using the Dirichlet Process mixture model as a generative model of data sets provides a simple and effective method for transfer learning. In particular, we present a hierarchical extension of the classic Naive Bayes classifier that couples multiple Naive Bayes classifiers by placing a Dirichlet Process prior over their parameters and show how recent advances in approximate inference in the Dirichlet Process mixture model enable efficient inference. We evaluate the resulting model in a meeting domain, in which the system decides, based on a learned model of the user's behavior, whether to accept or reject the request on his or her behalf. The extended model outperforms the standard Naive Bayes model by using data from other users to influence its predictions.
Introduction
In many real-world scenarios, we want to build personalized models for many different users. For example, a spam filter should ideally learn what you consider spam. However, a new user starts with zero data, leading to poor initial performance—the classic "cold start" problem. While pooling everyone’s data together is an option, it fails when users have conflicting preferences. The challenge is finding a middle ground: how can a machine automatically decide which users are similar enough to share data with, and which are different enough that their data should be kept separate?
Methods
The authors propose the Clustered Naive Bayes model, which builds on the classic Naive Bayes classifier by adding a hierarchical layer for task-level transfer learning.
Generative Clustering with Dirichlet Processes
The model assumes there is a hidden partition of tasks (users) into clusters. Tasks within the same cluster share the same model parameters. To handle the unknown number of clusters, the authors use a Dirichlet Process (DP) prior. This non-parametric Bayesian approach allows the number of clusters to grow naturally as more data is observed.
Efficient Inference with BHC
Calculating the exact probability over every possible way to group users is computationally intractable. Instead, the authors use Bayesian Hierarchical Clustering (BHC). This algorithm greedily builds a tree of tasks, allowing for a fast, approximate way to sum over the most likely groupings by restricting the search to "tree-consistent" partitions.
Experimental Evaluation
The model was evaluated on a meeting classification dataset consisting of 3,966 labeled requests from 21 users across various domains (military, academic, and industrial). The task was to predict whether a user would accept or reject a meeting request based on features like sender, duration, and location.
Key Results
- Superior Early Performance: The clustered model significantly outperformed the standard Naive Bayes when training on very few examples (1 to 32 per user). This demonstrates the power of transfer learning in data-sparse regimes.
- Convergence: After approximately 32 training examples, the standard model's performance caught up to the clustered model, suggesting that transfer learning is most valuable for getting a system "up to speed" quickly.
- Automated Expertise Discovery: The model successfully grouped users into clusters that made intuitive sense, such as grouping military personnel together before having enough data to distinguish individual quirks.
Conclusion
The Clustered Naive Bayes model provides an efficient and effective framework for task-level transfer learning. By leveraging the Dirichlet Process and BHC, the system can automatically discover and exploit similarities between users, significantly mitigating the cold-start problem in personalized applications.
Summary updated