A Personal Learning Apprentice

C. Lisa Dent, Jesus Boticario, John P. McDermott, Tom M. Mitchell, David ZabowskiAAAI 1992

Introduction

In the early 1990s, the field of Artificial Intelligence was dominated by "knowledge-based systems." While powerful, these expert systems suffered from a significant "knowledge bottleneck": the arduous process of manually interviewing experts to encode hundreds of "If-Then" rules. This approach was expensive, difficult to scale, and fragile in the face of changing environments.

To address this, researchers at Carnegie Mellon University developed the Calendar Apprentice (CAP). CAP was designed as a "Learning Apprentice System" (LAS)—an interactive assistant that starts with minimal knowledge and becomes increasingly useful by observing the user's daily activities. By integrating directly into the user's workflow, CAP could assimilate scheduling preferences without requiring explicit instruction.

The CAP System and THEO Framework

CAP was implemented as an extension to the Gnu Emacs text editor, assisting a secretary in managing a faculty member's calendar. The system's intelligence was powered by the THEO framework, which employed two distinct learning paradigms:

  1. ID3 (Decision Trees): This symbolic approach induced rules from training examples. For instance, if the system observed multiple meetings with graduate students occurring in a specific office, it would generalize this into a rule for future predictions.
  2. Backpropagation (Neural Networks): To complement symbolic learning, CAP used multi-layer neural networks. Categorical attributes like attendee names or departments were converted into numerical "1-of-n" encodings for training.
Table 3-2: Examples of symbolic rules automatically learned by CAP from user interactions.

As shown in Figure 4, the system was capable of learning nuanced rules regarding meeting locations, durations, and even the typical days of the week for specific types of events.

The Accuracy-Coverage Tradeoff

A central challenge for any learning apprentice is knowing when to offer advice. The authors introduced the concept of "Reliable Rules" to manage this. By ranking learned rules based on their historical performance, CAP could distinguish between high-confidence predictions and mere guesses.

This led to a critical insight: the tradeoff between coverage (how often the system provides a suggestion) and accuracy (how often that suggestion is correct). While the system might achieve moderate accuracy when forced to predict every event, its utility skyrockets when it only speaks up when it is "sure."

Key Results and Performance

The system was evaluated using real-world data collected from a CMU secretary over several months. This dataset was particularly challenging because it captured "environmental drift"—the shift from the flexible summer schedule to the rigid academic fall semester.

Comparison of prediction accuracy across different methods. Note the significant boost in accuracy for the "ID3-Reliable" subset.

As illustrated in Figure 6, CAP's automated learning eventually matched or exceeded the performance of rules that had been painstakingly hand-coded by human engineers. Specifically, while the general ID3 model achieved roughly 68% accuracy for location prediction, the "ID3-Reliable" subset reached an impressive 90% accuracy, albeit with a reduced coverage of 52%. This demonstrated that a learning apprentice could provide highly reliable assistance for a significant portion of a user's workload with nearly zero manual engineering effort.

Conclusion and Legacy

CAP proved that the knowledge bottleneck could be bypassed by systems that learn through observation. By providing a useful interface (the "enticement factor") and employing robust learning algorithms, CAP transformed from an uninformed tool into a personalized assistant. This work laid the foundation for modern proactive agents and personalized AI, highlighting that the key to a successful assistant is not just learning, but knowing the limits of its own knowledge.

Summary updated

BibTeX
@inproceedings{dent1992personal,
  author = {C. Lisa Dent and Jesus Boticario and John P. McDermott and Tom M. Mitchell and David Zabowski},
  title = {{A Personal Learning Apprentice}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
}
Figure