Teaching With Watson

Michael WollowskiAAAI 2014 · DOI

Abstract

Introductory Artificial Intelligence (AI) courses often struggle to connect foundational algorithms with the complex, large-scale systems used in industry. This paper details a pedagogical experiment at Rose-Hulman Institute of Technology that integrated the 2013 IBM Watson "Great Minds Challenge" into an undergraduate AI course. By tasking students with the final scoring component of the Watson pipeline, the course moved beyond "toy" problems to address real-world challenges in data processing, model evaluation, and ensemble learning. The experiment resulted in high student performance and provided critical insights into the computational and pedagogical requirements of teaching modern AI.

Introduction

The gap between classic AI algorithms—such as A* search or basic decision trees—and "engineering marvels" like IBM’s Watson can be daunting for students. Often, AI is perceived as a collection of disconnected techniques rather than a cohesive, functional system. To address this, Michael Wollowski at Rose-Hulman Institute of Technology utilized the IBM Watson Technical Edition competition as a framework for an introductory AI course. The goal was to provide students with a "real-world" experience involving large datasets and multi-stage pipelines without overwhelming them or sacrificing foundational theory.

Methods

The course adopted a competition-driven approach, focusing on the final scoring component of the Watson pipeline rather than building a Question-Answering (QA) system from scratch.

  • Scoring as Machine Learning: Students developed software to distill information from hundreds of components, each providing confidence factors, into a ranked list of answers.
  • Iterative Rounds: The competition was split into two phases. Round 1 focused on broad exploration of machine learning approaches (e.g., Logistic Regression, SVMs), while Round 2 emphasized refinement and ensemble methods.
  • Literature Integration: To demystify the "black box" of Watson, students presented technical papers on sub-systems like DeepQA and temporal reasoning.
  • Tooling: The Weka Data Mining Software served as the primary platform for algorithmic experimentation.

Datasets

IBM provided three primary datasets consisting of numerical feature vectors rather than raw text. This abstraction simplified the Natural Language Processing (NLP) requirements, allowing students to focus on statistical machine learning.

  • Training Set: Approximately 378 MB, containing answer IDs, question IDs, and over 200 columns of feature data (confidence scores from various Watson components).
  • Evaluation Set: Approximately 81 MB, used for iterative testing during the competition.
  • Hold-back Set: A secret dataset used by IBM for final rankings to prevent students from over-fitting their models to the evaluation data.
A sample of the numerical feature data provided by IBM, illustrating the abstract nature of the dataset.

Key Insights and Results

The project provided several visceral lessons for the students regarding the scale of modern AI systems:

  • Computational Complexity: The intensity of the tasks led some teams to exhaust campus computing resources, triggering automated warnings from system administrators. This provided a rare lesson in the practical limits of hardware.
  • The Nature of Data: A significant point of discussion was the "abstractness" of the data. Working with 200+ columns of numbers (as seen in the provided dataset snippet) highlighted the difference between building an AI and tuning one.
  • Competitive Success: Rose-Hulman teams excelled, with one team tying for 1st place nationally and others securing 3rd and 4th.
  • Algorithm Performance: Logistic Regression and non-linear Support Vector Machines (SVMs) proved most effective. Interestingly, Neural Networks did not perform as well on this specific scoring task, confirming results previously published by the IBM Watson team.

Limitations and Future Work

The primary limitation was the pre-processed nature of the dataset, which meant students missed out on higher-level AI challenges like inference and natural language parsing. Consequently, the instructor decided to move the Watson competition to a dedicated follow-up course. This allows the introductory course to maintain its broad focus while the advanced course dives deep into the Watson framework. Future iterations aim to include raw text data to enable a wider variety of AI techniques beyond statistical classification.

Summary updated

BibTeX
@inproceedings{wollowski2014teaching,
  author = {Michael Wollowski},
  title = {{Teaching With Watson}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i3.19041},
}
Figure