Machine Learning of User Profiles: Representational Issues

AAAI 1996

TL;DR

This paper explores how to build better machine learning models for user profiles in information filtering systems. The key insight is that using a generalization hierarchy (like a thesaurus) to represent document features, alongside traditional keyword features, significantly improves profile accuracy and intelligibility. This hybrid approach leads to more effective and trustworthy personalized content recommendations.

What Problem Does This Paper Address?

In today's information-rich digital landscape, finding relevant content is a major challenge. Personalized information filters, which learn user interests to recommend articles, are crucial. However, existing systems often struggle to capture evolving user needs effectively and to explain why certain content is recommended. This lack of transparency can erode user trust. The paper addresses the gap in creating user profiles that are not only predictively accurate but also comprehensible to the user, aiming to bridge the divide between machine learning performance and human understanding.

Methods

The core of this research lies in exploring different document representations for machine learning models. They experimented with:

  • TFIDF (Term Frequency-Inverse Document Frequency): A standard technique in information retrieval that represents documents based on the importance of individual keywords.
  • Subject Field Coder (SFC): This method uses a thesaurus to assign subject categories to text segments. Crucially, they leverage a generalization hierarchy within the thesaurus, allowing the system to understand broader concepts (e.g., "water sports" encompassing "scuba" and "kayaking").
  • People, Organizations, and Locations (POLs): Features extracted using a name-tagger that identifies and categorizes named entities within text, along with their attributes.
  • Hybrid Representation: Combining TFIDF, SFC, and POL features.

They then evaluated two machine learning algorithms:

  • AQ15c: An algorithm that generates disjunctive normal form (DNF) expressions and can utilize generalization hierarchies directly in its rule construction.
  • C4.5-Rules: An algorithm that generates rules from decision trees, which they extended to incorporate the generalization hierarchy.

The goal was to see how these representations and algorithms perform in learning user profiles, with a focus on predictive accuracy, precision, and recall.

Datasets

The experiments were conducted using two datasets:

  • USMED (User-Selected Medicine): Derived from the Colorado Springs Gazette Telegraph newspaper. A user manually set up an initial profile for "Medicine in the US," and provided feedback on recommended articles. This dataset was used for training and validation with a 70/30 split.
  • T122 (TREC-92): A standard test collection from the Wall Street Journal (1987-92) for the topic "RDT&E of New Cancer Fighting Drugs." This dataset was used with a 10-fold cross-validation approach.

Key Insights and Ideas

  • Generalization is Key for Intelligibility: The ability to infer broader concepts (like "water sports" from specific activities) is not just about accuracy; it's crucial for making profiles understandable to users. This helps build trust and allows users to validate or correct the system's understanding.
  • Hierarchical Features Boost Performance: When a relevant generalization hierarchy is available (like the SFC), summary-level features derived from it can outperform traditional keyword-based (TFIDF) features, even when the hierarchy isn't very deep.
  • Hybrid Representations Offer Robustness: In cases where a domain-specific hierarchy isn't readily available or as effective, combining different feature types (TFIDF, POL, SFC) leads to the most accurate profiles, especially for algorithms like C4.5.
  • Intelligibility Matters: The paper demonstrates that learned rules can be quite simple and interpretable, especially when using specific feature sets. For example, C4.5 generated concise rules based on POL tags for the USMED dataset.

Key Results

  • Superior Performance with Hierarchies: The SFC feature set, leveraging the generalization hierarchy, achieved the highest predictive accuracy, precision, and recall on the USMED dataset, outperforming TFIDF alone.
  • Hybrid Approach Wins When No Hierarchy: For the T122 dataset, where a domain-specific hierarchy might be less impactful or available, the hybrid representation (combining TFIDF, POL, and SFC) performed best for both AQ15c and C4.5-Rules.
  • AQ15c and C4.5-Rules are Effective: Both learning algorithms, when provided with the appropriate representations, demonstrated strong performance. AQ15c directly utilized the hierarchy, while C4.5 benefited from an extended attribute set incorporating it.
  • Comparison to Rocchio: The machine learning approaches significantly outperformed a modified Rocchio algorithm (a traditional relevance feedback method) in terms of predictive accuracy and average precision/recall. For instance, the best AQ15c (SFC) achieved 0.78 predictive accuracy on USMED, compared to Rocchio's 0.49.

Limitations and Future Work

  • Hierarchy Depth and Balance: The paper notes that the SFC hierarchy had limited depth, requiring manual extension for the "medicine" category. They suggest that the balance and depth of the hierarchy can impact performance.
  • Dummy Values: Extending the hierarchy sometimes led to dummy values for certain attributes, which might affect learning.
  • Future Work: The authors plan to track learner performance over time as users interact with the IDD News Browser, allowing for profile correction. They also aim to explore constructive induction to automate the search for improved representations. Additionally, they suggest that techniques for augmenting thesauri with corpus-derived terms could further enhance performance.

Summary updated

BibTeX
@inproceedings{1996machine,
  title = {{Machine Learning of User Profiles: Representational Issues}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}
Figure