Learning From Atypical Behavior: Temporary Interest Aware Recommendation Based on Reinforcement Learning

Ziwen Du, Ning Yang, Zhonghua Yu, Philip S. Yu

TL;DR

TIARec (Temporary Interest Aware Recommendation) is a novel reinforcement learning framework that shifts the paradigm of "robust" recommendation: instead of treating atypical user behavior as noise to be filtered out, it treats it as a valuable signal of temporary interest. By employing a dual-agent system—a recommender and an auxiliary classifier—the model learns to balance a user’s long-term general preferences with their fleeting, context-dependent curiosities.

What Problem Does This Paper Address?

Traditional recommender systems face a dilemma when a user deviates from their usual patterns. For example, a football fan might suddenly watch basketball videos during the NBA finals.

Most "robust" recommendation models view these outliers as noise and use filtering techniques to minimize their impact. However, the authors argue this approach has two major flaws:

  1. Loss of Personalization: Atypical interactions often represent genuine, albeit temporary, interests. Filtering them out makes the system less responsive to the user's current needs.
  2. Lack of Supervision: In the real world, we don't have labels telling us which click was a "mistake" and which was a "temporary interest." This makes it incredibly difficult for standard models to learn the difference.

Methods

The authors propose a multi-agent reinforcement learning (RL) framework called TIARec, which consists of three main components:

  • Recommender Agent ($\Pi_r$): This agent is responsible for suggesting items. It uses an attention mechanism to weigh a user's historical interactions and generate a "prototype" item embedding.
  • Auxiliary Classifier Agent ($\Pi_c$): This is the "secret sauce." It observes the recommender's actions and judges whether an interaction is atypical. It maintains two separate sets for each user: $M_t$ (normal interactions/general preference) and $N_t$ (atypical interactions/temporary interest).
  • Critic ($Q$): Using the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm, the Critic evaluates the joint actions of both agents.
The architecture of TIARec, showing the interaction between the Recommender, Classifier, and Critic.

The "Aha!" Moment in Training: During joint training, the classifier "teaches" the recommender. If the classifier correctly identifies an atypical interest that leads to a high reward (a successful recommendation), that knowledge is back-propagated to the recommender. Eventually, the recommender learns to make these "balanced" choices on its own, even without the classifier's active help during inference.

Datasets

The model was evaluated on three diverse real-world datasets:

  • Movielens-1M: A classic movie rating dataset (6,040 users, ~1M ratings).
  • Beauty (Amazon): Product reviews from Amazon's beauty category (over 1.2M users).
  • Tianchi: Data from TMALL, a major Chinese e-commerce site.
Statistics of the three datasets used in the study.

Key Insights and Ideas

  • Atypicality as Diversity: The classifier agent is rewarded when it identifies atypical items that are dissimilar to each other. This prevents it from simply labeling everything as "atypical" and encourages the discovery of niche, temporary interests.
  • Knowledge Transfer: The dual-agent setup is essentially a training wheels mechanism. The classifier helps the recommender navigate the "noise vs. interest" ambiguity during training so the recommender can stand on its own later.
  • Beyond Greedy Matching: Unlike many interactive recommenders that try to fit every single new feedback (greedy approach), TIARec uses RL to maximize long-term cumulative return, making it more robust to actual random noise.

Key Results

TIARec consistently outperformed state-of-the-art robust models (like CDAE and APR) and other RL-based recommenders across all metrics (Hit Rate, Recall, and NDCG).

  • Significant Gains: On Movielens-1M, TIARec improved HR@5 by 22.65% and NDCG@10 by 39.80% over the best baseline.
  • Capturing the "Pivot": A case study (Figure 5) showed a user who primarily liked Drama and Comedy but recently watched a few War movies. While baselines stuck strictly to Drama/Comedy, TIARec successfully recommended War movies that also had Drama elements, perfectly capturing the temporary shift.
Case study showing how TIARec balances general preference (Drama) with temporary interest (War) compared to baselines.
  • Robustness: Even when the "noise level" (simulated atypical behavior) was increased to 80%, TIARec maintained a significantly higher Hit Rate than its competitors.

Limitations and Future Work

  • Computational Overhead: Training multi-agent RL systems is more computationally intensive than standard collaborative filtering or deep learning models.
  • Cold Start: The paper relies on pre-trained embeddings (Skip-gram and PMF). The performance on entirely new users or items (the "cold start" problem) is not explicitly explored.
  • Future Direction: The authors suggest that incorporating more context (like time of day or location) could further refine the classifier's ability to distinguish between noise and temporary interest.

Summary updated

Abstract

Traditional robust recommendation methods view atypical user-item interactions as noise and aim to reduce their impact with some kind of noise filtering technique, which often suffers from two challenges. First, in real world, atypical interactions may signal users’ temporary interest different from their general preference. Therefore, simply filtering out the atypical interactions as noise may be inappropriate and degrade the personalization of recommendations. Second, it is hard to acquire the temporary interest since there are no explicit supervision signals to indicate whether an interaction is atypical or not. To address this challenges, we propose a novel model called Temporary Interest Aware Recommendation (TIARec), which can distinguish atypical interactions from normal ones without supervision and capture the temporary interest as well as the general preference of users. Particularly, we propose a reinforcement learning framework containing a recommender agent and an auxiliary classifier agent, which are jointly trained with the objective of maximizing the cumulative return of the recommendations made by the recommender agent. During the joint training process, the classifier agent can judge whether the interaction with an item recommended by the recommender agent is atypical, and the knowledge about learning temporary interest from atypical interactions can be transferred to the recommender agent, which makes the recommender agent able to alone make recommendations that balance the general preference and temporary interest of users. At last, the experiments conducted on real world datasets verify the effectiveness of TIARec.
BibTeX
@article{du2023learning,
  author = {Ziwen Du and Ning Yang and Zhonghua Yu and Philip S. Yu},
  title = {{Learning From Atypical Behavior: Temporary Interest Aware Recommendation Based on Reinforcement Learning}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2023},
  url = {https://openreview.net/forum?id=uGcohfPAca},
}

Loading related content...

Figure