The Adaptive Q-Network for Recommendation Tasks with Dynamic Item Space

Jianxiang Zhu, Dandan Lai, Zhongcui Ma, Yaxin PengAAAI 2025 · DOI

Introduction

Standard reinforcement learning (RL) approaches for recommendation systems typically operate under the assumption of a static environment. In these "idealized" scenarios, the set of items available during the testing phase is identical to those encountered during training. However, real-world platforms like TikTok, YouTube, or Amazon are inherently dynamic; new content is uploaded, and products go out of stock every second.

This paper introduces AdaQN (Adaptive Q-Network), a framework designed to bridge the gap between static RL models and the fluid nature of modern recommendation item spaces. By shifting the focus from learning specific item IDs to understanding item characteristics, AdaQN enables effective zero-shot recommendations for items never seen during the training phase.

The Dynamic Recommendation Challenge

Traditional RL algorithms, such as the Deep Q-Network (DQN), treat each item as a unique, independent action. This discrete approach fails in a Dynamic Recommendation Task, where the training and testing item sets are mutually exclusive. When a new item appears during testing, a standard DQN lacks a corresponding Q-value, rendering it unable to rank or recommend the item effectively.

State distribution shift between training and testing phases, highlighting the challenge of generalizing to new item spaces.

As shown in the figure above, the state distribution can diverge significantly between training and testing environments as the item pool evolves. While continuous-action RL methods exist, they often struggle to capture the long-term impact of these emerging items on user behavior.

The AdaQN Framework

The authors propose AdaQN to address the dynamic item space through three primary architectural innovations:

  1. From Items to Characteristics: Instead of learning a value function $Q(s, v)$ for a discrete action, AdaQN learns $Q(s, v)$, where $v$ represents a continuous characteristic vector (embedding) of the item. This allows the model to generalize across the item space; if the agent learns that "fast-paced action videos" yield high rewards, it can immediately value a new action video it has never encountered before.
  2. Stable Mapping via Frozen Embeddings: To generate these characteristic vectors, AdaQN utilizes a pre-trained embedding layer (e.g., from a DeepFM model). Crucially, these embeddings are frozen during the RL training process. This prevents the "moving target" problem, where simultaneous updates to both the RL policy and the item representations could lead to training instability.
  3. Characteristic-based Efficient Q-Network (CEQN): To avoid the parameter explosion common in continuous RL—where state and action vectors are often concatenated—AdaQN employs a Hadamard product (element-wise multiplication) to fuse state and item features. This lightweight architecture ensures smoother optimization and higher computational efficiency.

Experimental Evaluation

The researchers evaluated AdaQN on two prominent recommendation benchmarks: KuaiRec (short-video exposure data) and MovieLens (1M). To rigorously test the model's adaptability, they split the datasets such that the training and testing item sets had zero overlap. They further simulated various user behaviors, including "FreeB" (where users can exit the session at any time) and "NX" modes (which prohibit repeated recommendations).

Key Results

AdaQN demonstrated substantial improvements over traditional RL baselines and specialized continuous RL methods. In the dynamic task environment, AdaQN's ability to treat the test phase as a zero-shot transfer task allowed it to significantly outperform models tied to specific item IDs.

Cumulative reward performance of AdaQN compared to baselines on the KuaiRec and MovieLens datasets.

On the KuaiRec dataset under FreeB mode, AdaQN achieved a cumulative reward of 13.11, vastly exceeding the performance of standard DQN (4.58) and PPO (8.15). As illustrated in the training curves, AdaQN (represented in red) shows superior learning stability and reaches higher reward plateaus across both datasets. Even in traditional static tasks where item sets overlap, AdaQN remained highly competitive, suggesting that characteristic-based modeling is a more robust general approach for recommendations.

Conclusion and Future Work

AdaQN effectively solves the dynamic item space problem by leveraging continuous characteristic spaces and efficient network architectures. While the model's performance is currently dependent on the quality of pre-trained embeddings, its success opens new avenues for research. The authors suggest that because AdaQN learns universal characteristics rather than platform-specific IDs, it holds significant potential for cross-domain recommendations, such as applying insights from movie preferences to improve book recommendation systems.

Summary updated

BibTeX
@inproceedings{zhu2025adaptive,
  author = {Jianxiang Zhu and Dandan Lai and Zhongcui Ma and Yaxin Peng},
  title = {{The Adaptive Q-Network for Recommendation Tasks with Dynamic Item Space}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33467},
}
Figure