TL;DR
This paper introduces a novel method for generating personalized music playlists using a combination of knowledge graphs (KGs) and reinforcement learning (RL). It addresses the challenge of capturing users' long-term preferences and guiding them towards new music by leveraging KG information for efficient RL training and allowing users to customize playlist characteristics through a flexible reward function.
What Problem Does This Paper Address?
The proliferation of music streaming platforms has led to an overwhelming abundance of music, making it difficult for users to discover tracks that truly resonate with them. While existing playlist generation methods often focus on short-term preferences, they struggle to capture users' evolving tastes over time. Furthermore, these methods can inadvertently create "filter bubbles," limiting exposure to new music. This paper tackles the problem of generating playlists that not only accurately predict user preferences but also actively guide users towards novel music that aligns with their evolving tastes, all while considering long-term user satisfaction.
Methods
The core of the proposed method lies in a reinforcement learning framework enhanced by a knowledge graph.
- Knowledge Graph (KG) Integration: A KG is constructed from music tracks, artists, and genres, capturing rich, hierarchical relationships between them. This KG is then used to extract music features that represent higher-order relationships, going beyond simple metadata.
- Reinforcement Learning (RL) Framework: The problem is framed as a Markov Decision Process (MDP).
- State Representation: The state is a combination of the user's listening history and KG information, designed to capture past, current, and future preferences. This includes historical preference (using GRU), current preference (aggregating KG features of listened tracks), and future preference (predicted using an induction network).
- Action: The agent's action is to select music tracks to build a playlist.
- Reward Function: A novel, flexible reward function is introduced. It comprises:
- Prediction Reward: Measures the similarity between the generated playlist and the user's listening history.
- Guiding Reward: Incorporates user-defined weights for acoustic similarity, popularity, and novelty, allowing users to steer the playlist generation towards their specific preferences.
- Policy Gradient: The REINFORCE algorithm is used to train the RL agent's policy, optimizing for expected cumulative reward.
- Induction Network Training: A separate induction network is trained to predict future preferences, using a pairwise ranking strategy to overcome the sparsity of listening history data.
Datasets
The experiment utilized the Spotify Million Playlist Dataset [56], which contains listening histories of 1006 users. Metadata for music tracks was sourced from the Spotify API. The dataset comprises 57,880 music tracks, 1006 users, 14,973 artists, and 2517 genres.
Key Insights and Ideas
- KG for Enhanced RL: Using KG embeddings to enrich the state representation in RL significantly improves optimization efficiency and addresses the sparsity of user listening history. This is a novel application of KGs in music playlist generation.
- Controllable Playlists: The introduction of a flexible reward function, where users can tune parameters for popularity, novelty, and acoustic similarity, is a key innovation. This allows for truly personalized playlists that can guide users towards new music while respecting their existing tastes.
- Long-Term Preference Modeling: By combining historical, current, and predicted future preferences in the state representation, the model moves beyond short-term recommendations to capture evolving user tastes.
- Balancing Exploration and Exploitation: The guiding reward mechanism implicitly encourages exploration by allowing users to prioritize novelty, while the prediction reward ensures exploitation of known preferences.
Key Results
The proposed method (PM) consistently outperformed several baseline methods (CM1-CM4) in terms of prediction performance, as measured by nDCG@k and Hit Rate@k. For instance, PM-based models achieved higher nDCG@k values than all CMs.
Qualitatively, the PM-based models demonstrated the ability to generate playlists tailored to specific user preferences. For example:
- PM-P and PM-N models showed the highest values for popularity and novelty metrics (Mpop and Mnew), indicating their effectiveness in generating popular and recent music.
- PM-A models showed high acoustic similarity (Maco), catering to users who prefer smooth track transitions.
- Figure 2 visually demonstrates how adjusting reward parameters leads to playlists that meet diverse user requirements, such as prioritizing popularity and novelty.
Limitations and Future Work
The authors acknowledge several limitations and propose future research directions:
- Lack of Real-World User Evaluation: The current evaluation is offline. Future work should involve large-scale online evaluations with actual users to assess the practicality and effectiveness of the PM model in real-world scenarios.
- Incomplete Reward Function: The current reward function considers four indices. Future work could incorporate additional factors like artist and label preferences, and explore methods to automatically estimate optimal reward parameters for individual users.
- Ignoring Music Track Structure: The model does not currently consider the internal structure or length of music tracks, which can impact user satisfaction (e.g., boredom from repetitive long tracks). Future research should incorporate these aspects into playlist quality assessment.
Summary updated