TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling

Seungheon Doh, Keunwoo Choi, Juhan Nam

TL;DR

TalkPlay-Tools transforms Large Language Models (LLMs) from simple chat interfaces into sophisticated "conductors" for music recommendation. By allowing an LLM to call a specialized suite of tools—ranging from SQL filters to multimodal vector searches—the system can handle complex user requests like "find me a high-BPM German electronic track similar to this one" with much higher precision than generative models alone.

What Problem Does This Paper Address?

While LLMs are excellent at natural language interaction, they are often "black boxes" when it comes to recommendation. They tend to rely on internal knowledge (which can be outdated or hallucinated) and struggle to utilize structured metadata like release years, genres, or specific audio attributes (BPM, key).

Traditional recommendation systems, on the other hand, are great at filtering databases but lack the conversational nuance to understand complex, multi-turn user intents. The authors address this gap by formulating conversational music recommendation as a tool-calling problem, where the LLM acts as an agent that plans and executes a sequence of specialized retrieval and reranking steps.

Methods

The authors propose the Music Recommendation Agent, which uses a three-stage "Chain-of-Thought" prompting strategy:

  1. Planning: The LLM analyzes the user query and dialogue history to decide which tools to use and in what order.
  2. Retrieval: The agent executes tools to gather a pool of candidate tracks.
  3. Reranking: The agent uses additional tools to refine and reorder the candidates to best match the user's specific constraints.
The TalkPlay-Tools architecture showing the LLM orchestrating various retrieval and reranking tools.

The "orchestra" of tools includes:

  • Boolean Retrieval (SQL): For hard constraints like "songs over 130 BPM."
  • Sparse Retrieval (BM25): For exact text matches (artist names, song titles).
  • Dense Retrieval: Using embeddings (CLAP for audio, SigLIP2 for images, BPR for user history) to find items based on semantic similarity.
  • Generative Retrieval (Semantic IDs): Representing music as discrete "Semantic IDs" (quantized indices) that the LLM can process as text to understand multimodal properties.

Datasets

The study utilizes TalkPlayData 2, a synthetic dataset designed specifically for multimodal conversational music recommendation. This dataset includes diverse user profiles derived from the LFM-2b dataset and multimodal representations including audio features, lyrics, album art, and semantic tags. Evaluation was performed on 1,000 test conversations, each consisting of 8 turns.

Key Results and Insights

The TalkPlay-Tools approach outperformed baseline methods across the board:

  • Accuracy: It achieved a Hit@1 of 0.022, compared to 0.018 for a standard LLM+BM25 approach and 0.017 for BM25 alone. While these numbers seem small, they represent a significant relative improvement in the difficult "zero-shot" recommendation task.
  • Tool Success Rates: The system was highly successful at personalization (98.8% success rate for User-to-Item tools) and using Semantic IDs (95.8%).
  • The LLM as Orchestrator: The core shift is moving the LLM away from generating the recommendation itself and toward managing specialized retrieval engines. This allows the system to be "right" more often by using the best tool for the specific job.
Analysis of tool calling frequency and success rates at the first attempt.

Limitations and Future Work

Despite the overall success, several challenges remain:

  • SQL Complexity: The SQL tool had the lowest success rate (24.7%) because LLMs still struggle with precise syntax, often hallucinating column names or making minor coding errors that cause the query to fail.
  • Track ID Hallucination: Predicting exact, unique track IDs for item-to-item matching remains a challenge (68.4% success), as these IDs are essentially "private" strings the LLM hasn't seen during pre-training.
  • Future Directions: The authors suggest using Reinforcement Learning to help the agent learn more efficient tool-calling strategies and developing more robust "retry" mechanisms to handle failed tool calls automatically.

Summary updated

BibTeX
@inproceedings{doh2025talkplay,
  author = {Seungheon Doh and Keunwoo Choi and Juhan Nam},
  title = {{TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling}},
  booktitle = {AI4Music @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=viZfL6OROR},
}
Figure