Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations

TL;DR

Traditional video recommenders are blind to high-level semantics like humor, parody, and cultural context because they rely on low-level pixels and audio spectrograms. This paper introduces a zero-finetuning, plug-and-play framework that uses off-the-shelf Multimodal Large Language Models (MLLMs) to translate raw video and audio into rich textual descriptions. When fed into standard recommendation algorithms, these MLLM-generated captions yield massive performance boosts—up to a 60% relative gain on short-video recommendation tasks.

What Problem Does This Paper Address?

When you watch a short clip on TikTok or YouTube Shorts, what makes it resonate with you isn't just the colors on screen or the volume of the audio; it's the deeper context. Is a 30-second clip just "someone dancing on a rooftop," or is it an ironic parody of a 1990s superhero trope?

Existing video recommendation systems struggle with this distinction. They typically rely on creator-written titles (which are often clickbait) or low-level features extracted by specialized visual and acoustic encoders (like VideoMAE or CLAP). While these encoders capture motion and sound frequencies, they lack the "world knowledge" required to understand intent, style, and cultural references. The authors bridge this gap by leveraging MLLMs as on-the-fly knowledge extractors to make recommendation systems more intent-aware.

Methods

The authors propose a simple, backbone-agnostic framework that requires zero finetuning of the heavy MLLM components, making it highly scalable and cheap to deploy.

1. Video Captioning: They use a frozen MLLM (Qwen-VL) to analyze video frames and generate detailed, context-aware textual descriptions.
2. Audio Processing (Two-Stage): Since unified audio-visual MLLMs are still maturing, the authors decouple the audio. First, they use Whisper to transcribe speech and classify background sounds. Second, they feed these transcriptions into Qwen-Audio to generate intent-aware descriptions of the soundtrack (e.g., "upbeat soundtrack with a lighthearted tone").
3. Recommendation Integration: The video and audio text descriptions are concatenated and passed through a state-of-the-art text encoder (BGE-large) to create embeddings. These rich embeddings are then plugged directly into standard recommendation architectures: a classic Two-Towers model and a generative/sequential model (SASRec).

Expand

Datasets

The framework was evaluated on MicroLens-100K, a large-scale, content-driven dataset collected from a real-world short-video platform. It contains raw, full-length video files and metadata, encompassing 100,000 users, 19,738 video items, and over 719,000 interactions. To mimic a true production environment, the authors used a global time-based split with a 1-day rolling window for evaluation.

Key Insights and Ideas

• Pixels vs. Meaning: Pixels show what happens on-screen, and titles reflect what the uploader hopes will get clicks. MLLM-generated text captures why viewers care by explicitly identifying tone, parody, and cultural cues.
• Text as a Universal Interface for Audio: Converting raw audio waveforms into text summaries actually preserves abstract concepts like mood and theme better than traditional spectrogram-based embeddings.
• Diminishing Returns on Model Scaling: Surprisingly, upgrading to a larger MLLM backbone (Qwen-VL 7B) or a larger text encoder did not yield measurable improvements in recommendation accuracy. This suggests that once an MLLM provides a coherent, video-level description, the baseline model already captures the most relevant information for the recommender.

Key Results

Replacing raw features with MLLM-generated text summaries consistently and significantly outperformed traditional baselines across both recommendation models:

• Audio Boost: Replacing raw background audio features with MLLM text boosted the Two-Towers model's HitRate@10 (HR@10) from 0.0253 to 0.0405 and nDCG@10 from 0.0130 to 0.0214—a massive ~60% relative gain. SASRec saw a similarly strong 35% relative gain.
• Video Boost: Using scene-level MLLM captions instead of raw video frames boosted Two-Towers HR@10 by 24% (from 0.0393 to 0.0489) and outperformed creator-written titles by 4% to 18%.

Limitations and Future Work

While powerful, the framework currently relies on a decoupled approach to handle audio and video separately due to the lack of a mature, unified audio-visual-text MLLM at the time of the study. This modular design prevents the system from capturing simultaneous audio-visual cues natively.

Additionally, to keep computational costs manageable, the authors restricted their focus to short-form content (videos under 4 minutes). The authors note that emerging "omni-MLLMs" (such as Qwen Omni or Vita) that natively model vision, audio, and text simultaneously represent the logical next step for this framework.

@misc{nadai2025describe,
  author = {Marco De Nadai and Andreas Damianou and Mounia Lalmas},
  title = {{Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations}},
  year = {2025},
  eprint = {2508.09789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.09789},
}