From Discrete IDs to Semantic Modalities in Recommendation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2487 papers515 conceptsfrozen 2026-10-04

For decades, recommender systems have operated on a "lookup table" logic. In this paradigm, every user and item is assigned a unique, arbitrary ID—a numerical placeholder that carries no inherent meaning. To recommend a movie, the system doesn't need to know it’s a "sci-fi thriller about Mars"; it only needs to know that User_402 frequently interacts with Item_909. While powerful, this ID-based collaborative filtering creates a "modality silo": the system is blind to the actual content it suggests, leading to severe "cold-start" problems where new items cannot be recommended until they accumulate interaction data.

A new research direction is dismantling these silos by replacing discrete IDs with semantic modalities. By leveraging Large Language Models (LLMs) and multi-modal encoders, researchers are building systems that "understand" items through text, images, and video, allowing recommendation to function as a generative semantic task rather than a simple pattern-matching exercise.

The Shift from IDs to Modality-based Recommendation (MoRec)

The core intuition behind this shift is that the semantic features of an item (its description, its poster, its trailer) are often sufficient to predict user interest, even without historical interaction data. Fajie Yuan and colleagues demonstrated in Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited that purely modality-based models (MoRec) can now match or outperform traditional ID-based systems.

By using foundation models like BERT or Swin Transformer to encode items directly from their raw content, these systems inherit the "world knowledge" of the pre-trained model. This allows a recommender to realize that a user who likes "space exploration" might enjoy a new documentary about NASA, even if that documentary has zero previous clicks.

Bridging Behavior and Semantics

The challenge in this new frontier is alignment. Recommendation requires two distinct types of signals:

  1. Collaborative Signals: What users actually do (the "wisdom of the crowd").
  2. Semantic Signals: What the items actually are (the content).

Frameworks like EAGER, developed by Ye Wang, use two-stream architectures to bridge this gap. One stream captures behavior patterns while the other captures item semantics, using contrastive learning to ensure the two representations "speak the same language." Similarly, the PAD (Pre-train, Align, and Disentangle) framework by Junwei Pan uses a "Rec-Anchored" alignment strategy to prevent the model from forgetting collaborative patterns when it learns semantic ones, a common pitfall known as catastrophic forgetting.

Generative Recommendation and Vector Quantization

If an item is defined by its semantics rather than an ID, how do we "retrieve" it? Researchers are increasingly treating recommendation as a generative task. In models like M6-Rec (Chang Zhou), user behavior is converted into plain text, and the model "writes" the recommendation.

To make this efficient, Yupeng Hou and Wayne Xin Zhao introduced VQ-Rec. Instead of using raw, high-dimensional text embeddings, they use Vector Quantization to map item descriptions into discrete "codes." This turns a continuous semantic space into a vocabulary that an LLM can navigate, effectively creating "semantic IDs" that are transferable across different platforms and domains.

The Multimodal Frontier

While text-based recommendation has seen massive gains, the field is currently pushing toward deep multimodality. As noted in the development of the OpenP5 platform by Shuyuan Xu and Yongfeng Zhang, most current generative models are still "text-centric."

The next step, as seen in specialized domains like fashion (Xingchen Li) or short-video (Chi Liu), involves integrating visual and auditory signals directly into the recommendation backbone. The goal is a "Universal User Representation" (Jie Gu) that doesn't just track which IDs a user clicked, but understands the aesthetic, tonal, and semantic preferences that drove those clicks across every possible medium.

Papers in this version

Showing 60 of 487 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 581 papers
  2. Version 1 · 2026-10-04 · 487 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_side_information_integration_f750687a,
  title = {side information integration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/side-information-integration/rdr_f750687a5a6f41e1b1a6704a7e022cb6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_from_discrete_ids_to_semantic_modalities_f750687a_v1,
  title = {From Discrete IDs to Semantic Modalities in Recommendation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/from-discrete-ids-to-semantic-modalities-in-recommendation/rdr_f750687a5a6f41e1b1a6704a7e022cb6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}