Take a Step Back: A Dual-Granularity Raw Feature Empowered Intermediate Fusion Graph Convolutional Network for Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Wenhao Yuan, Ziyue Peng, Puzhen Wu, Shujie Li, Jian Chen, Zheng Lin, Edith C. H. NgaiarXiv 2026 · arXiv

TL;DR

DIGCN (Dual-granularity Intermediate fusion GCN) improves multimodal recommendation by using Large Language Models (LLMs) to generate coarse-grained raw features before training, bypassing the difficulty of learning complex representations from sparse interaction data. It introduces an intermediate fusion mechanism that dynamically weights different modalities during graph propagation, preserving unique modality-specific information while capturing cross-modal signals.

What Problem Does This Paper Address?

Multimodal recommender systems aim to understand user preferences by combining collaborative filtering (user-item interactions) with content features like product images and descriptions. However, they face two major hurdles:

  1. The Representation Bottleneck: Learning "multi-granularity" representations (e.g., distinguishing between a specific "floral summer dress" and the general category of "clothing") is difficult because user-item interaction data is notoriously sparse. There isn't enough supervisory signal to train these complex features from scratch.
  2. The Fusion Bottleneck: Existing models usually fuse modalities either too early (before the Graph Convolutional Network) or too late (after the GCN). Furthermore, many rely on contrastive learning to "align" modalities, which can accidentally over-compress data and erase the unique, discriminative features that make a specific modality useful for recommendation.

The authors suggest we "Take a Step Back"—instead of trying to learn these complexities during training, we should leverage the zero-shot reasoning of LLMs to prepare the data better and use a more flexible fusion architecture.

Methods

The proposed DIGCN framework consists of three core components:

  • Dual-Granularity Raw Feature Construction: Instead of asking the model to learn granularities, the authors use LLMs (GPT-4o for text) and MLLMs (DALL-E-3 for images) to generate "coarse-grained" versions of the raw data. For example, a verbose product description is abstracted into its core semantics. This results in two sets of raw features: the original fine-grained features and the LLM-generated coarse-grained features.
  • Intermediate Fusion GCN: Unlike standard models that fuse once, DIGCN performs fusion at every layer of the GCN. It treats the user/item ID (behavioral modality) as an anchor. At each layer, it calculates a dynamic cross-modality weight correction based on the cosine similarity between the anchor and the content modalities (text/visual). This allows the model to adaptively decide which modality is most important at different depths of the graph.
  • Item-Item Semantic Graph: To further combat data sparsity, the model pre-computes a frozen similarity graph between items based on their raw features. This provides a "shortcut" for information to flow between similar items, even if they don't share many common users.

Datasets

The model was evaluated on four widely used Amazon Review datasets, filtered using the standard "5-core" setting (users and items must have at least 5 interactions):

  • Baby
  • Sports and Outdoors
  • Pet Supplies
  • Office Products

These datasets vary in scale and sparsity (all >99.5% sparse), providing a robust test for recommendation performance.

Key Insights and Ideas

  • LLMs as Feature Engineers: The "aha moment" here is realizing that LLMs are better at abstracting semantics than a recommendation model trained on sparse clicks. By moving abstraction to a pre-processing step, the model can focus on the recommendation task itself.
  • The Anchor Strategy: By using the ID embedding as an anchor for fusion, the model avoids the "alignment trap." Instead of forcing text and images to look like each other, it forces them to support the behavioral signal, preserving their unique strengths.
  • Efficiency via Freezing: By pre-constructing the item-item graph and the dual-granularity features offline, DIGCN achieves better performance than state-of-the-art models while maintaining competitive training times and lower memory usage.

Key Results

  • Superiority over Baselines: DIGCN consistently outperformed 15 strong baselines (including MambaRec and LGMRec). It showed a relative improvement in Recall@20 of up to 8.81% on the Pet dataset.
  • Cold-Start Robustness: In scenarios where 20% of items were completely hidden during training (item cold-start), DIGCN maintained a significant lead, proving that the LLM-generated coarse features provide a strong semantic safety net.
  • Sparsity Resilience: The model showed the largest gains for users with the fewest interactions (0-5 training samples), validating the effectiveness of the dual-granularity approach in low-data regimes.
  • Extensibility: While DIGCN doesn't require contrastive learning, the authors showed that adding optional alignment tasks (like InfoNCE) can further boost performance, making it a flexible foundation for future research.

Limitations and Future Work

  • LLM Stability: The authors noted that while adding more granularities (3 or 4 levels) could theoretically help, LLMs become less stable and more prone to randomness at higher abstraction levels, leading to diminishing returns.
  • Computational Cost of Pre-processing: While training is efficient, the one-time cost of calling LLM APIs for every item in a massive catalog (millions of items) could be significant for production systems.
  • Future Directions: The authors suggest exploring more sophisticated ways to handle the inherent randomness of LLM outputs and investigating how to better integrate the "anchor" mechanism with other graph architectures beyond GCNs.

Summary updated

Abstract

Although multimodal recommendations have shown the ability to learn user preferences and item properties by leveraging explicit multimodal information, they remain constrained by two critical limitations: low-quality representation learning and modality fusion. For the former, existing solutions primarily focus on multi-granularity representation learning; however, we argue that training such representations is difficult due to the inherent data sparsity problem in recommendation scenarios. For the latter, existing solutions often rely on contrastive learning to align representations of different modalities to facilitate modality fusion, yet alignment-only objectives may over-compress valuable modality-specific information. To this end, we suggest \underline{``Take a Step Back"} and propose a \textbf{D}ual-granularity raw feature empowered \textbf{I}ntermediate fusion \textbf{G}raph \textbf{C}onvolutional \textbf{N}etwork (\textbf{DIGCN}) for multimodal recommendation. DIGCN leverages the multimodal understanding capabilities of Large Language Models (LLMs) to construct dual-granularity raw features directly. Furthermore, DIGCN introduces an intermediate fusion graph convolutional network equipped with dynamic cross-modality weight correction, designed to capture cross-modality signals and achieve effective modality fusion. Comprehensive experiments on four real-world datasets validate that DIGCN consistently outperforms a variety of strong baselines. Additional analyses confirm its superiority under varying levels of data sparsity and cold-start scenarios. Notably, DIGCN demonstrates high extensibility, capable of further benefiting from additional contrastive learning-based alignment tasks.
BibTeX
@misc{xu2026take,
  author = {Jinfeng Xu and Zheyu Chen and Shuo Yang and Jinze Li and Wenhao Yuan and Ziyue Peng and Puzhen Wu and Shujie Li and Jian Chen and Zheng Lin and Edith C. H. Ngai},
  title = {{Take a Step Back: A Dual-Granularity Raw Feature Empowered Intermediate Fusion Graph Convolutional Network for Multimodal Recommendation}},
  year = {2026},
  eprint = {2611.19974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19974},
}
Figure