DuTri: Dual-Branch Modeling with Triadic Factorization for Multimodal Multi-Scenario Recommendation

Jintao Huang, Chaohua Yang, Kailiang Hao, Dugang Liu, Xing Tang, Zhong MingarXiv 2026 · arXiv

TL;DR

DuTri is a novel dual-branch framework designed for recommendation systems that operate across multiple business scenarios and utilize multimodal data (like text and images). By separating collaborative preference learning from multimodal fusion and linking them through triadic factorization, it overcomes the limitations of traditional "early-fusion" models to achieve state-of-the-art performance.

What Problem Does This Paper Address?

Modern recommender systems often have to juggle two complexities: multiple scenarios (e.g., a single app serving different categories like "Books" and "Movies") and multimodal content (e.g., item titles and product images).

The standard approach in Multi-Scenario Recommendation (MSR) is to take multimodal features and simply concatenate them with ID features before feeding them into a shared neural network. However, the authors discovered a surprising "performance trap": while sophisticated MSR models beat simple MLPs when using only ID data, they often underperform simple scenario-specific MLPs once multimodal data is added.

Figure 1: Multimodal modeling strategies for multi-scenario recommendation: (a) MSR backbones with early fusion of multimodal features, (b) Scenario MLP with early fusion of multimodal features and scenario-specific prediction net- works, and (c) a dual-branch modeling paradigm.

This suggests that current architectures aren't effectively balancing the "collaborative" signals (who clicked what) with the "semantic" signals (what the item actually is). Collaborative and multimodal data live in different representation spaces, and forcing them through the same parameter-sharing bottlenecks is suboptimal.

Methods

The authors propose DuTri (Dual-Branch Modeling with Triadic Factorization), which splits the task into two specialized, interconnected branches:

  1. Collaborative Branch: Instead of standard 2D user-item modeling, it uses Triadic Factorization (specifically CP decomposition) to model the 3-way interaction between User, Scenario, and Item. This allows the model to explicitly capture what preferences are shared across scenarios and what is scenario-specific. It also uses Residual Quantization (RQ) to compress these preferences into discrete "codewords," which helps the model find and reuse common behavioral patterns.
  2. Multimodal Branch: This branch handles the fusion of text, images, and IDs. Crucially, it isn't static. It uses a gating network conditioned on the user and scenario preferences learned in the first branch to decide the importance of each modality. For example, a user might care more about images in a "Fashion" scenario but more about text in a "Books" scenario.
  3. Cross-Branch Alignment: To ensure the two branches work in harmony, the authors use item-level contrastive learning. This pulls the collaborative item representation and the multimodal item representation closer together in the embedding space.
Figure 3: Overview of DuTri, a dedicated dual-branch framework for multimodal multi-scenario recommendation. Triadic preferences guide item-level alignment and user–scenario-conditioned fusion. Collaborative and multimodal representations are combined with the user ID representation for scenario-specific prediction.

Datasets

The model was evaluated on two large-scale datasets:

  • AntM$^2$C: An industrial benchmark from Alipay containing click logs from five different production scenarios, including rich textual and visual features.
  • Amazon: A public dataset where the authors constructed three scenarios based on product categories (Sports & Outdoors, Movies & TV, and Books).

Key Insights and Ideas

  • The "Early Fusion" Penalty: Simply adding multimodal features to existing MSR backbones can actually hurt performance compared to simpler models. Dedicated branches are necessary to handle the different "languages" of behavior and content.
  • Preferences Should Guide Fusion: Multimodal fusion shouldn't be a "one-size-fits-all" process. By using behavioral preferences to gate the fusion process, the model can adapt to how different users value visual vs. textual information in different contexts.
  • Triadic Modeling is Essential: Modeling the scenario as a first-class citizen in the factorization process (User-Scenario-Item) is significantly more effective than treating the scenario as just another input feature.

Key Results

DuTri consistently outperformed all baselines, including popular industrial models like STAR and HMoE, across both datasets.

  • Accuracy: On the AntM$^2$C dataset, DuTri achieved the highest AUC (Area Under the Curve) in every single scenario.
  • Efficiency: Despite its dual-branch complexity, DuTri maintains competitive inference speeds. As shown in the paper's efficiency analysis, its inference time is comparable to existing industrial solutions like AMS, making it viable for real-time production use.
Efficiency comparison: DuTri offers a strong balance between parameter count and inference speed.

Limitations and Future Work

The authors note a few areas for improvement:

  • Frozen Encoders: Currently, the textual and visual encoders (like BERT or CLIP) are frozen during training to save on computation. Fine-tuning these end-to-end with the recommendation objective could potentially unlock even more performance.
  • Task Generalization: The current work focuses on Click-Through Rate (CTR) prediction. Future research could explore how this dual-branch triadic approach handles other objectives like session-based recommendation or improving the diversity of recommended items.

Summary updated

Abstract

As recommendation scenarios become increasingly diverse and multimodal information grows more prevalent, effectively modeling cross-scenario user preferences and multimodal semantics has become a critical challenge in recommender systems. When incorporating multimodal information, existing multi-scenario recommendation (MSR) methods typically treat textual and visual representations as generic input features and directly feed them into their original parameter-sharing architectures. However, this extension may not fully accommodate the different modeling requirements of collaborative and multimodal information and may not be the best modeling paradigm. Our controlled experiments show that directly adopting these architectures is not the optimal modeling approach. To this end, we propose Dual-Branch Modeling with Triadic Factorization (DuTri), a framework for multimodal multi-scenario recommendation comprising a collaborative branch and a multimodal branch. Specifically, the collaborative branch learns structured collaborative preference representations through CANDECOMP/PARAFAC (CP) decomposition of user-scenario-item interactions and residual quantization, thereby capturing cross-scenario commonalities and scenario-specific differences. The multimodal branch leverages the learned user and scenario preferences to guide adaptive modality fusion; additionally, item-level contrastive learning aligns collaborative item preference representations with multimodal representations to facilitate effective collaboration between the two branches. Extensive experiments on two real-world datasets, AntM$^2$C and Amazon, demonstrate that DuTri outperforms existing multimodal and multi-scenario recommendation baselines in overall recommendation performance, validating the effectiveness of the proposed framework.
BibTeX
@misc{huang2026dutri,
  author = {Jintao Huang and Chaohua Yang and Kailiang Hao and Dugang Liu and Xing Tang and Zhong Ming},
  title = {{DuTri: Dual-Branch Modeling with Triadic Factorization for Multimodal Multi-Scenario Recommendation}},
  year = {2026},
  eprint = {2610.36836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36836},
}
Figure