TL;DR
DuTri is a novel dual-branch framework designed for recommendation systems that operate across multiple business scenarios and utilize multimodal data (like text and images). By separating collaborative preference learning from multimodal fusion and linking them through triadic factorization, it overcomes the limitations of traditional "early-fusion" models to achieve state-of-the-art performance.
What Problem Does This Paper Address?
Modern recommender systems often have to juggle two complexities: multiple scenarios (e.g., a single app serving different categories like "Books" and "Movies") and multimodal content (e.g., item titles and product images).
The standard approach in Multi-Scenario Recommendation (MSR) is to take multimodal features and simply concatenate them with ID features before feeding them into a shared neural network. However, the authors discovered a surprising "performance trap": while sophisticated MSR models beat simple MLPs when using only ID data, they often underperform simple scenario-specific MLPs once multimodal data is added.
This suggests that current architectures aren't effectively balancing the "collaborative" signals (who clicked what) with the "semantic" signals (what the item actually is). Collaborative and multimodal data live in different representation spaces, and forcing them through the same parameter-sharing bottlenecks is suboptimal.
Methods
The authors propose DuTri (Dual-Branch Modeling with Triadic Factorization), which splits the task into two specialized, interconnected branches:
- Collaborative Branch: Instead of standard 2D user-item modeling, it uses Triadic Factorization (specifically CP decomposition) to model the 3-way interaction between User, Scenario, and Item. This allows the model to explicitly capture what preferences are shared across scenarios and what is scenario-specific. It also uses Residual Quantization (RQ) to compress these preferences into discrete "codewords," which helps the model find and reuse common behavioral patterns.
- Multimodal Branch: This branch handles the fusion of text, images, and IDs. Crucially, it isn't static. It uses a gating network conditioned on the user and scenario preferences learned in the first branch to decide the importance of each modality. For example, a user might care more about images in a "Fashion" scenario but more about text in a "Books" scenario.
- Cross-Branch Alignment: To ensure the two branches work in harmony, the authors use item-level contrastive learning. This pulls the collaborative item representation and the multimodal item representation closer together in the embedding space.
Datasets
The model was evaluated on two large-scale datasets:
- AntM$^2$C: An industrial benchmark from Alipay containing click logs from five different production scenarios, including rich textual and visual features.
- Amazon: A public dataset where the authors constructed three scenarios based on product categories (Sports & Outdoors, Movies & TV, and Books).
Key Insights and Ideas
- The "Early Fusion" Penalty: Simply adding multimodal features to existing MSR backbones can actually hurt performance compared to simpler models. Dedicated branches are necessary to handle the different "languages" of behavior and content.
- Preferences Should Guide Fusion: Multimodal fusion shouldn't be a "one-size-fits-all" process. By using behavioral preferences to gate the fusion process, the model can adapt to how different users value visual vs. textual information in different contexts.
- Triadic Modeling is Essential: Modeling the scenario as a first-class citizen in the factorization process (User-Scenario-Item) is significantly more effective than treating the scenario as just another input feature.
Key Results
DuTri consistently outperformed all baselines, including popular industrial models like STAR and HMoE, across both datasets.
- Accuracy: On the AntM$^2$C dataset, DuTri achieved the highest AUC (Area Under the Curve) in every single scenario.
- Efficiency: Despite its dual-branch complexity, DuTri maintains competitive inference speeds. As shown in the paper's efficiency analysis, its inference time is comparable to existing industrial solutions like AMS, making it viable for real-time production use.
Limitations and Future Work
The authors note a few areas for improvement:
- Frozen Encoders: Currently, the textual and visual encoders (like BERT or CLIP) are frozen during training to save on computation. Fine-tuning these end-to-end with the recommendation objective could potentially unlock even more performance.
- Task Generalization: The current work focuses on Click-Through Rate (CTR) prediction. Future research could explore how this dual-branch triadic approach handles other objectives like session-based recommendation or improving the diversity of recommended items.
Summary updated