TL;DR
Multi-modal recommendation systems often struggle to balance information shared across modalities with unique, modality-specific features. This paper introduces D3ER, a framework that disentangles these features and uses a novel knowledge-distillation-enhanced gradient boosting approach to optimize them alternately. This allows the model to capture specialized, sample-oriented user preferences without the massive storage overhead typical of traditional boosting algorithms.
What Problem Does This Paper Address?
Modern recommendation systems leverage multiple modalities—like text descriptions and product images—to better understand user preferences. In multi-modal recommendation (MR), features are generally split into two categories:
- Modal-Homogeneity Discriminative Information (HOI): Information shared across modalities (e.g., a text description saying "hooded sweatshirt" and an image showing a hoodie).
- Modal-Heterogeneity Discriminative Information (HEI): Information unique to a single modality (e.g., the specific fabric texture described in text, or the exact style details visible in an image).
The core limitation of existing MR methods is that they optimize HOI and HEI jointly using a naive ensemble strategy. The authors demonstrate that this joint learning approach actually dilutes the unique predictive power of each feature type. Because different users have distinct preferences for shared versus unique modal information, a one-size-fits-all joint optimization degrades the discriminative ability of the learned representations.
Methods
To solve this, the authors propose D3ER (Disentangle and Distillation-based Dynamic Ensemble for multi-modal Recommendation), which consists of two main modules:
1. Feature Component Disentanglement (FCD)
The FCD module maps raw features into shared (HOI) and specific (HEI) components. To ensure proper separation, it applies:
- Inter-Modal Alignment: Uses instance-level contrastive learning (InfoNCE loss) alongside distribution-level alignment (Wasserstein distance via Radon transform) to ensure shared components match across modalities.
- Intra-Modal Separation: Employs a disparity loss ($L_{sep}$) based on $L_2$ distance to force the modality-specific components to remain distinct from the shared ones.
2. Knowledge Distillation-enhanced Gradient Boosting (KDBoost)
Instead of a naive joint optimization, D3ER introduces gradient boosting to MR for the first time. Recommenders for HOI, visual-HEI, and textual-HEI are trained sequentially to fit the negative gradient (residuals) of the task loss (BPR loss). This forces each specialized model to focus on the subsets of samples it is best at predicting.
Because traditional gradient boosting requires storing all historical weights—which is prohibitively expensive for deep neural networks—the authors introduce a distillation step. The cumulative model acts as a student that learns to approximate the combined scores of the previous models, keeping the storage footprint minimal. A global correction regularization term is also added to prevent the boosting process from getting stuck in greedy local optima.
Datasets
The framework was evaluated on three widely used real-world categories from the Amazon review dataset:
- Baby
- Sports and Outdoors
- Clothing and Shoes
Textual features were extracted using a pretrained Sentence-BERT model. Visual features were extracted using CLIP-ViT for the Baby and Sports datasets, and VGG-16 for the Clothing dataset.
Key Insights and Ideas
- Joint Optimization is a Compromise: Forcing a model to learn shared and unique features simultaneously dilutes sample-oriented preferences. Training them to focus on their respective strengths yields much sharper user-preference signals.
- Gradient Boosting for Deep MR: Gradient boosting is highly effective for multi-modal learning because it naturally directs models to learn from the "errors" (residuals) of the existing ensemble, leading to specialized optimization.
- Distillation Solves the Boosting Storage Bottleneck: By distilling historical ensemble knowledge into a single student network, deep gradient boosting becomes practical for resource-constrained deployment environments.
Key Results
D3ER was integrated into three popular multi-modal backbones (MGCN, DiffMM, and PGL) and consistently achieved state-of-the-art results:
- Performance Gains: When added to the MGCN backbone, D3ER achieved up to a 7.58% improvement in Recall@50.
- Consistency: It consistently outperformed 15 baseline models across all three datasets in both Recall@N and NDCG@N metrics.
- Efficiency: Computational cost analysis showed that while the distillation step adds minor training overhead, D3ER maintains an inference speed and memory footprint comparable to standard backbone models during deployment.
Limitations and Future Work
The primary limitation noted is the additional training overhead introduced by the distillation step ($L_d$). While it successfully eliminates the high storage cost of traditional gradient boosting, it requires extra backpropagation steps during training. The authors suggest that for specific real-time or highly resource-constrained applications, researchers might need to simplify or adjust this distillation step based on their efficiency requirements.
Summary updated