Cross-Modal Residuals for Training MLLM-based Sequential Recommendations
Lists
Only you see this
Abstract
RIFT-Rec models the relationship between image and text similarities as a bidirectional regression problem. Each unordered catalog pair supplies one similarity from a frozen visual encoder and another from a frozen text encoder. Median centering and a strictly positive scale standardize the two coordinates. Robust monotone fits then predict each coordinate from its counterpart using training pairs without recommendation labels. Large residuals identify relationships that depart from the usual correspondence between the two views. Averaging the two residual magnitudes yields an order-invariant conflict score. Clipping, a positive offset, and global rescaling convert these scores into bounded comparison coefficients. These coefficients remain fixed while an MLLM learns to rank next-item candidates from interaction histories, images, and descriptions. The training criterion adds a nonnegative multiple of weighted pairwise logistic risk to full-candidate cross-entropy after an initial cross-entropy fit. Every observed next item is contrasted with every alternative. At fixed contexts with positive label probabilities, coefficient symmetry ensures that both terms share the unrestricted-score population minimizers of cross-entropy. Planned diagnostics will examine calibration residuals, weight concentration, and sensitivity to catalog shifts; the population result makes no claim about finite-model optimization.An exponential kernel and symmetric normalization then convert the absolute differences between these normalized similarities into fixed positive comparison weights.
BibTeX
@misc{cai2026cross,
author = {Pengjie Cai and Ye Yuan and Haitian Liu and Chong Liu and Bo Tang},
title = {{Cross-Modal Residuals for Training MLLM-based Sequential Recommendations}},
year = {2026},
url = {https://openreview.net/forum?id=H56ixvQGUz},
}Loading related content...