Logic-Calibrated Contrastive Learning for Sparse Multimodal Recommendation

Recommender systems learn what people like from a matrix of user–item interactions that is almost entirely empty. On Amazon Electronics, 99.98% of user–item pairs carry no signal at all (DOGE). The usual remedy is to add item images and text, plus a self-supervised contrastive loss (an objective that pulls two views of the same item together and pushes all other items apart). Both remedies assume that closeness is the right language for preferences. Many preference facts are not about closeness, though. An iPhone case and a Samsung case look almost identical and have near-identical descriptions, yet a user who owns an iPhone wants one and not the other. A crib and a crib mattress look nothing alike, yet buying one predicts buying the other. Image similarity gets the first pair wrong. Blanket contrastive repulsion gets the second pair wrong.

We propose to express these facts as weighted logical rules over discrete item attributes read off the images and text, such as "compatible device", "brand ecosystem", "size class" and "category". The rules enter training in two places. First, they act as a training signal on the millions of unobserved user–item pairs, in the way that a logic-based loss has acted as supervision on unlabeled examples in semi-supervised learning. Second, they decide which item pairs the contrastive loss should pull together, push apart, or leave alone. In two sentences: modality features are good at telling us what an item is, rules are good at telling us what follows from that for a given user, and in the sparse regime that second kind of knowledge is exactly what the interaction data cannot supply. Because the rules are stated over attributes rather than item IDs, they transfer to long-tail and zero-interaction items whose attributes are visible in their images and text.

Core Claim

Under extreme sparsity, part of the cross-modal gap in recommendation is not geometric misalignment but relational mismatch. Modality similarity is symmetric and graded. Preference relations include exclusions ("owns ecosystem A, so does not want accessories for B") and entailments between dissimilar items. No calibration of a similarity kernel can represent these. We claim that a small set of mined, confidence-weighted rules over modality-grounded attributes will do three things. It will improve ranking on long-tail and cold items beyond the best similarity-calibrated multimodal recommender. The gain will concentrate on "look-alike but exclusive" and "unlike but complementary" item pairs. The gain will shrink as interaction density grows. That last prediction is what separates a sparsity-specific inductive bias from a generic regularizer.

Relation to Prior Work

Three recent multimodal recommenders motivate the problem, and the closest of them constrains the design. CM3 shows that standard contrastive uniformity pushes semantically related items apart. Its fix softens repulsion in proportion to multimodal similarity. This is the closest prior work, and it fixes the half of the problem where similar items should stay close. It cannot strengthen repulsion between similar items that are mutually exclusive for a user, and it cannot attract dissimilar complements. DOGE prompts a multimodal LLM to write an intermediate "semantic modality" and builds a hypergraph from it. It reports that raw visual and text similarity often diverges from user preference, which is direct evidence for our relational-mismatch claim. However, it models the resulting relations as untyped message passing, with no notion of exclusion or of conditioning on the user. FindRec aligns ID, text and visual distributions with a Stein-kernel objective. Like the other two, it is purely distributional.

On the symbolic side, NLQ4Rec is the only neuro-symbolic recommender in the file. It encodes interaction histories as first-order logic queries answered by neural logic gates over item IDs. It uses no item content, and its authors list multimodal information as future work. Because its predicates are item IDs, it cannot generalize a rule to an item nobody has interacted with. Our predicates are attributes, so it can.

The method draws on three sources. A Semantic Loss Function for Deep Learning with Symbolic Knowledge shows that a logic-satisfaction loss on unlabeled data turns 100 labels into 98.38% MNIST accuracy, up from about 78%. We transpose this "logic as supervision on unlabeled examples" idea to unobserved user–item pairs. We do not use its exact circuit compilation, which scales exponentially with formula size. LogicCBMs shows that a differentiable logic layer over concept activations can be trained end to end from images. We borrow the pattern of grounding discrete attributes from pixels and text, but we supervise it with interactions rather than dense concept labels.

Comparing Differentiable Logics for Learning with Logical Constraints constrains the design in two ways. It finds that Łukasiewicz-style relaxations give the most informative gradients. It also documents the "vacuous truth" shortcut, where a model satisfies a rule "if A then B" by making A false. It further reports large accuracy losses when constraints are imposed carelessly, with Fashion-MNIST falling from about 90% to 75–79%. Our design answers each of these directly. Finally, KGCL shows that relational side information in recommendation is noisy and must be filtered by consistency. Hard-constraint layers such as DisjunctiveNet assume rules are correct. We therefore treat rules as soft and confidence-weighted, and we deliberately avoid hard projection.

Research Program

Attribute grounding. Each item gets a vector of attribute probabilities over a vocabulary of a few hundred discrete predicates per domain, for example compatible-device, brand-ecosystem, size class, age range and leaf category. The weak labels come from structured Amazon metadata where it exists. A small attribute head on frozen image and text encoder features is trained on those weak labels and then fine-tuned through the downstream losses, following LogicCBMs. This head is the cross-modal bridge: it turns dense visual and text features into a shared discrete vocabulary. It also lets items with missing metadata receive predicates from their images alone.

Rule mining and weighting. We do not hand-write rules. Rule templates are restricted to two forms, each short enough to evaluate in parallel on a GPU without circuit compilation. The first is user-conditioned: "user u has interacted with items carrying predicate p, therefore u prefers or does not prefer items carrying q". The second is item-pair: "items sharing p and differing in q are complements, substitutes, or exclusive". Candidate rules are scored on training interactions only. Each surviving rule $r$ gets a confidence $w_r$ equal to its precision on a held-out slice of training interactions. Rules below a precision threshold are dropped, which is our analogue of KGCL's consistency filtering.

Rule loss on unobserved pairs. Let $a_r(u,i)\in[0,1]$ be the truth of rule $r$'s antecedent for user $u$ and item $i$. Let $c_r(u,i)$ be the truth of its consequent. For a positive consequent, $c_r = \sigma(s(u,i))$, where $s$ is the model score and $\sigma$ the logistic function. For a negative consequent, $c_r = 1-\sigma(s(u,i))$. The Łukasiewicz relaxation of the implication gives

$$\mathcal{L}_{\text{rule}} = \sum_{r}\, w_r \sum_{(u,i)\in \mathcal{U}_r} \max\!\big(0,\ \mathrm{sg}[a_r(u,i)] - c_r(u,i)\big),$$

where $\mathcal{U}_r$ is a sample of unobserved pairs on which the antecedent fires and $\mathrm{sg}$ is stop-gradient. The stop-gradient is the key guard against vacuous truth. The antecedent is computed from the user's observed history and the item's grounded attributes, and the model is never allowed to satisfy a rule by lowering it. Gradient flows only into the score and, through $c_r$, into the attribute head on the consequent side.

Rule-calibrated contrastive repulsion. In the item-side contrastive loss, the repulsion weight on a negative pair $(i,j)$ is set by the rules rather than by similarity. It is reduced toward zero when a complement or substitute rule fires, raised above one when an exclusion rule fires, and left at one otherwise. This addresses the uniformity–logic tension identified in the file. Uniformity still spreads unrelated items across the embedding sphere, while logic decides the exceptions.

The full objective is the standard pairwise ranking loss plus these two terms, with weights chosen by validation. We also test a straight-through variant in the style of CL-STE, with hard thresholded attributes in the forward pass, because that paper reports large speedups over probabilistic logic programming. This variant is a secondary engineering question, not the core claim.

Baselines

The naive reference is LightGCN trained with the ranking loss on IDs plus the same frozen image and text features concatenated into the item embedding. It is the system a practitioner would build first, and it sets the floor every other method must clear.

The strongest existing method is CM3 (CM3). It is the strongest competitor because it addresses the same failure we target, contrastive repulsion of related items, and it is tuned for sparse and cold-start multimodal data. The file reports an average NDCG@20 improvement of 11.95% over DA-MRS across five Amazon and MicroLens datasets. No raw NDCG or Recall values were found on Lacuna for CM3 on Baby, Electronics or MicroLens, so we will reproduce it on our splits rather than quote numbers.

DOGE (DOGE) is the strongest representative of the "let a multimodal LLM describe the item" approach. It reports an average 7.2% gain over 12 baselines on Baby, Home & Kitchen and Electronics, but no raw metric values were found on Lacuna. SimGCL (Are Graph Augmentations Necessary?) is the uniformity-only reference.

All methods use the same frozen image and text encoders, the same embedding dimension, the same 80/10/10 splits on 5-core data, and an equal hyperparameter search budget of the same number of trials per method on the validation set. DOGE is additionally given the same attribute vocabulary we use, inserted as extra semantic text, so it is not handicapped on information.

Beating CM3 means a statistically significant improvement in NDCG@20 and Recall@20 over five seeds on at least two of the three datasets, with the gap at least as large on the bottom-degree item quintile as overall.

Four ablations isolate the key ingredient. The first, rule-as-similarity, feeds the same grounded attribute vectors into CM3's calibration as a similarity kernel. If it matches our method, the logical structure adds nothing beyond the attributes themselves. The second, rule-as-feature, concatenates the attribute vector to the item embedding. The third, no stop-gradient, uses a plain product t-norm with gradient on the antecedent and directly tests the vacuous-truth shortcut. The fourth, shuffled rules, keeps each rule's firing frequency but permutes its consequents. It controls for the possibility that any extra loss on unobserved pairs helps as a regularizer.

Why not just…?

Why not just use CM3's similarity-calibrated uniformity, with better features? This is the real threat. If attributes alone carry the signal, a similarity kernel over attribute vectors will capture it, and our method reduces to feature engineering. The case against it is structural: a symmetric similarity can only soften repulsion, and it cannot be conditioned on the user. The rule-as-similarity ablation settles the question empirically. We additionally build a diagnostic test set of look-alike exclusive pairs, defined as high image and text cosine with disjoint compatible-device or ecosystem predicates. On this set, similarity calibration should rank the wrong item higher. If the ablation matches us overall and on that set, we drop the logic claim.

Why not just prompt a multimodal LLM for descriptions, as DOGE does? DOGE itself reports that raw modality similarity diverges from preference, and better descriptions do not change the fact that relations are still modeled as untyped propagation. LLM prompting is also expensive and cannot be refreshed online (DOGE). Running DOGE with our attribute vocabulary as input tests whether the gain comes from the information or from how it is used.

Why not just add a soft fuzzy-logic penalty with no extra machinery? Our rule loss is close to a soft penalty, and that is deliberate. We are not arguing for hard solvers here: DisjunctiveNet-style projection would lock the model into noisy rules. The two things a naive penalty gets wrong, as documented in Comparing Differentiable Logics, are uninformative gradients and vacuous satisfaction. The no-stop-gradient ablation measures how much the antecedent guard matters. Its signature would be predicted attribute probabilities collapsing toward zero on antecedent predicates.

Why not just rely on SimGCL-style uniformity, which already fights sparsity? SimGCL treats every negative pair alike. CM3's results already show that this costs ranking quality in multimodal data. SimGCL is included as a baseline, and the degree-stratified results will show whether uniformity alone helps cold items.

Proposed Experiments

The main benchmarks are Amazon Baby, Amazon Electronics (99.98% sparse) and MicroLens, reporting Recall@20 and NDCG@20 overall, by item-degree quintile, and on items held out with zero training interactions. Amazon-Book serves as a unimodal sanity check in which knowledge-graph relations are treated as predicates. There, KGCL reports Recall@20 of 14.96% (KGCL), which gives a concrete number our rule loss on top of LightGCN should approach or exceed. Beyond ranking, we measure the test-set violation rate of high-confidence rules and accuracy on the look-alike-exclusive and unlike-complement diagnostic pair sets.

Three checks decide the proposal. First, gains over CM3 must concentrate in the long tail and on the diagnostic pair sets, not in the head. Second, in a density sweep that subsamples training interactions from 100% down to 10%, the advantage over CM3 should grow as data thins and shrink toward zero at full density. If it is flat, the rules are acting as a generic regularizer, not a sparsity-specific prior. Third, the rule-as-similarity and shuffled-rules ablations must both fall clearly short. Failing the first or third check would mean the logical form of the knowledge is not what matters, and we would report that negative result.