Loss Weights as a Hidden Tuning Cost in Multimodal Recommender Benchmarks

Recommenders that mix user behavior with item content, such as product images and descriptions, are usually trained on a sum of several losses. One loss ranks items a user clicked above items they did not. Others align image and text embeddings, pull similar items together, or regularize. Each extra loss carries a weight that someone must tune. Benchmark studies already show that tuning effort, not architecture, drives many claimed gains in this area. We propose that loss weights are a specific, measurable part of that problem. Models with more auxiliary losses have larger search spaces, so under the fixed trial budgets that benchmarks can afford they are systematically under-tuned relative to simple baselines. Automatic loss balancing during training could remove most of that asymmetry cheaply.

The core claim is testable and new. If we replace each model's hand-tuned loss weights with a cheap running-average rescaling, the tuning budget needed before a benchmark's model ranking stops changing should drop. The ranking we reach should also be closer to the large-budget ranking. This is not a proposal to add dynamic weighting to a recommender to raise accuracy. The question is whether static loss weights are a hidden degree of freedom that distorts comparisons between recommenders, and whether one inexpensive change to training removes it.

Where this sits

Two results motivate the work. Disentangling the Performance Puzzle of Multimodal-aware Recommender Systems re-ran five multimodal recommenders (VBPR, MMGCN, MGAT, GRCN, LATTICE) under one protocol in the Elliot library. The 2016 model VBPR came out second in accuracy, and LATTICE stayed first. That study kept loss weights static and searched them inside fixed bounds. It therefore cannot say how much of the spread between models comes from how hard each model's loss weights are to tune.

Transformers vs LSTMs sensitivity study found that Transformer superiority on character-level tasks vanishes under extensive random search. Its authors measured this with formal sensitivity metrics: the gap between the best and the k-th best run, the share of runs within 1% of the best, and the number of isolated performance peaks. Nobody has computed these metrics for recommenders. We reuse them as our measuring instrument.

Two further lines of work supply the method and constrain it.

  • Supplying the method. Mitigating Negative Transfer in Multi-Task Learning with Exponential Moving Average Loss Weighting Strategies divides each loss by its own running average, so every term contributes on a scale near one. It needs no per-task gradients. CoBa reweights tasks from the slopes of their validation losses and runs about 1.5x faster per epoch than GradNorm or FAMO on CodeLlama-13B.
  • Constraining the method. GradNorm balances training pace using per-task gradient norms, which is costly for large graph models. The authors of the running-average method also admit that their smoothing parameter needs tuning. That warns us the problem might simply move from the loss weights to the smoothing parameter.

Neither dynamic-weighting line was tested on recommenders, and neither measured whether it reduces sensitivity to hyperparameters.

Elliot gives us standardized splitting, filtering, search through the HyperOpt library, and significance tests. It has no adaptive loss weighting, and adding it is part of what we build. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline is the template for the conclusion we might reach. Once training protocols were held fixed, a 0.8M-parameter projection model scored 93.0% on ModelNet40, ahead of more complex point-based models.

Core Claim

Let model $m$ be trained on losses $\mathcal{L}_1,\dots,\mathcal{L}_K$. In the dynamic condition, loss $k$ is weighted by

$$\lambda_k(t) = \frac{1}{\bar{\mathcal{L}}_k(t)}, \qquad \bar{\mathcal{L}}_k(t) = \beta\,\bar{\mathcal{L}}_k(t-1) + (1-\beta)\,\mathcal{L}_k(t),$$

where $t$ is the training step and $\beta$ is a smoothing factor. This removes $K-1$ dimensions from the search space and adds one, $\beta$.

We predict three things for models with auxiliary losses (MMGCN, GRCN, LATTICE, and CM3 from CM3, whose calibrated uniformity loss is explicitly an extra term):

  1. The dynamic condition reaches near its best performance in fewer search trials than the static condition.
  2. The dynamic condition has a larger share of best-equivalent configurations.
  3. Benchmark rankings at small budgets agree better with the large-budget ranking.

VBPR is essentially a single-loss model, so it should barely change. That makes it a natural control.

Research Program

We will add three loss-weighting modes to Elliot's training loop, each selectable from the configuration file. The first is running-average rescaling. The second is a CoBa-style slope balancer computed on per-loss validation curves. The third is a static normalization, where each loss is divided once by its value at initialization.

We then run a budget ladder. For each model, dataset, and weighting mode, we draw 128 configurations by random search over a shared space. The space covers learning rate, embedding size, regularization, model-specific structural parameters, and, in the static mode only, the loss weights. From these 128 runs we compute the expected best validation score after $b$ trials by resampling:

$$E_b = \mathbb{E}\big[\max_{j \le b} s_j\big],$$

where $s_j$ is the validation score of the $j$-th sampled configuration.

This expected-best curve is the main object we compare. For each ladder we also report the Wiemerslage sensitivity metrics. Test-set metrics are computed only for the configuration selected on validation, which avoids the leak that Goyal et al. exposed in point-cloud work.

Baselines

Naive reference. Each model is trained with its loss weights set to the defaults in its original paper and never re-tuned. This is how models are often used in practice. It is the floor that any tuned condition must clear.

Strongest existing method. Each model gets full Bayesian search (tree-structured Parzen estimator search via HyperOpt, as in Elliot) over static loss weights together with every other hyperparameter, at the same 128-trial budget. The protocol follows Pomo et al.: their three Amazon categories, 4,096-dimensional visual and 1,024-dimensional text features, and splits. This baseline is strongest because it is exactly the procedure behind the current standardized result.

The reported number to check against is LATTICE's top-20 Recall of 0.1165 on Amazon Toys, against 0.0426 for MMGCN (Pomo et al.). Our re-run must first reproduce this within seed noise. On the other two categories no specific number was extracted in the file, so no reported number is available to check against there. For CM3, the file reports only an 11.95% average NDCG@20 gain over DA-MRS, not an absolute value. We therefore treat our own reproduction as that model's reference.

Dynamic weighting "beats" the strongest baseline if it meets two conditions. It must match the static condition's 128-trial test Recall@20 and nDCG@20, with no significant difference by Wilcoxon test over seeds. It must also get within 1% of its own best in at most a quarter as many trials.

Key ablation. Static initialization normalization separates adaptive reweighting over time from plain rescaling of loss magnitudes. GradNorm is included once, at a reduced budget, as a reference for both cost and accuracy.

Why not just…?

Why not just give every model a larger static search budget? This is the real threat. With enough trials, static weights should reach at least what running-average weights reach, and results on the facial benchmarks cannot rule that out for recommenders. Our claim is therefore about budget efficiency and ranking stability, not peak accuracy. The expected-best curve settles the question directly. If the static and dynamic curves coincide at every budget, the hypothesis is false, and we will report that benchmark loss-weight tuning is already adequate.

Why not just normalize each loss once at initialization? If loss magnitudes are the whole story, a single division by the initial loss value gives the benefit for free. CoBa's authors argue that tasks converge at different rates, so the right weights drift during training, but they showed this for large language model fine-tuning, not recommenders. The static-normalization ablation decides it. If it matches running-average weighting, we will state the simpler conclusion.

Why not just trust tuned simple baselines like VBPR, as Pomo et al. suggest? That advice does not address why complex models land where they do. Pomo et al. also show that LATTICE keeps a real lead, so "use VBPR" is not a complete answer. Treating VBPR as the unaffected control tells us whether gaps between models shrink, widen, or reorder once the tuning burden is equalized.

Why not just use GradNorm, the best-known balancer? Per-task gradients multiply the cost of every trial. That undercuts the very budget argument we are testing, and CoBa's timing results document this overhead. We report GradNorm's accuracy and wall-clock time so readers can see the trade-off rather than assume it.

Proposed Experiments

Datasets. Amazon Office, Toys, and Clothing, following the Pomo et al. protocol. For external validity we add Sports and Baby, following LATTICE.

Protocol variation. Every ladder is repeated under random holdout and under temporal splitting, both provided by Elliot. This tests whether dynamic weighting also stabilizes rankings across protocols, not only across budgets.

Metrics. Recall@20 and nDCG@20 measure accuracy. Item coverage and Gini index measure beyond-accuracy effects, because reweighting auxiliary losses may shift the coverage collapse Pomo et al. observed for MMGCN (about 10% on Office). Kendall's rank correlation measures agreement between the model ranking at budget $b$ and at budget 128. The Wiemerslage metrics measure sensitivity.

Decisive checks. There are four:

  1. Kendall agreement at 8 to 16 trials is higher under dynamic weighting than under static tuning, consistently across datasets and splits.
  2. The share of best-equivalent runs rises for multi-loss models but not for VBPR.
  3. A separate sweep of $\beta$ from 0.5 to 0.999 shows that the smoothing parameter is less sensitive, by those same metrics, than the loss weights it replaces. If it is not, the tuning burden has only moved, and we will say so.
  4. Static normalization does not already account for the effect.

If all four hold, benchmark studies can drop loss-weight search at negligible cost and compare architectures on firmer ground. If only the first fails, loss weights are not the hidden variable, and the field learns where not to look.