Demand-Aware Hardening of Text-Based Recommenders Against Item-Text Edits

Recommenders are moving away from opaque item ID numbers toward models that read item titles and descriptions. Text lets a system recommend brand-new items and carry knowledge across catalogs. It also opens a door that ID-based systems never had: whoever writes an item's text can now steer how the model represents that item. A seller who rewrites a product title can, in principle, push it into many users' top-ten lists. No poisoned user accounts or fake clicks are needed. Defending against this is expensive the usual way. Adversarial training, which repeatedly searches for the worst-case input and trains on it, is especially costly for text, because each search over word substitutions takes thousands of model evaluations.

This note proposes a cheap defense built on one observation about how promotion attacks work. A text recommender scores a user–item pair by the dot product between a user vector and an item vector. So an edit to an item's text only matters through how it moves the item vector along directions that many users' vectors point in. Those "demand directions" can be computed once from the user population. We propose to penalize, during the normal fine-tuning of a compact recommendation language model, only the sensitivity of item vectors along those few directions. This costs about one extra gradient step per batch instead of a discrete search. The bet is that it buys robustness to adversarial item-text edits at a much smaller loss of clean and cold-start accuracy than spreading robustness over all directions.

Core Claim

Compact language models specialized for recommendation, such as LLM2Rec, are vulnerable to small adversarial edits of item text. LLM2Rec turns Qwen2-0.5B into an item embedder through collaborative fine-tuning on next-item title prediction plus item-level contrastive learning, and it beats much larger general embedders. We hypothesize that this specialization makes the vulnerability worse than in a generic embedder, because the model learns to rely heavily on a few title tokens.

The vulnerability can be sharply reduced by a targeted regularizer. The regularizer restricts robustness to the low-dimensional subspace that controls exposure across users. It needs no inner discrete search, and it costs a small, measured multiple of clean training. The new element is that the threat (promotion through content), the cost constraint (no text-space min-max), and the specialized-model setting together give the inner maximization a structure the defense can exploit cheaply.

Positioning Against Prior Work

Several papers motivate the gap without filling it.

The move to text and semantic identifiers is well established. P5 recasts recommendation as text-to-text. IDGenRec replaces numeric IDs with textual identifiers. DIGER learns discrete semantic IDs end to end from content. All of these evaluate only clean accuracy, and none asks what happens when the content itself is adversarial. LLM2Rec explicitly leaves data noise as future work.

The robustness work in recommendation addresses a different threat:

  • CAT scales continuous perturbations on ID embedding tables according to each interaction's cascade influence. It is limited to SASRec and GRU4Rec, and the authors flag its overhead.
  • LoRec defends against fake-user poisoning by calling Llama2 on every user sequence to reweight suspicious users.
  • AC-TSR calibrates attention in ID-based transformers.

All three target the user side or ID embeddings. Our threat model leaves user histories untouched and edits only item text, which none of them covers.

Three cheap-robustness methods supply candidate mechanisms and serve as our strongest generic competitors:

  • Robustness Tokens trains about ten extra tokens on a frozen vision transformer. The authors note that extending it to language is open.
  • Dynamic Attention masks abnormally high-attention tokens at inference without retraining.
  • EPIC filters gradient-space outliers among poisons.

None of these uses the structure of a ranking objective. Our proposal differs by deriving where to be robust from the user distribution.

One constraint shapes the design. The research file notes, following Towards Improving Adversarial Training of NLP Models, that continuous defenses do not translate cleanly to discrete tokens. Our regularizer acts in the continuous token-embedding space while the attack is discrete. The experiments must therefore test against discrete, defense-aware attacks, not only the continuous surrogate.

Research Program

Threat model. An attacker controls the text of one target item and may change at most $b$ title tokens, either by substitution or insertion. The goal is to raise the item's hit rate across all users. We consider two attack types:

  • A gradient-guided greedy token-substitution attack, which is white-box, defense-aware, and the strongest case.
  • A fluent rewrite produced by prompting an LLM to "make this title appeal to more shoppers" while preserving meaning, which is black-box and the realistic case.

Measurement study. Before building anything, we measure the vulnerability across three model families under identical attacks:

  • An ID-based model (SASRec), which is immune to text edits by construction.
  • A generic text embedder plus sequence model.
  • LLM2Rec.

Semantic-ID generators are included as an extra victim class. We reproduce a two-stage RQ-VAE tokenizer and DIGER, measure how many edits flip an item's code sequence, and measure what that does to exposure.

This study alone answers whether specialization amplifies the vulnerability. It is publishable even if the defense disappoints.

Defense. The defense rests on two definitions and two equations.

Let $q_u$ be the user vector the model produces from user $u$'s history, and $e(x)$ the item vector the encoder produces from item text $x$. If an edit moves an item vector by $\Delta$, user $u$'s score changes by $q_u^\top\Delta$. Summed over users, the squared change is

$$\sum_u (q_u^\top \Delta)^2 = \Delta^\top M \Delta, \qquad M = \tfrac{1}{N}\textstyle\sum_u q_u q_u^\top ,$$

where $N$ is the number of users. Edits matter most along the top eigenvectors of $M$. These eigenvectors include the mean user direction, because $M$ is not centered. Let $U_k$ hold the top $k$ eigenvectors, recomputed every few hundred steps from a sample of users. We add the regularizer

$$R(x) = \big\| U_k^\top \big(e(x+\delta^\star) - e(x)\big) \big\|^2, \qquad \delta^\star = \epsilon \,\frac{\nabla_\delta R}{\|\nabla_\delta R\|}\Big|_{\delta \sim \text{small random}} ,$$

where $\delta$ perturbs the token embeddings of $x$ and $\epsilon$ is the perturbation radius. $\delta^\star$ is found with a single gradient step. The training loss is LLM2Rec's objective plus $\lambda R$, with weight $\lambda$, applied during the collaborative fine-tuning stage. The cost is about one extra forward and backward pass on item encodings per batch.

The design choice under test is the projection onto $U_k$. Without it, $R$ becomes ordinary one-step embedding-space adversarial training.

Cost accounting. For every method we report GPU-hours relative to clean LLM2Rec training. Every method is run on the same hardware with the same training steps, so "cheap" is a measured quantity rather than an assertion.

Baselines

Undefended LLM2Rec. This is the do-nothing reference: LLM2Rec trained exactly as published on Qwen2-0.5B. It sets clean accuracy and the attacked baseline. The only reported figure on Lacuna is a 15% relative in-domain improvement over prior embedders across Games and Arts, reported in LLM2Rec. We found no reported result on Lacuna for LLM2Rec under item-text attacks, so we establish that number ourselves.

SASRec. This is the ID-based reference. It cannot be attacked through text, but it fails on cold items. It marks the robustness ceiling and the cold-start floor, which together define the trade-off we aim to escape.

Discrete adversarial fine-tuning (expensive gold standard). LLM2Rec's collaborative fine-tuning is run with LoRA on a pool of attacked titles. The pool is regenerated by the greedy substitution attack every epoch. This is the method a skeptic would trust most. It receives the same LoRA rank, the same number of training steps, and a tuning grid of equal size. Its attack cost is reported separately.

Uniform one-step embedding adversarial training. This is the key ablation: identical to our method but with $U_k$ replaced by the identity. It isolates whether demand-aware projection matters.

Robustness Tokens and Dynamic Attention. Robustness Tokens is ported to the item encoder, with ten prefix tokens trained on a frozen backbone using its dual clean-invariance and correction loss. Dynamic Attention is applied training-free at inference. These are the strongest existing cheap defenses, though neither has been run in this setting. In their own domains they report the following:

  • Robustness Tokens: white-box ImageNet accuracy for DINOv2-G recovered from 0.3% to 63.1%, with clean accuracy at 85.2% versus 85.6% (Robustness Tokens).
  • Dynamic Attention: up to 33% better resistance than defensive dropout against TextFooler and PWWS (Dynamic Attention).

Cosine scoring with normalized item vectors. This simple structural fix removes any attack that works by inflating vector length.

Criterion for winning. Our method beats the field if two conditions hold. First, it matches discrete adversarial fine-tuning in reducing target-item promotion to within its seed variance, at a small fraction of that method's GPU-hours. Second, it beats uniform one-step training and Robustness Tokens on cold-start Recall@10 when attack reduction is held equal.

Why not just…?

Why not just normalize item vectors and score by cosine? If promotion attacks work mainly by lengthening item vectors, normalization kills them for free. In that case the demand subspace adds little. This is a real threat to the proposal. We will run the cosine variant first. We will also report how much of the attacked score gain comes from changes in vector length versus changes in angle. The regularizer can be applied on the unit sphere as well, so the two combine. If normalization alone closes most of the gap, that becomes the headline result.

Why not just do adversarial fine-tuning with LoRA on offline attacked titles? This is the method we expect to be most robust, and it is our gold standard rather than a straw man. Its weakness is cost. The greedy discrete search must be rerun as the model changes, or the attacks go stale; Towards Improving Adversarial Training of NLP Models documents that text attacks need thousands of forward passes per example. If a stale, one-time pool turns out to give the same robustness, the cost argument weakens. We will test exactly that variant.

Why not just use a training-free defense like Dynamic Attention? It requires no training. However, it was designed for adversarial tokens that grab attention inside long inputs. Item titles are short, and a manipulative but fluent rewrite may not produce any attention outlier. The fluent-rewrite attack is the test case. We expect Dynamic Attention to help against the greedy attack and fail against rewrites. If it handles both, it is the better choice.

Why not just blend in ID embeddings for warm items? This makes established items as robust as SASRec. But new listings, which are exactly the items a promoting seller controls, have no ID history. We report cold and warm splits separately so this baseline's apparent robustness is not inflated by warm items.

Proposed Experiments

Datasets. We use Amazon Games and Arts, matching LLM2Rec and LoRec, Amazon Beauty for the semantic-ID victims, and Goodreads for out-of-domain checks. Items are split into warm items and a held-out set of cold items.

Metrics. Clean quality is measured by Recall@10 and NDCG@10, overall and on cold items. Attack effect is measured by target-item hit rate at 50 (T-HR@50, following LoRec) over 100 randomly chosen cold target items per dataset, at edit budgets $b \in \{1,3,5\}$. We also report GPU-hours.

For reference only, since the threat is different: LoRec reports T-HR@50 reductions averaging 78.81% on Games and 51.63% on Arts against user poisoning (LoRec). For the semantic-ID victims, DIGER reports Recall@10 of 0.0683 versus 0.0610 for two-stage tokenization on Beauty (DIGER). Our reproductions must land near those numbers before attacked results are trusted.

Decisive checks. There are four.

  1. Vulnerability exists and is amplified by specialization. At $b=3$, LLM2Rec's target hit rate rises by a larger multiple than the generic embedder's. If neither model is meaningfully promotable, the project stops at a negative but useful measurement.
  2. Projection matters. At matched attack reduction, our method keeps more cold-start Recall@10 than uniform one-step training. We also sweep $k$. Near-flat behavior as $k$ approaches the full dimension would falsify the demand-subspace idea.
  3. Robustness transfers from the continuous surrogate to discrete attacks. Robustness holds against a greedy substitution attack that is given the regularized model's gradients, and against fluent LLM rewrites. A defense that only blocks the surrogate perturbation does not count.
  4. Cost. The added GPU-hours, reported against discrete adversarial fine-tuning, stay under one extra pass per batch. If the method loses to discrete adversarial training on robustness by more than seed variance, we will report it as a cheaper, weaker point on the cost–robustness curve rather than a replacement.