Embedding models that match records, such as two product listings that describe the same laptop, learn by pulling matching items together and pushing non-matching ones apart. They learn little when the non-matching examples are obviously different. A listing for a laptop is easy to tell apart from a listing for shoes. The hard part is telling an i5 laptop from an otherwise identical i7 one. Large language models (LLMs) can now write such near-miss negatives on demand: copy the listing and change exactly one specification. On the WDC product benchmark this already improves top-5 retrieval. Yet the hardest target, getting the correct match ranked first, barely moves. We think the reason is a mismatch in timing. The LLM writes all of its negatives once, before training, and with no idea of what the model can currently tell apart. Early in training those one-edit negatives are too hard and destabilize learning. Later, many become easy and stop contributing.
We propose to have the LLM write, for each anchor listing, a short ladder of negatives that differ from it by three, two, and then one specification, along with a very-hard final rung (a lookalike model from the same brand family). During training, the learner picks, for each anchor, the rung that currently sits just inside the safety margin of the triplet loss. The ladder turns a fixed synthetic pool into a difficulty dial that follows the model's own geometry. Every anchor then keeps an informative negative throughout training, without the corpus-mined "stabilizer" negatives that prior synthetic-negative methods needed. A second, lighter mechanism closes the loop: each epoch, we spend the remaining generation budget on the specification types where the model still ranks a wrong product first.
Background and the gap
The method builds on the triplet loss from deep metric learning. The loss takes an anchor $a$, a matching positive $p$ and a non-matching negative $n$, and penalizes $\max(0, d(a,p) - d(a,n) + \alpha)$, where $d$ is embedding distance and $\alpha$ is a margin. FaceNet established two things about this loss. Random triplets mostly give zero gradient. Training on only the hardest negatives early on drives the model into poor solutions, while "semi-hard" negatives, which are farther from the anchor than the positive but still inside the margin, train stably.
The isometric approximation analysis gives this result a mechanism. It proves that the embedding diameter is at least the initial spread minus the initial loss, so a spike of ultra-hard negatives can collapse every input onto a single point. Curriculum schedules (Yu et al.) and group-restricted mining (Deep Triplet Quantization) manage difficulty, but only among examples that already exist in the data. These papers motivate and constrain our design: the right amount of hardness depends on where the model is in training, and too much hardness is dangerous.
Two papers supply the generation step and also expose the gap. LLM-HN prompts GPT-4o-mini to list a product's discriminating attributes and perturb exactly one, then trains a bi-encoder with triplet loss on WDC LSPC. It finds an Acc@1 ceiling near 4.5% that neither 10x more data nor 5x more training breaks. Its synthetic negatives lift Acc@5 and mean reciprocal rank (MRR, the average of one over the rank of the correct match) significantly, but give only a non-significant directional lift at Acc@1. Its gains also vanish in attribute-sparse categories such as Cameras and Watches. SyNeg generates hard negative passages with GPT-4o for dense retrieval. It reports a "stability paradox": training on synthetic negatives alone is unstable because they sit too close to the positives. It fixes this heuristically, mixing roughly 0.7 synthetic negatives per positive with BM25-mined corpus negatives.
Both methods generate offline and ignore the learner's distances. CemiFace is the closest precedent for targeting a similarity level during generation. It conditions a diffusion model (an image generator that works by gradually denoising random noise) on a requested cosine similarity, so that it produces semi-hard face images. Those images are positives of the same identity, and the similarity is enforced by a frozen face recognizer, not by the model being trained. Latent-space synthesis methods, LoOp and SSCL, place hard negatives precisely in embedding space. Their outputs are not real inputs, though, so they cannot encode "same laptop, different CPU."
Core Claim
Our claim is that how hard each synthetic negative is matters more than how many there are. Controlling hardness per anchor and per training step, by choosing among LLM negatives of graded edit depth, will do two things. It will resolve the stability paradox without corpus mixing, and it will turn LLM-HN's Acc@5 gains into a significant Acc@1 gain on attribute-dense WDC categories.
The selection rule is simple. Let $n_1, \dots, n_K$ be the ladder for anchor $a$, ordered from most edits (easiest) to fewest (hardest). At each step we choose
$$k^\* = \max\{k : d(a,p) < d(a,n_k) < d(a,p)+\alpha\},$$
which is the hardest rung that is currently semi-hard. If no rung qualifies, we fall back to the easiest rung that still violates the margin. The ladder's value comes from what happens early in training. A naive bi-encoder will typically place a one-edit negative closer to the anchor than a differently worded true match from another shop, which makes that negative "hardest" and risky. A three-edit negative, by contrast, falls inside the margin.
Research Program
Generation. For each anchor offer, an LLM lists the attributes that discriminate it from other products and writes one negative per rung. Each rung changes a nested set of attributes, so rung 1's edits contain rung 2's, and so on. The top rung is the one-edit negative, plus the same-brand lookalike. We run this with GPT-4o-mini, to stay comparable with LLM-HN, and with an open-weight Llama 3 model, addressing LLM-HN's untested lock-in. A lightweight check verifies that each edit is actually present in the output and that no other attribute changed.
False negatives. The prompt also asks the LLM to flag attributes whose change may not break a match, such as color in some categories. Negatives that alter only flagged attributes are held out. A human audit of 300 sampled negatives per category estimates the false-negative rate under each variant.
Selection inside training. The ladder is embedded on the fly with the current encoder, and $k^\*$ is chosen per anchor. We log three quantities across training: the fraction of anchors with an active triplet, the mean chosen rung, and the embedding diameter. The diameter directly tests the collapse bound from the isometric analysis.
Error-driven top-up. After each epoch, we tally which attribute type differs between each anchor and the wrong product ranked first. The next generation round, spending a fixed fraction of the total token budget, writes new one-edit negatives targeting those attribute types. This is the closed-loop element. It is ablated separately because it is the most expensive part.
Baselines
The naive reference is the LLM-HN setup without synthetic data: a bi-encoder trained with triplet loss on real WDC LSPC pairs, using in-batch negatives, the same encoder, the same steps, and FaceNet-style semi-hard selection among in-batch candidates. This is where LLM-HN reports Acc@1 of about 4.5% on Computers. A second real-data baseline adds hard negatives mined from the corpus with BM25 and positive-aware filtering, in the spirit of NV-Embed. It tests whether the corpus already contains enough near-misses.
The strongest existing method is LLM-HN itself: its static single-edit negatives (component-swap, phonetic, abbreviation, semantic distractor) used in triplet training. On Computers at the 10% split it reports Acc@5 reaching 0.5459 (+6.9%) and MRR +4.5%, with a non-significant Acc@1 lift. We re-run it ourselves with the same LLM, the same total generated tokens as our ladder, the same encoder and step count, and an equal hyperparameter search budget (margin, learning rate, negatives per anchor) across five seeds. No per-category numbers are reported for Cameras, Watches or Shoes, so we report our reproduction.
The SyNeg-style baseline mixes these static synthetic negatives with BM25 negatives at SyNeg's reported ratio of about 0.7 synthetic per positive, tuned under the same budget. A latent baseline adds LoOp embeddings to real-data triplet training.
The key ablation separates the ladder from online selection. It uses LLM-HN's single-edit pool with the same online semi-hard filter, at a matched token budget, so the only difference from our method is graded edit depth. A second ablation removes the error-driven top-up.
Beating LLM-HN means a statistically significant Acc@1 gain over it on Computers and Shoes. Significance is assessed with a paired bootstrap over test queries, pooled across five seeds. The ladder must also beat SyNeg-style mixing without using any BM25 negatives.
Why not just…?
Why not just filter LLM-HN's static pool by current distance? This is the most serious threat, and the pool-plus-filter ablation is designed to settle it. Our prediction is that early in training most single-edit negatives fall on the "too hard" side of the margin, so the filter either discards them, leaving no synthetic signal, or admits them and risks collapse. If the filtered single-edit pool matches the ladder at equal tokens, the honest conclusion is that online selection was the missing ingredient, not graded depth. We would report it that way.
Why not just mix in BM25 negatives as SyNeg does? Mixing stabilizes training by diluting hardness, which also dilutes the fine-grained signal that rank-1 accuracy needs. LLM-HN's ceiling suggests that the corpus-mined pool lacks single-attribute contrasts. The SyNeg-style baseline tests whether mixing reaches the ladder's Acc@1, and the diameter and active-triplet logs show whether the two methods stabilize training for the same reason.
Why not just synthesize negatives in embedding space with LoOp or SSCL? These methods are cheaper and give exact geometric control. However, their outputs are not listings. LoOp also assumes a spherical class structure, and SSCL notes that interpolated features may correspond to no real input. If LoOp matches the ladder at Acc@1, it would mean the LLM's semantic edits add nothing, and the comparison is included to detect exactly that.
Why not just train longer, use more data or larger batches? LLM-HN already shows that 10x data and 5x training leave Acc@1 unchanged. The isometric analysis shows that larger batches raise the probability of ultra-hard negatives and therefore of collapse.
Why not just use a collapse-resistant loss such as selectively contrastive triplets? That loss, from Stylianou et al., is described in the file without a Lacuna id. It repels hard negatives only inside a "danger zone," which makes it a complementary fix rather than a competitor. We will cross it with the ladder to check whether the two gains add.
Proposed Experiments and Decisive Checks
The primary benchmark is WDC LSPC across Computers, Cameras, Watches and Shoes, at the 10%, 25%, 50% and 100% splits, measured by Acc@1, Acc@5 and MRR. The secondary benchmark is BEIR (FiQA, SciFact, Quora, HotpotQA) with BGE-large, measured by NDCG@10, to test whether ladders transfer to passage negatives where "attributes" are claims. SyNeg reports +3.0 NDCG@10 for BGE-large and +2.9 for COCO-DR over its baselines. No absolute values were found on Lacuna, so we reproduce the comparison ourselves.
The decisive check is a significant Acc@1 gain over both LLM-HN and the filtered single-edit pool on attribute-dense categories, at matched tokens. Supporting evidence should include more active triplets in early training, a mean rung that rises over training (evidence of a self-paced curriculum), and an embedding diameter that stays bounded away from zero where single-edit-only training shrinks it.
We expect little gain on Cameras and Watches, where model-number strings dominate and attribute edits have little to work with. A gain there would come from the error-driven top-up writing model-number lookalikes. Whatever happens in those categories will be reported per category rather than averaged away.
References
- FaceNet: A unified embedding for face recognition and clustering
- Mathematical Justification of Hard Negative Mining via Isometric Approximation Theorem
- CodeCMR: Cross-Modal Retrieval For Function-Level Binary Source Code Matching
- Deep Triplet Quantization
- LLM-Guided Hard Negative Mining for Structured Product Data Matching
- SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrieval
- CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition
- LoOp: Looking for Optimal Hard Negative Embeddings for Deep Metric Learning
- Synthetic Hard Negative Samples for Contrastive Learning
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models