Lookalike Curricula for Reinforcement-Learned Personalization

Multimodal language models can tell you that a photo contains a mug or a dog. They cannot tell you whether it is your mug or your dog when a near-identical one sits beside it. A recent fix trains these models with reinforcement learning against automatically checkable rewards, such as "did you name the right person" or "does your bounding box overlap the true object." It works far better than supervised fine-tuning, but it still collapses on crowded scenes with several personal concepts. We propose that much of this collapse is a scheduling problem. When a training scene is far too hard, every sampled answer fails. When it is far too easy, every answer succeeds. In both cases the learning signal is zero. Personalization data has a natural difficulty knob for this: how closely the distractors resemble the personal concept.

The core idea fits in two sentences. Build training scenes whose difficulty is set by two controllable knobs, the number of personal concepts and the visual similarity of same-category lookalike distractors. Then pace reinforcement learning so that the model mostly sees scenes it solves some but not all of the time, moving from dissimilar distractors to near-twins. We test whether this beats a uniform mixture of the same scenes at a fixed rollout budget. We also test whether the gain appears exactly where an existing theory of easy-to-hard curricula says it should.

Core Claim and What Is New

RePIC post-trains a multimodal model with Group Relative Policy Optimization (GRPO), a reinforcement learning method that scores a group of sampled answers to the same input and pushes the model toward the above-average ones. It uses three verifiable rewards: object consistency, localization (box overlap above 0.5), and identity consistency. With 2K samples it beats supervised models trained on 210K. Its data is a static mixture drawn from Subject200K+. On an out-of-domain 4-concept captioning test it reaches 71.0 F1, well below its single-concept performance, while the supervised baseline RAP-Qwen falls to 21.3.

Because GRPO learns only from differences within a group, an input on which all sampled answers receive the same reward contributes no gradient. A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula formalizes the related loop for reward-filtered self-training. It proves that easy-to-hard ordering beats a fixed mixture only inside a "Goldilocks zone." That zone requires three things: the model can already solve the easiest stage (feasible initialization), adjacent stages are distinct but bridgeable (moderate separation), and the sample budget exceeds a threshold (critical budget). Outside this zone curricula do not help. A 1B model even regressed (−0.28% on GSM8K) when the starting problems were out of reach. This paper motivates the proposal, and it also constrains it, because it predicts where our method should fail.

Our claim is that, for instance-level personalization, the difficulty axis that matters most is distractor similarity, not just concept count. A competence-paced curriculum along this axis should improve multi-concept and lookalike performance at equal compute. The gain should vanish when the backbone fails the easiest stage or the budget is too small. No paper in our search orders personalization training by difficulty for modern foundation models. Mass Personalization of Deep Learning studies the closest question: it orders personal versus population data for a VGG classifier on handwriting. It has no notion of lookalike instances, open-vocabulary scenes, or reward-driven training. PolyGen ramps hard counterfactual negatives from 0 to 0.5 during contrastive pre-training. It works at the category and attribute level, not the instance level. We borrow its ramp idea as a fixed-schedule ablation.

Research Program

The first component is a scene generator with controllable difficulty. For each personal concept, we follow Personalized Representation from Personalized Generation. That work fine-tunes DreamBooth generators on about three reference photos to produce diverse views, and we reuse its generation recipe. We add a step it lacks: generating same-category lookalikes by editing one or two visible attributes of the concept, such as a chip, a logo, or a color band. This is the instance-level analogue of PolyGen's single-attribute counterfactuals. Each distractor is placed into a similarity bin by the DINOv2 cosine similarity between its crop and the concept's reference crops. Bins run from generic category members up to near-twins. Scenes are then composed from 1, 2, or 4 personal concepts plus distractors drawn from a chosen bin.

Generated lookalikes may accidentally reproduce the true identity, which is the drift problem the personalization literature flags. We therefore discard any distractor whose similarity exceeds that of held-out real views of the concept itself. We also hand-audit a sample from each bin.

The second component is the pacing rule. During GRPO we track, for each cell of the grid (concept count by similarity bin), the fraction of sampled answers that pass all three RePIC rewards. We draw training scenes preferentially from cells whose pass rate falls in an intermediate band. As the model improves, harder cells enter the band and easier cells leave it. This turns the theory's moderate-separation condition into an online, measurable rule rather than a hand-set schedule. It needs no oracle judge, unlike the GPT-4-based difficulty scores in TAPIR, because the rewards are already verifiable. The training answer format stays direct, without reasoning tags, following RePIC's negative result on chain-of-thought.

The third component is a test of the theory itself. We run the curriculum on a strong backbone and on a deliberately weak one. We first check which backbones solve the easiest cell (feasible initialization). We then sweep the training budget across a range spanning RePIC's 2K-sample setting, to locate the critical budget. Throughout training we log the fraction of zero-variance groups, meaning groups where every answer got the same reward. This is the mechanism we claim the curriculum fixes. If the curriculum wins without lowering that fraction, our explanation is wrong.

Baselines

The naive reference point is the untuned backbone, the same model RePIC starts from, prompted with the reference images in context. It anchors how much any training helps.

The strongest existing method is RePIC itself: GRPO with the same three rewards on a fixed uniform mixture. Its reported result on the out-of-domain 4-concept captioning test is 71.0 F1 (RePIC). For a fair comparison we rerun RePIC on our scene pool rather than only quoting its number, using the same backbone, rollout count per group, total number of policy updates, and learning-rate search budget. The only difference is the order and frequency with which scenes are drawn. We count it as beaten if the paced run exceeds the matched uniform run on the 4-concept test and on our held-out near-twin cells across three seeds, while matching it on single-concept benchmarks. We also report the published 71.0 for context.

The supervised reference is RAP-Qwen-style fine-tuning on retrieval-augmented data, reported at 21.3 F1 on the same 4-concept test (RePIC; see RAP). On recognition and rejection we compare against P-LLaVA, which trains on 3M uniformly mixed personalized instructions and reports 96.69% accuracy on answerable multiple-choice questions and 99.72% on unanswerable ones on P-Bench (Personalized Visual Instruction Tuning).

The ablations isolate the key ingredient. The first is a random-order run over exactly the scene set the curriculum ends up using, which separates ordering from data content. The second is a reverse, hard-first curriculum. The third is a PolyGen-style fixed linear ramp of distractor similarity. The fourth paces concept count only, and the fifth paces similarity only. The sixth is online filtering that drops zero-variance groups without imposing any easy-to-hard order.

Why not just…?

Why not just train longer on the uniform mixture? RePIC already shows that GRPO is sample-efficient, so more steps might close the gap. Our budget sweep tests this directly. If uniform training catches up at higher budgets, the curriculum is a speed-up rather than a capability gain, and we will report it that way. The theory's critical-budget condition leaves both outcomes possible.

Why not just drop groups where every answer gets the same reward, with no ordering? This is the most serious threat, because it attacks the same zero-gradient mechanism with less machinery. The difference is that filtering only discards uninformative scenes after their rollouts are spent. Pacing chooses scenes so those rollouts are rarely wasted, and it also builds discrimination in stages. Ablation six settles the question at equal rollout count. If filtering matches the curriculum on near-twin and 4-concept cells, we will conclude that ordering adds nothing beyond filtering, and the contribution shrinks to the lookalike generator.

Why not just put hard lookalikes into the supervised data, as PVIT does with adversarial negatives? P-LLaVA trains with adversarial unanswerable examples yet still fails on siblings and age gaps. RePIC finds that supervised models copy names from the prompt without checking the image. We include supervised training on our identical scene pool, so that the generator's value is separated from the training method's value.

Why not just let the model reason step by step on hard scenes? RePIC reports that reasoning templates degraded personalized captioning by pulling attention away from visual grounding. We keep direct answers and do not revisit this.

Proposed Experiments

Training scenes come from our generator, built on concepts disjoint from all evaluation sets. Evaluation uses RePIC's single-concept benchmarks (Yo'LLaVA, MyVLM, DreamBooth) and its multi-concept tests (RAP-MLLM and the 4-concept set), scored by F1 for captioning and recognition. We also evaluate on P-Bench for answerable and unanswerable accuracy. To probe real rather than synthetic lookalikes, we convert the distractor split and the pose-plus-distractor split of PODS into multiple-choice identification queries. PODS contains real same-category household objects (Personalized Representation from Personalized Generation). We found no MLLM result on Lacuna for PODS in this format, so all methods are run fresh. To check that general skills survive, we report MME and POPE, the retention suites used by D-MoLE.

Three outcomes would be decisive. First, the paced run beats the matched uniform RePIC run and the filtering-only ablation on 4-concept and real-lookalike PODS queries, with the gain concentrated in high-similarity cells. Second, the zero-variance group fraction is measurably lower under pacing than under the uniform run. Third, the theory's boundaries show up: no gain on the backbone that fails the easiest cell, and no gain below the critical budget. A win on synthetic near-twins without a corresponding PODS gain would mean the model learned generator artifacts rather than instance identity, and we would treat that as a failure of the proposal.