When Sim-to-Real Translation Erases Rare Attributes

Robots that can follow requests like "pick up the clear red cup" or "grab the chrome can" have to see fine-grained properties such as color, material, and surface finish. Simulators can render any combination of these properties, so training in simulation looks like an easy way to cover rare cases. But simulated images do not look like camera images, and the usual fix is unpaired image-to-image translation: a generator that learns to make simulated pictures look real without ever seeing matched sim/real pairs. We argue that this fix fails in a specific, predictable way. The translator is trained against a critic, called a discriminator, that compares its outputs with the overall distribution of real images. In that distribution, rare attributes are rare. The generator is therefore rewarded for turning a rendered chrome can into something that looks like the common painted cans in the real data. The tail of the attribute distribution is quietly pulled toward the head, and it happens in exactly the cases simulation was supposed to supply.

This note proposes to measure that "tail-to-head drift" directly and to remove it with a discriminator that judges realism separately for each attribute factor (color, material, finish), not only for the image as a whole. Because the factors are judged independently and then added up, an attribute combination that never appears in the real data, such as a transparent red object, can still be held to real-world standards. It borrows real evidence from red objects and from transparent objects separately. The downstream test is attribute-conditioned robotic grasping on real novel objects whose attribute combinations were held out of all real data.

Core Claim

Our claim has two parts. First, sim-to-real translators trained adversarially against the real distribution as a whole change attributes at a rate that rises with the attribute's rarity in the real target set. They also change attribute combinations that are absent from that set, even when geometry is preserved. Second, a discriminator conditioned on factorized attributes removes most of this drift. The simulator gives exact per-pixel attribute labels on the synthetic side for free. On the real side it needs only cheap object-level tags, entered once per physical object. The result is attribute-grounded grasping that transfers to held-out combinations better than feature-level adaptation, better than translation with a detector-based consistency check, and better than attribute-preserving domain randomization.

Relation to Prior Work

The closest work is RetinaGAN. It extends CycleGAN with a frozen EfficientDet whose bounding boxes and class scores must agree before and after translation. That stops the gripper warping and object deletion that vanilla CycleGAN causes, and it reaches 65.6% real grasp success with 10,000 real episodes. RetinaGAN supplies our geometric consistency term. However, its critic only knows a fixed category vocabulary and says nothing about color or material. Its discriminator is also unconditional, so the drift we study is exactly the failure it cannot see.

PixelDA plays a similar supplying-and-limiting role. It shows that pixel-level translation with a masked content loss generalizes across object classes unseen during translator training, cutting LineMod pose error to 23.5° against 53.27° for DSN. It is explicitly limited to low-level photometric shifts and was evaluated only on balanced data.

RCAN goes the other way: it translates real images into flat canonical renderings and reaches 70% zero-shot real grasping, against 33–37% for domain randomization. Its canonical domain deliberately discards color and texture. We treat it as evidence that translation beats randomization for geometry, and as a design we cannot adopt when the task depends on attributes.

The downstream task comes from Attribute-Based Robotic Grasping with Data-Efficient Adaptation and its predecessor Attribute-Based Robotic Grasping with One-Grasp Adaptation. These map an RGB-D heightmap and a text query to pixel-wise grasp maps. They pretrain on 36 simulated blocks, bridge the gap with adversarial feature alignment, and reach 81.7% real instance-grasping success on novel objects. Their attribute palette is basic colors and shapes, and their adaptation acts on features rather than on the images themselves. We reuse their architecture and protocol and push the attribute palette into the tail.

The factorization idea comes from Concept Sharing Networks. That work separates where a part is from what pattern it shows, so rare attribute combinations share statistics (bill-grey AP rises from 38.5% to 46.8% on CUB). We move this sharing from the recognizer into the translator's discriminator.

Finally, Open-vocabulary Attribute Detection constrains the design. X-VLM reaches only 28.1 mAP on attributes, against a supervised ceiling of about 48. A CLIP-style critic is therefore a weak referee exactly on the tail, which is why we do not build our constraint on one. Directional semantic grasping motivates including depth: its domain-randomized policy fell to 33.3% success on a specular soup can.

Research Program

We first build a tabletop simulator scene from YCB meshes (YCB) plus procedural objects. Every object carries three discrete attribute factors: color, material class (matte plastic, painted metal, bare or specular metal, transparent, translucent, rubber), and finish. Each material class is rendered from a sampled range of reflectance and refraction parameters rather than a single value, and simulated depth includes material-dependent dropout priors.

On the real side we collect unpaired RGB-D workspace images of about 60 physical objects. We tag each object once with its factor values, and every image inherits those tags through an instance mask obtained by a generic segmenter. The real object set is chosen so that each factor value appears with at least a few objects but a designated set of combinations never appears. Examples are transparent-and-red, specular-and-green, and translucent-and-dark. Objects with those combinations are purchased for testing only.

The translator maps simulated RGB-D to real-looking RGB-D. It keeps RetinaGAN's perception-consistency term for geometry and replaces the unconditional patch discriminator with a factorized projection discriminator. For an image patch $x$ with attribute factors $a = (a_1, \dots, a_K)$, the discriminator computes

$$D(x, a) = \psi(\phi(x)) + \sum_{k=1}^{K} \langle e_k(a_k), \phi(x) \rangle .$$

Here $\phi$ is a shared patch feature extractor, $\psi$ is an unconditional realism head, and $e_k$ is a learned embedding table for factor $k$. On synthetic outputs the factor labels come from the simulator's per-pixel attribute map. On real patches they come from the object tags, and background patches carry a null label. Because the conditional score is a sum over factors, "real-looking transparent" and "real-looking red" are each learned from real data and compose for pairs never seen together. A jointly conditioned discriminator, with one embedding per combination, has no such support. That contrast is the key ablation. We also test a one-sided variant without cycle reconstruction, since round-trip consistency is known to struggle when the real domain carries more visual entropy than the simulated one.

Translated synthetic data then trains the attribute-grasp network of Yang et al. The network is trained only on translated simulation, with no real grasps. We optionally add their one-grasp adaptation as a separate condition.

Baselines

The naive reference point is the attribute-grasp network trained on raw simulated renders, deployed directly. It establishes how large the gap is on tail materials.

The strongest existing system on this exact task is Attribute-Grasp with adversarial feature adaptation (Yang et al. 2025), which reports 81.7% real instance-grasping success on unknown objects. For fairness we retrain it on our simulator's full attribute palette with the same unlabeled real images our translator sees. We give it the same hyperparameter-search budget in GPU hours and evaluate it on the identical real test objects and query set. Beating it means exceeding its success rate on the held-out-combination objects by a margin outside the per-object binomial confidence interval, while not losing on head-combination objects. We do not expect its 81.7% to reproduce on our harder palette; the comparison is within our protocol.

The strongest translation baseline is RetinaGAN, reported at 65.6% grasp success with 10,000 real episodes in its own setting. We run it with its detector critic on our images, with the same generator architecture, real image budget, and training steps as our method. We run a second version where real tail images are oversampled to a balanced factor distribution.

We also include vanilla CycleGAN; RCAN for geometric grasping only (reported 70% zero-shot, but unable to use attribute queries); and domain randomization that preserves attributes, randomizing lighting, camera, background, and in-class reflectance parameters while fixing each object's factor values. The ablations isolating our ingredient are the same translator with an unconditional discriminator, with a jointly conditioned discriminator, and with factor labels shuffled on real images.

Why not just…?

Why not just use domain randomization that keeps attributes fixed? This is a serious competitor, because it gives the policy correct attributes by construction and needs no real data. The evidence against it is geometric and material. RCAN reports that domain randomization alone reaches 33–37% zero-shot versus 70% for translation, and Iqbal et al. saw specular and textureless objects collapse under randomization. Neither tested attribute-conditioned targeting, though, so we run this baseline at full strength. If it matches our method on held-out combinations, the translation component is not earning its cost, and we will report that.

Why not just rebalance the real data and keep RetinaGAN? If drift were caused only by marginal imbalance, oversampling rare real images would cure it. That is why the balanced RetinaGAN run is a baseline. Rebalancing cannot help combinations with zero real examples, and that is where our prediction differs. We expect balanced RetinaGAN to close the gap on rare-but-present factor values and still drift on absent combinations. The factorized discriminator should close both. If balancing closes both, our claim reduces to "rebalance," and the experiment will show it.

Why not just add a CLIP or open-vocabulary attribute consistency loss to the translator? Such a critic can only enforce attributes it recognizes. OVAD shows even X-VLM at 28.1 mAP, with the weakest performance on rare attributes. We include this variant anyway, and we separately report the critic's own accuracy on our tail factors. That lets us attribute any failure to critic blindness rather than to the loss design.

Why not skip translation and pair a vision-language model for target selection with a geometric grasper? A system that picks the target with an open-vocabulary detector and then runs a geometric grasp planner of the GraspNet-1Billion type needs no simulation at all. It inherits the attribute laziness above and the depth corruption on transparent and specular objects. We run it as an extra baseline on the same queries.

Proposed Experiments

The first decisive check needs no robot. We train a held-out attribute classifier on real tagged images, using objects disjoint from the translator's real set, and apply it to translated synthetic images. We report the attribute flip rate as a function of each factor value's real-data frequency, and separately for absent combinations. The prediction is that flip rate rises with rarity and peaks on absent combinations for CycleGAN and RetinaGAN, and stays roughly flat for the factorized discriminator. We found no reported result on Lacuna for attribute drift under sim-to-real translation, so this curve is itself a contribution. For depth, we compare the pattern of missing depth values in translated renders against real captures of held-out transparent and specular objects.

The second check is real attribute-conditioned grasping. It follows Yang et al.'s instance-grasping protocol with cluttered bins of 5–8 objects on the held-out-combination objects plus head-combination controls, with at least 10 trials per object–query pair. The primary metric is instance-grasp success. Secondary metrics are wrong-object grasps, which separate targeting errors from grasp failures, and success on specular and transparent objects specifically.

The third check uses GraspNet-1Billion's real RGB-D scenes as an external test. We ask whether a grasp network trained on our depth-translated synthetic data ranks grasps better on its reflective and adversarial objects. We found no reported result on Lacuna for per-material GraspNet-1Billion performance, so this comparison is internal to our runs.

The proposal fails if attribute-preserving randomization or balanced RetinaGAN matches the factorized translator on absent combinations. It also fails if the drift curve is flat for standard translators. That would mean tail-to-head drift is not the bottleneck, and the tail problem lies in the policy rather than in the pictures.