Denoising Distant Relation Labels with a Coarse-to-Fine Image Generator

Programs that label data automatically are cheap but wrong often enough to matter. Take the task of learning which relations hold between objects in an image, such as "person riding bicycle." One cheap recipe detects objects in unlabeled photos, looks the object pair up in a knowledge base mined from web captions, and assigns every relation the knowledge base lists for that pair. This is distant supervision. It gives broad coverage, but many of its labels are false: the person is standing next to the bicycle, not riding it. The best existing cleanup, Visual Distant Supervision for Scene Graph Generation, re-scores the candidate labels with CLIP, a model trained to match whole images to whole captions, together with a relation classifier. It then retrains and repeats in an expectation-maximization loop. Relations, however, are mostly about where things sit relative to each other, and a single image-to-caption similarity score is poorly matched to that kind of evidence.

We propose to let a coarse-to-fine image generator act as the judge. A next-scale autoregressive generator of the kind introduced in Visual Autoregressive Modeling produces an image as a stack of token maps of increasing resolution, from 1×1 up to 16×16. It commits to global layout in the first few tiny maps and to texture in the later ones. If such a generator is conditioned on a relation triple, the probability it assigns to an image's coarse token maps under "person riding bicycle," compared with "person next to bicycle," measures whether the image's layout fits the relation. Pixel detail does not dilute that measurement. The core claim is that this coarse-scale likelihood ratio is a better noise filter for distant spatial-relation labels than a global cross-modal similarity score. As a side effect, the same loop yields a relation-conditioned text-to-image generator trained without a single human-drawn box.

Core Claim and Mechanism

Let $x$ be an unlabeled image whose tokenized scales are $t_1,\dots,t_K$ (coarsest to finest), and let $r=(s,p,o)$ be a candidate subject–predicate–object triple proposed for a detected object pair by knowledge-base matching. A generator $G$ conditioned on the text of the triple, encoded by a frozen language encoder, defines the coarse-scale evidence

$$ S_k(r;x)=\sum_{j\le k}\Big[\log G(t_j\mid t_{<j},r)-\log \tfrac{1}{|P_{s,o}|}\textstyle\sum_{p'\in P_{s,o}} G(t_j\mid t_{<j},(s,p',o))\Big], $$

where $k$ is a small cutoff scale (we expect 3–5 of the roughly ten scales) and $P_{s,o}$ is the set of predicates the knowledge base lists for that subject–object pair. The E-step posterior over candidate predicates is

$$ q(p\mid x,s,o)\propto \pi(p\mid s,o)\,\exp\big(\lambda\, S_k((s,p,o);x)\big), $$

where $\pi$ is the knowledge-base frequency prior and $\lambda$ is a temperature tuned on a small validation set. The M-step retrains $G$ on images weighted by $q$. Only the coarse scales are scored, because in next-scale generation composition is fixed early and later scales refine texture. That same property makes errors at coarse scales impossible to undo downstream, the error-propagation problem noted for hierarchical pipelines such as Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. Our method turns this weakness into a measuring instrument: the scales where noise does the most damage are the ones used to detect it.

The idea draws on each research direction in a different role. Distant supervision supplies the label source. The coarse-to-fine generator is the denoising mechanism. Cross-modal transfer enters through the language encoder, which lets predicate semantics generalize across object pairs, in the spirit of the word-vector transfer shown in Towards Context-Aware Interaction Recognition. The cross-modal gap sets the design constraint. Learning Robust Visual-Semantic Embeddings argues that instance-level contrastive matching leaves the image and text distributions misaligned. A Visual Attention Grounding Neural Model shows that a single global image vector underuses visual structure. Both motivate scoring relations against layout-level tokens rather than against one global CLIP vector.

Relation to Prior Work

The two closest methods split the problem between them. Visual Distant Supervision for Scene Graph Generation supplies our label-generation recipe and our EM structure: a knowledge base parsed from Conceptual Captions, detector-proposed pairs, and CLIP-assisted denoising. It is purely discriminative, and its verifier sees the image globally. ControlVAR supplies our generator backbone, its public checkpoints and pseudo-labeled ImageNet data (lxa9867/ControlVAR), and its demonstration that off-the-shelf labelers can supervise next-scale generation. It has no language conditioning and no relational knowledge. We borrow the loop from the first and the backbone from the second, and we replace the first's verifier with the second's likelihood.

Relational Distant Supervision for Image Captioning shows that relation-level distant labels can replace paired data for image-to-text generation. We work in the opposite direction and use generation as the verification step, not as the end product.

Layout-driven generators set the bar for the downstream generator and show what annotation normally costs. Image Generation From Scene Graphs, Hong et al., and HCMA all depend on human boxes or masks. Cross-Modal Contrastive Learning for Text-to-Image Generation (XMC-GAN) argues that multi-stage pipelines accumulate errors and that single-stage contrastive training suffices. We take that claim seriously as a competing hypothesis.

Research Program

We first build the candidate label pool by reproducing the VDS knowledge-base construction over Conceptual Captions and running an off-the-shelf detector on ImageNet images (reusing ControlVAR's released pseudo-label set) and on Visual Genome images with their relation labels hidden. Each detected pair receives the predicates the knowledge base licenses. We then add a triple-conditioning pathway to a ControlVAR checkpoint of depth 16 or 20: the frozen text embeddings of $s$, $p$, $o$, together with coarse box tokens for the subject and object, prepended to the scale sequence. We fine-tune it on the uniformly weighted candidate labels to obtain the initial generator.

We then alternate E-steps, using the scoring rule above, with M-steps that retrain on the reweighted labels, for three to five rounds. The central analysis sweeps the cutoff $k$ from 1 to $K$. We predict a peak at intermediate $k$. Very coarse scales (1×1, 2×2) cannot resolve relative position, and fine scales add texture likelihood that does not depend on the predicate and so swamps the ratio. If the curve is flat, the hierarchy is doing no work and the claim fails.

We also track a failure mode specific to this method: the generator trained on noisy labels may learn the noise and then confirm it. We measure that by computing, after each round, the agreement between $q$ and held-out Visual Genome human labels.

Baselines

The naive reference point uses raw distant labels with no denoising. The candidate predicates weighted only by the knowledge-base prior $\pi$ train both a scene-graph model and the generator. Every method must improve on this.

The strongest existing denoiser is VDS itself, with its CLIP-plus-relation-model E-step. It is the published method for exactly this label source, and it reports a mean of 29.76% on the 20-predicate Visual Genome split for Motif trained on its denoised labels. Under the same protocol, relation models trained on image captions reach 20.76% (Visual Distant Supervision). We will rerun VDS on our exact candidate pool, with the same detector and the same number of EM rounds. We will also give its CLIP scorer the same validation-set tuning of the temperature. Beating it means a statistically reliable improvement in denoised-label precision against held-out human labels, and a higher mean than our reproduced VDS number when Motif is trained on our labels. We do not claim to beat 29.76 itself unless our reproduction matches it.

A hybrid baseline sums the CLIP score and $S_k$. This separates whether the generator adds information or merely substitutes for CLIP.

The key-ingredient ablations are full-image likelihood ($k=K$) versus coarse-only scoring, which isolates the role of the hierarchy, and a raster-order generator of matched size scored on its first tokens, which isolates next-scale structure from autoregressive likelihood in general.

On the generation side, the upper reference is the same architecture trained on human Visual Genome relations and COCO boxes. The published comparators are HCMA at FID 8.74 and CLIP score 0.3231 on COCO 2014 validation (HCMA) and XMC-GAN at FID 9.33 on COCO (XMC-GAN). Both use human-paired or human-boxed data, so they are reference points rather than same-setting competitors. No reported result was found on Lacuna for relation-conditioned next-scale autoregressive generation trained from distant labels.

Why not just…?

Why not just keep VDS's CLIP-based E-step and train the generator on its output? This is the real threat. The file shows that CLIP-EM already yields a nine-point gain over caption supervision. The claim that a global similarity score is weak for spatial predicates is our hypothesis, backed only indirectly by the granularity arguments in Learning Robust Visual-Semantic Embeddings and MAF. The per-predicate comparison settles it. If $S_k$ wins only on geometric predicates (on, under, riding, behind) and loses on semantic ones (eating, wearing), the right result is the hybrid scorer, and we will report that instead of the pure method.

Why not just caption the images and train an ordinary text-to-image model? Captions under-report mundane relations. The file quantifies the cost for relation learning (20.76% versus 29.76% on Visual Genome), and a generator trained on captions inherits the same blind spots. We still include a captioner-supervised generator. We will evaluate it with the relation-accuracy test below, so any advantage of explicit triples must show up in measured relation fidelity, not in FID.

Why not just condition on detector boxes or segmentation masks, as ControlVAR does, and skip relations? Boxes fix positions but not interactions. A person box overlapping a horse box is consistent with riding, feeding, or standing behind. This alternative is cheap, so we run it as a baseline generator. If box conditioning alone matches our model on relation accuracy, then predicate denoising adds no value for generation, and the contribution reduces to the scene-graph side.

Why not just a single-stage generator with multi-level contrastive losses? XMC-GAN reports strong COCO FID without hierarchy. But its losses require paired captions, and it offers no coarse-scale likelihood that could score a triple, so it cannot serve as a denoiser at all. Its argument against hierarchy concerns error accumulation during generation. We test that argument directly through the round-by-round agreement tracking.

Decisive Experiments

Three checks decide the project. First, measure denoised-label precision and recall on hidden Visual Genome relations, split by predicate type, for raw labels, VDS, $S_k$, and the hybrid scorer. The method succeeds only if $S_k$ or the hybrid beats reproduced VDS, and only if the gain peaks at intermediate $k$.

Second, train Motif on each label set and report mean recall on the 20-predicate split and micro- and macro-recall@50 for predicate classification. VDS reports +8.3 and +7.8 point gains over fully supervised models in its semi-supervised setting, and we repeat that setting.

Third, generate images from held-out COCO and Visual Genome triples. We report FID and CLIP score, plus relation accuracy: the rate at which a scene-graph model trained only on human labels, and never exposed to our pseudo-labels, recovers the conditioned predicate from the generated image. We compare against the raw-label, VDS-label, box-only, captioner-supervised, and human-label generators.

The reasons to abandon the approach are clear in advance: a flat curve over $k$, a pure $S_k$ scorer that never beats CLIP on geometric predicates, or agreement with human labels that falls across EM rounds.