A Clean-Provenance Mask Labeler Trained on Formulas

Dense vision tasks such as segmentation need a label for every pixel. Producing these labels by hand is very slow: a single finely annotated Cityscapes image takes over 1.5 hours (The Cityscapes Dataset for Semantic Urban Scene Understanding). The current shortcut is to let a large pretrained model do the work. Annotators draw cheap boxes, and a foundation segmenter such as the Segment Anything Model (SAM, a network that outputs an object mask when prompted with a box or point) turns them into masks. That shortcut has a legal blind spot. Every mask inherits whatever terms and training-data lineage the labeler carries, and nobody has audited that lineage for vision. In text, the same pattern is already documented: using commercial model outputs as training data drove a 61% jump in non-commercial dataset licenses in 2023 (A large-scale audit of dataset licensing and attribution in AI).

We propose to build a box-to-mask labeler whose entire training history is legally clean by construction, and to measure how much of a foundation labeler's value it recovers. The labeler is pretrained only on images and masks generated from mathematical formulas. It is then adapted to a pool of permissively licensed real images, using nothing but human boxes and its own refined predictions. Throughout adaptation, the exact formula-generated labels stay in the training mix as an anchor. The question is concrete: can a labeler that never saw a scraped image produce training masks good enough that a downstream segmenter matches one trained on SAM-generated masks?

Core Claim

The formula-driven data that SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning used only to pretrain a segmentation backbone is better spent teaching a labeler. Boundary-finding inside a given box is a narrower and more geometric skill than recognizing scene semantics, so it should suffer less from the gap between abstract shapes and real photographs. SegRCDB generates overlapping "radial contour" polygons with pixel-perfect masks. Fine-tuned after this pretraining, it reaches 73.06 mIoU on Cityscapes against 71.00 for pretraining on the GTA5 game engine, but it still trails ImageNet-21k on linear probing. Our bet is that this gap matters much less when the task is "find the object edge inside this box" than when it is "name the scene." The decisive measurement is the fraction of the downstream-accuracy gap between a provenance-free classical method (a conditional random field, CRF, that snaps box edges to color boundaries) and SAM-generated labels that the clean labeler closes.

Research Program

The labeler has two stages.

Stage one: pretraining on formulas. We extend the SegRCDB generator in three ways, so that the labeler learns to separate an object from clutter:

  • crops sampled around a jittered bounding box of one instance;
  • procedural texture and illumination fields painted inside and outside each shape;
  • an explicit generator of thin, high-aspect-ratio and small shapes.

The thin-shape generator targets the known SAM failure modes: long thin scars and defects under 100 pixels (Weakly Supervised Panoptic Segmentation for Defect-Based Grading of Fresh Produce). The labeler takes an image crop plus a box and outputs a mask. Its backbone is also pretrained on formulas, so no step touches scraped imagery.

Stage two: adaptation on permissive real images with human boxes. We initialize targets with a CRF, in the spirit of the box-supervised DeepLab-CRF expectation-maximization of Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation. We then alternate two steps: refine the masks with the labeler, and retrain the labeler on those masks under two hard constraints. A mask may not leave its box, and it must touch all four box sides.

The design choice that separates this from ordinary self-training comes from Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World. That work shows that retraining only on a model's own outputs (its "replace" workflow) collapses, while accumulating ground-truth data keeps training stable. Licensing restrictions often forbid accumulating restricted real labels. The formula data, however, is unlimited, exact, and unencumbered. We therefore keep a fixed share of formula samples with true masks in every round, and treat that share as the clean anchor. We will report how the anchor fraction trades off against label drift.

Every produced mask carries a provenance record listing generator code, formula seed, image license and box annotator. This makes the license audit that A large-scale audit of dataset licensing and attribution in AI performed for text a mechanical property of the vision pipeline.

Baselines

Do nothing: boxes as masks. The naive reference fills each box as its mask and trains the downstream segmenter on those labels, using the same images and schedule. Everything else must beat it by a wide margin, or box supervision alone is the story.

Clean classical labeler: DeepLab-CRF with box supervision. This is the strongest competitor that is itself free of provenance concerns (Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation). It reports 62.2% mIoU on PASCAL VOC 2012 from bounding boxes. We will rerun it with the same formula-pretrained backbone, the same boxes and the same number of refinement rounds, so the only difference from our method is the learned labeler.

Foundation labeler: SAM prompted with the same boxes. This is the strongest existing method and the provenance-uncertain upper reference. On the banana defect set, SAM-generated masks gave 78.2% panoptic quality (PQ, a joint score of mask overlap and instance matching) against 77.9% for hand masks (Weakly Supervised Panoptic Segmentation for Defect-Based Grading of Fresh Produce). For box-supervised Cityscapes semantic segmentation, no reported result was found on Lacuna, so we run SAM ourselves. The downstream segmenter, compute and tuning budget are identical across labelers; only the masks change.

Fully supervised ceiling. We also report full human-mask supervision: Dilated10 at 67.1% mIoU on Cityscapes (The Cityscapes Dataset for Semantic Urban Scene Understanding) and SegRCDB-pretrained fine-tuning at 73.06 mIoU.

Ablations. Three ablations isolate the key ingredients:

  • self-training from CRF targets with a randomly initialized labeler, which removes formula pretraining;
  • adaptation without the formula anchor, which is the replace workflow;
  • the formula-only labeler with no real-image adaptation.

What counts as success. Beating the CRF baseline by a clear margin on all three benchmarks, and closing at least half of the CRF-to-SAM gap, counts as success. Matching SAM on thin and small structures counts as a strong result.

Why not just…?

Why not just use SAM, since it already works? This is the real threat on accuracy: on produce defects, SAM labels match hand labels. The objection is not accuracy but lineage. The labeler's training corpus and terms are inherited by every mask, and the file contains no audit establishing that inheritance is clean for vision. That is precisely the gap A large-scale audit of dataset licensing and attribution in AI leaves open, and BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks flags opaque base-model pretraining as unresolved even after permissive curation. We do not claim SAM's outputs are restricted. We claim a team needing certainty has no measured clean alternative, and our experiment prices one. If the clean labeler trails SAM by a large margin, that is a useful negative result: it quantifies what provenance costs.

Why not just run a CRF or GrabCut-style refinement on the boxes, which needs no learned prior at all? This is the simplest clean option and could be enough on large, high-contrast objects. That is why it is our main baseline rather than a straw man. The evidence that it falls short is indirect. The 62.2% VOC box-supervised number sits well below the 73.9% achieved with more data (Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation), and color-edge snapping should fail on low-contrast thin structures such as poles and scars. If our labeler only matches the CRF, the formula pretraining adds nothing and we will say so. The randomly initialized ablation will show whether gains come from formulas or from self-training alone.

Why not just pretrain the downstream backbone on SegRCDB and fine-tune on a small set of hand-drawn masks? SegRCDB already shows strong transfer this way. In the low-label regime, though, the bottleneck is the number of masks, not backbone quality: fine labels cost 1.5 hours per image against about 7 minutes for coarse ones (The Cityscapes Dataset for Semantic Urban Scene Understanding). We will compare at an equal annotation-time budget, spending it either on a few fine masks or on many boxes passed through our labeler. Whichever wins at each budget is the answer.

Why not just filter for permissively licensed datasets that already have dense masks? Such datasets are scarce. In documents, permissive filtering discarded 80% of 133 candidate datasets (BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks), and permissive collections are measurably less diverse (A large-scale audit of dataset licensing and attribution in AI). Raw permissive images are far easier to find than permissive masks. Our pipeline needs only the former plus boxes.

Proposed Experiments

We evaluate on three benchmarks:

  • PASCAL VOC 2012, using boxes derived from instance annotations;
  • Cityscapes, using boxes from the fine instance annotations and evaluating on the validation set;
  • the banana defect panoptic set, which supplies thin and small structures.

These benchmarks are used as stand-ins for a permissive image pool. Their own licenses are not the object of study, and we state this explicitly.

We report two kinds of numbers. The first is label quality: the IoU of generated masks against held-out human masks on training images. The second is downstream quality: the mIoU or PQ of a fixed segmenter trained on each label source. Results are stratified by object size (under versus over 100 pixels) and by aspect ratio. Robustness to domain shift is checked on ACDC, following the evaluation protocol of Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation, whose diffusion-based generation we exclude because its base generator's lineage is outside our clean constraint.

Three checks are decisive:

A positive outcome on all three would give practitioners the first measured, clean-provenance replacement for foundation-model mask labeling.