Logic-Pinned Self-Training for Sim-to-Real Concept Transfer

A neuro-symbolic model has two parts: a neural network that reads raw inputs and outputs human-readable concepts ("red light", "pedestrian", "the digit 7"), and a fixed set of logical rules that turns those concepts into a decision ("stop"). When such a model is trained only on final decisions, it often learns reasoning shortcuts. These are wrong concept meanings that still produce the right answers, such as treating every pedestrian as a red light, since both imply "stop." Simulators seem to offer an easy fix, because they label every concept for free. A model trained in simulation, however, makes systematic errors on real images. The natural next step is to self-train on real data, predicting concept labels, filtering them with the rules, and retraining. This proposal argues that the step fails in a predictable way. The rules can catch a real-image error only if it breaks consistency with the known decision. The most damaging sim-to-real errors are exactly those that don't, because they coincide with a reasoning shortcut.

We propose to split sim-to-real concept errors into two kinds. Some errors are visible to the logic. Others are invisible because they fall inside the set of concept assignments that the rules and the decision label cannot tell apart. We then build a self-training loop that assigns each part of the problem to the component that can handle it. The logic certifies a concept only on the real examples where the decision label fully determines that concept's value. Only those certified examples are allowed to move the concept anchors learned in simulation. For every concept the logic leaves ambiguous, the simulation anchor is the sole source of meaning and is protected from drift. The claim to test is simple. Self-training that ignores this split slides toward reasoning shortcuts on real data. Self-training that respects it keeps concept quality close to what a handful of real concept labels would give.

Core Claim

Write $R$ for the rules, $z=(z_1,\dots,z_k)$ for a concept vector, and $y$ for a downstream label. A concept $z_j$ is pinned on a real example $(x,y)$ if every concept vector consistent with the label agrees on it:

$$ z_j \text{ pinned on } (x,y) \iff |\{ z_j : z \in R^{-1}(y) \}| = 1, $$

where $R^{-1}(y)$ is the set of concept vectors the rules map to $y$. In MNIST-Addition, a sum of 0 pins both digits to 0. In driving, a "move forward" label pins every stop-cause to absent. Pinned values are certified ground truth on real images at no annotation cost. Unpinned values are exactly the degrees of freedom a reasoning shortcut exploits. The core claim has three parts. First, a sim-pretrained model's errors on real data that fall in unpinned coordinates cannot be removed by logic filtering, however many iterations are run. Second, standard self-training moves the model's remaining errors into those coordinates over time; this is the direction the file's "Hypothesis B" predicts, and it has never been tested. Third, the fix is structural rather than statistical. Concept prototypes (per-concept embedding centroids, as in PNet) are initialized in simulation, and each prototype is updated on real data only from examples where that concept is pinned.

This sits against three groups of prior work. The first motivates the problem. Not All Neuro-Symbolic Concepts Are Created Equal frames shortcuts as unintended optima and shows that on real BDD-OIA driving data a standard neuro-symbolic model reaches about 71% action F1 but only about 39% concept F1. rsbench builds a synthetic twin, SDD-OIA, and shows a 46% accuracy drop for standard networks under an out-of-distribution "ambulance may cross a red light" rule. Both papers treat the synthetic and real driving datasets as separate benchmarks and never transfer between them. Generative Pseudo-label Refinement names "shift noise," meaning structured pseudo-label errors caused by domain shift, and removes it with a conditional GAN. Its classes are unrelated atoms, with no rules that could separate certifiable errors from uncertifiable ones.

The second group supplies our method. Right for the Right Reasons (PNet) anchors concepts to prototypes built from one or two labeled examples. Its prototypes are static and come from a single domain, and we make them adapt across domains. CurABL uses low-ambiguity "simple samples" to veto candidate labels, linking samples through distances in a single feature space. Under a sim-to-real shift those distances are exactly what breaks. We keep CurABL's intuition but define certainty per concept from the rules and the label alone, with no reliance on feature similarity. ASPER filters neural predictions with answer-set constraints in text. It is the prototype of the logic-filtered self-training that we argue is insufficient on its own.

The third group constrains the design. The learnability analysis in A learnability analysis on neuro-symbolic learning shows that when the derived constraint problem admits several solutions, the true concepts cannot be recovered from labels. This means the unpinned coordinates must get their meaning from outside the real labels, and here that source is the simulator. Neurosymbolic Reasoning Shortcuts under the Independence Assumption warns that independent per-concept prediction interacts with shortcut formation. We therefore report results with and without it rather than assuming it helps.

Research Program

The first step is diagnostic and requires no new method. For each benchmark we enumerate $R^{-1}(y)$ per label and compute pin coverage, the fraction of real examples on which each concept is pinned, split by positive and negative value. This works for MNIST-style arithmetic and for BDD-OIA's small rule set. Where direct enumeration is too slow, a model-counting tool of the kind rsbench provides can serve instead. On test data that has concept labels, we then measure how each model's concept errors divide between pinned coordinates (logic-visible) and unpinned coordinates (logic-invisible). Tracking this invisible-error share across self-training rounds directly tests the Gap-4 tension.

The second step is the method. A concept encoder with PNet-style distance-based concept probabilities is trained with full concept supervision on synthetic data, which gives one prototype per concept value. On unlabeled real data with downstream labels, each round proceeds as follows. The model scores real examples. Pseudo-labels are restricted to $R^{-1}(y)$, as in ordinary logic-filtered training. Each prototype is then re-estimated as a running average of real embeddings, using only examples where that concept value is pinned. The encoder is trained with the usual label-likelihood loss plus a pull toward the current prototypes. Where a concept is unpinned on an example, the prototype pull is the only concept-level signal, and the prototype for that value moves only through pinned evidence gathered elsewhere.

Some concept values may never be positively pinned. Pedestrians, for example, co-occur with other stop-causes. In that case the prototype stays at its simulation value, and we report this as a measured coverage limitation rather than hiding it. Keeping synthetic data in the loop with a small replay weight is an optional variant, and its effect is tested separately.

The third step is analysis. We need to know when pinning is enough, so we relate final concept F1 on each concept to its pin coverage. We expect concepts with near-zero positive coverage to track the source-only model and concepts with high coverage to approach the few-shot real-label reference.

Baselines

The naive reference is source-only transfer: the concept encoder trained on synthetic data and applied unchanged to real images through the rules. It sets the size of the sim-to-real gap. For digits, the domain-adaptation paper reports source-only MNIST→SVHN classification at 31.4% (Generative Pseudo-label Refinement). No reported result was found on Lacuna for SDD-OIA→BDD-OIA transfer.

The strongest competitor on our specific question is sim-pretraining followed by end-to-end neuro-symbolic fine-tuning on real downstream labels. This uses DeepProbLog-style label likelihood, following the two-stage recipe of Simple and Effective Transfer Learning for Neuro-Symbolic Integration, which raised CIFARSum accuracy from 8.4% to 81.6%. It is the most likely way our result could turn out unnecessary. The initialization may already sit in the right basin, and the label loss alone may keep it there.

The second strong competitor is logic-filtered self-training, meaning ASPER- or abductive-learning-style pseudo-labels restricted to $R^{-1}(y)$ with confidence thresholding. We also run CurABL as its strongest published variant. On real BDD-OIA with no sim anchor, the reference point is standard neuro-symbolic training at about 39% concept F1, rising to about 61% only with real concept supervision and entropy regularization (Not All Neuro-Symbolic Concepts Are Created Equal).

For the domain-adaptation angle, we run PLR on the digit benchmarks. It reports 97.3% on SVHN→MNIST and 63.4% on MNIST→SVHN digit classification (Generative Pseudo-label Refinement). We run it as a concept-level adapter before the rules.

Two PNet references complete the set. PNet with static synthetic prototypes shows what anchoring without adaptation does. PNet with one or two real labeled examples per concept serves as a label-cost reference. For the latter, PNet reports concept F1 rising from 25% to 94% on Kand-Logic with one example per concept (Right for the Right Reasons).

Two ablations isolate the key ingredient. The first updates prototypes from all logic-consistent confident pseudo-labels rather than pinned ones only. The second removes the prototype pull on unpinned coordinates.

Fairness rules apply throughout. All methods share the same encoder architecture, the same synthetic pretraining checkpoint, the same real training images and downstream labels, and the same number of self-training rounds and wall-clock budget. Each method gets the same hyperparameter search size, selected on a real validation split that uses downstream labels only, since concept labels would leak supervision. "Beating" the strongest baseline means two things. Real concept F1 must be higher by more than the seed-to-seed spread over five seeds. The invisible-error share must be lower at matched or better action F1.

Why not just…?

Why not just fine-tune the sim-pretrained model end-to-end on real labels? This is the real threat. Pretraining is known to prevent collapse in-distribution (Simple and Effective Transfer Learning), but the label loss gives no gradient that separates a correct concept from a shortcut-swapped one. Nothing therefore holds unpinned coordinates at their simulation meaning once real-image features move. The invisible-error curve across epochs settles this question. If fine-tuning's invisible share stays flat and its concept F1 matches ours, the proposal's premise is wrong, and we will report it as such.

Why not just use standard domain adaptation, adversarial or GAN-based, before the rules? Feature alignment matches overall image statistics. It carries no knowledge of which concept confusions leave decisions unchanged. The CyCADA-versus-PLR gap on SVHN→MNIST (90.4% vs 97.3%) shows that structured shift noise survives alignment (Generative Pseudo-label Refinement). We run PLR as a baseline. We also run it combined with our pinning, because the two are compatible, so a win for the combination would still support the claim.

Why not just label a few real concepts? PNet's one-shot results show this is powerful (Right for the Right Reasons), and in many settings it will be the pragmatic choice. We do not argue around this. We state an exchange rate: the number of real labels per concept at which the few-shot PNet reference matches our label-free method. If that number is one, the contribution is narrow. It then matters only where no real concept labels can be obtained, as in the spacecraft and retail settings the sim-to-real literature targets.

Why not just threshold pseudo-labels by confidence? Confidence does not detect structured errors. A shortcut-swapped concept can be predicted with high confidence and is by construction logic-consistent. This is why the first ablation above, which updates prototypes from confident logic-consistent labels, is the most important comparison.

Proposed Experiments

The controlled testbed builds arithmetic tasks across a known visual shift. The source domain is SVHN digit pairs with digit labels. The target is MNIST pairs carrying only sum labels, for MNIST-Addition, or parity labels, for the deliberately ambiguous MNIST-EvenOdd, with both directions of shift. Addition has rich pinning at extreme sums. EvenOdd has almost none. The pair tests the coverage prediction directly.

The main testbed is SDD-OIA→BDD-OIA. We first check which of the 21 concepts and 4 actions align between the two datasets and restrict evaluation to those. BDD-OIA's held-out concept labels are used only at test time.

Metrics are concept F1, action F1, and the invisible-error share per round. We add a real-data counterfactual rule test. The ambulance-style rule is applied to predicted concepts, and its outputs are scored against the same rule applied to ground-truth BDD-OIA concepts. This carries rsbench's out-of-distribution test onto real footage.

Three results would be decisive. The first is whether standard fine-tuning and logic-filtered self-training show a rising invisible-error share over rounds, which would confirm the Gap-4 prediction. The second is whether our method holds that share flat while improving concept F1 above both. The third is whether concept-level gains track pin coverage, with near-zero gains on unpinned concepts. That last pattern would show the mechanism works as claimed and mark exactly where simulation must carry the meaning alone.