A robot that learns from a teacher with a different body faces two problems at once. The first is that the teacher's demonstrations look and move differently from anything the robot can do. Adversarial domain adaptation handles this: a network is trained to produce features from which a second network, the "domain discriminator," cannot tell which body generated them. The second problem is that a good learner should eventually go beyond its teacher and discover skills the teacher never showed. Socially guided intrinsic motivation handles this: the learner uses demonstrations to find productive regions and then explores on its own, driven by its measured progress. These two mechanisms work against each other. The discriminator is trained to make the learner's experience indistinguishable from the teacher's. When the learner starts doing new things, the discriminator reads that novelty as a domain mismatch. It then either penalizes it or folds it into the features of demonstrated skills.
We propose to resolve this conflict with information the curious learner already tracks: where each of its goals came from. A socially guided learner knows which goals it adopted by emulating a teacher's outcome and which it invented itself. It can also check whether an achieved outcome, such as an object position, lies inside the set of outcomes the teacher ever produced. We use this goal provenance as the label for a three-way discriminator (teacher, student-on-demonstrated-behavior, student-on-novel-behavior). Only the student's demonstrated-behavior experience is pulled toward the teacher, and novel behavior is explicitly kept apart. The same discriminator also informs the learner's strategy choice. When the student still cannot be made indistinguishable from a teacher in some goal region, that is direct evidence that the teacher's demonstrations are unusable there and that the learner should switch to autonomous exploration.
Core Claim
Open-set domain adaptation already knows that forcing unknown target samples into known source clusters causes negative transfer. Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation (UADAL) addresses this with a three-way discriminator over source, target-known and target-unknown samples. It estimates "unknown" weights from prediction entropy with a Beta mixture model, and the paper itself flags that this estimate fails when the network is overconfident. Our claim is that, in an embodied curious learner, "unknown" should be measured in outcome space rather than inferred from the network's own uncertainty. The outcome space holds object states and effects on the world. It is shared across embodiments and independent of the encoder being trained, so the labels cannot be corrupted by the thing they supervise. We predict three effects. A standard two-way discriminator will measurably erode competence on demonstrated tasks as autonomous exploration expands. Provenance gating will prevent that erosion. It will also beat entropy gating at identifying which student states to align.
Concretely, let $G$ be the behavior encoder and $D$ a three-way classifier with outputs $D_s, D_{tk}, D_{tu}$ for teacher, student-known and student-unknown. Let $w_x \in [0,1]$ be the provenance weight of student transition $x$. It is 0 when the episode pursued an emulated teacher goal whose outcome falls inside the teacher's outcome support, rises toward 1 as the achieved outcome moves outside that support, and is 1 for self-generated goals. The adversarial term is
$$ \mathcal{L}_{adv} = \mathbb{E}_{x \sim \text{teacher}}[\log D_s(G(x))] + \mathbb{E}_{x \sim \text{student}}\big[(1-w_x)\log D_{tk}(G(x)) + w_x \log D_{tu}(G(x))\big]. $$
$D$ maximizes $\mathcal{L}_{adv}$. As in UADAL, $G$ is trained through gradient reversal to confuse teacher with student-known samples while keeping student-unknown samples separable. Only UADAL's weighting source changes, but that change removes its stated failure mode.
Research Program
The learner is built from existing parts. Its outer loop follows Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner (SGIM-ACTS), which tracks competence progress over regions of outcome space and chooses among autonomous exploration, mimicry (copying the teacher's motor commands) and emulation (reproducing the teacher's outcome with one's own body). SGIM-ACTS needed to be told in advance which teachers had correspondence errors. We replace that oracle with a learned signal: the residual separability of teacher and student-known features within each goal region. If $D$ can still tell them apart after a fixed number of updates, the region is marked as one where teacher-derived reward is untrustworthy, and strategy selection is biased toward autonomous exploration there. The inner imitation machinery follows Domain Adaptive Imitation Learning with Visual Observation (D3IL): separate behavior and domain encoders, cycle-consistency, and a reward-generating discriminator. D3IL's learner is passive and has no notion of novel behavior. We keep its representation design and substitute our provenance-gated adversarial term for its two-way one.
The analysis focuses on one mechanism, how alignment pressure interacts with a growing exploration frontier. We will track three things over training: a linear probe's accuracy at decoding demonstrated-task identity from $G$, the discriminator's confusion rate on student-known samples, and the number of distinct outcome cells reached outside teacher support. The central prediction is that under two-way alignment, probe accuracy and demonstrated-task success decline after exploration moves past the teacher's support. Under provenance gating they should not.
Baselines
The naive reference point is the curious learner with no domain alignment at all. It uses an outcome-space emulation reward computed directly from object states plus the same competence-progress goal selection. This is SGIM-ACTS applied as-is to a cross-embodiment teacher, and it isolates what adversarial alignment contributes. It receives the same environment steps, network sizes and tuning budget as every other method.
The strongest existing competitor is D3IL trained on the same teacher demonstrations, with the same intrinsic goal sampler added so it can pursue self-generated goals. D3IL is the right benchmark because in its own experiments it was the only method that got the Ant to the goal in PointUMaze-to-Ant, and it outperformed TPIL, DisentanGAIL and GWIL. The file reports no numeric success rate for D3IL, so no reported result was found on Lacuna for D3IL's PointUMaze-to-Ant or Reacher success values, and we will reproduce it ourselves. The file notes that D3IL is highly sensitive to its adversarial hyperparameters. To keep the comparison fair, D3IL and our method get an identical hyperparameter search budget over adversarial weight and discriminator learning rate, and results are reported over at least ten seeds. Beating it means matching D3IL on demonstrated tasks within confidence intervals while reaching strictly more non-demonstrated goals, or showing D3IL's demonstrated-task success declining over training while ours holds.
The key ablation replaces provenance weights with UADAL's entropy-based Beta-mixture weights, keeping everything else fixed. This isolates the proposal's single new ingredient. A second ablation keeps the provenance-gated discriminator but removes the discriminator-to-strategy feedback. It reverts to SGIM-ACTS-style progress-only selection and tests whether residual separability is a useful teacher-rejection signal. We also run Domain Adversarial Reinforcement Learning for Partial Domain Adaptation-style instance selection, which picks which samples to align using a discriminator-derived reward. It serves as a learned-selection alternative to provenance, though its original setting is static image classification.
Why not just…?
Why not just align on demonstrations first, freeze the encoder, and explore afterward? This is the most serious threat. A frozen encoder cannot be corrupted by later novelty. The cost is that a frozen encoder trained only on demonstrated behavior may represent novel behaviors poorly, so curiosity-driven learning in those regions would rest on features that were never fit to them. The file has no evidence either way, so we include freeze-after-warmup as a baseline. If it matches provenance gating on both demonstrated and novel goals, the online three-way discriminator is unnecessary, and we will report that as a result.
Why not just anneal the adversarial weight to zero as exploration grows? Annealing removes negative transfer by eventually removing alignment. Continued alignment matters because the teacher's support also expands in scaffolded settings. The SocialAI School found that an 8-level scaffolding curriculum beat fixed-difficulty training, which implies demonstrations keep arriving. We will test annealing under a schedule where new demonstrations appear late in training. Our prediction is that annealing cannot absorb them and provenance gating can.
Why not just use metric-based cross-domain matching such as Gromov-Wasserstein and avoid adversarial training entirely? Cross-Domain Imitation Learning via Optimal Transport (GWIL) matches internal distance structure between trajectories without a discriminator. It treats states as unordered point clouds, however, and its Walker sometimes learned to run backward from Cheetah demonstrations. D3IL also outperformed it. It additionally has no mechanism for excluding novel states from the match. We still include GWIL on the locomotion transfer tasks as a check on whether adversarial alignment is needed at all.
Why not just use pure curiosity and skip the teacher? Adversarial Exploration Strategy for Self-Supervised Imitation Learning reached 100% FetchReach success without demonstrations, so on easy tasks this alternative could win. On harder multi-object tasks, Socially guided intrinsic motivation for robot learning of motor skills and SGIM-ACTS show that autonomous exploration stalls, with placing error nearly an order of magnitude above the socially guided learner. We run RND-style and adversarial-exploration learners with no teacher to locate the task difficulty at which demonstrations start to pay off.
Proposed Experiments
The primary testbed is Meta-World with a cross-embodiment and off-dynamics teacher. Teacher demonstrations for five of the MT10 tasks are rendered with altered visual appearance and generated under DARAIL-style dynamics shifts (1.5x gravity or a frozen joint, following Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation). The student acts in the standard environment and must master the five demonstrated tasks while discovering the other five from self-generated outcome goals. The outcome space is the object and goal state shared across domains. For context, Meta-World reports 85% success for multi-headed SAC on MT10 with full rewards, which serves as an upper reference rather than a directly comparable number. The second testbed is the D3IL morphology transfers, 2-link to 3-link Reacher and PointUMaze-to-Ant. We extend them with goals outside the teacher's demonstrated region, such as maze locations the point robot never visited. A Fetch variant (Push and PickAndPlace, with a teacher of different visual appearance) connects to the AIM and adversarial-exploration results.
The metrics are success on demonstrated tasks, success and coverage on non-demonstrated goals, probe accuracy of $G$ over time, and, for the gating ablation, the precision and recall of each weighting scheme against ground-truth "inside teacher support" labels.
The decisive check has two parts. First, the D3IL-with-curiosity baseline must show demonstrated-task success or probe accuracy falling as novel-outcome coverage rises. If it does not, the conflict this proposal targets is not real at this scale, and the project reduces to the freeze-encoder baseline. Second, provenance gating must beat entropy gating on the precision of its unknown labels and on demonstrated-task retention in the overconfident regime, which we induce deliberately by training with label smoothing turned off and larger networks. If provenance and entropy gating perform the same, the contribution is the diagnosis rather than the fix, and we will write it up as such.
References
- Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation
- Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner
- Domain Adaptive Imitation Learning with Visual Observation
- Domain Adversarial Reinforcement Learning for Partial Domain Adaptation
- The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents
- Cross-Domain Imitation Learning via Optimal Transport
- Adversarial Exploration Strategy for Self-Supervised Imitation Learning
- Socially Guided Intrinsic Motivation for Robot Learning of Motor Skills
- Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning