A camera-carrying agent that follows a moving target (an "active tracker") usually fails in the same way. The target slips behind a wall or pillar. The tracker's short-term memory fades. The tracker then either locks onto a similar-looking bystander or wanders off, and it never re-finds the target. Current training does little to prevent this. The obstacles in simulated training worlds are fixed and hand-built. Random placement of obstacles wastes most episodes on scenes that are either trivially open or hopelessly sealed. Training from raw pixels is also expensive enough that researchers first train a "teacher" policy on exact 3D coordinates and then copy it into a vision-based "student." That copied student is weakest in the one situation that matters here: the target is out of sight and the scene is confusing.
We propose to let a learned environment designer build the occluding geometry itself (walls, pillars, partitions, and the target's route among them). The designer is rewarded by a specific quantity: how much better a perception-privileged reference agent does than the pixel-based student in that layout. The reference agent sees the same camera view and the same map as the student. The one difference is that it is told exactly where the target is in the image whenever the target is visible, and that it is not visible when it is hidden. It does not know where the target is through walls. The score gap between the two agents therefore isolates what the student loses by having to perceive and remember the target. That covers failing to spot reappearance, drifting onto distractors, and forgetting where the target went. It does not reward layouts that are impossible for any searcher. The designer is steered toward exactly the occlusion scenarios where the student's recovery breaks.
Core Claim
Regret-based environment design, applied to static occluder layouts and computed against a perception-privileged rather than fully privileged reference agent, should produce active visual trackers that re-acquire targets after long occlusions more often, and do so with fewer environment steps than domain randomization or distillation from a fully privileged teacher.
The novelty is not "use an adversary for tracking." That exists: the adversary is an evasive target in AD-VAT and moving distractors in Towards Distraction-Robust Active Visual Tracking. The novelty is twofold. First, the designer controls static geometry. Second, the regret signal is defined so that feasibility checking and diagnosis of the student–teacher gap come from the same measurement.
Background and Positioning
Regret-based design comes from PAIRED (Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design). PAIRED trains a designer to maximize the score gap between a helper "antagonist" agent and the learning "protagonist." Impossible and trivial levels both yield zero reward, so the designer settles on levels just beyond the protagonist's ability. PAIRED reports 40% zero-shot success on a Labyrinth transfer task, against 10% for minimax and 0% for domain randomization. We reuse this mechanism. PAIRED was only shown in grid mazes and low-dimensional control, and its antagonist is an equal learner, which triples training cost. We replace the antagonist with a cheap, low-dimensional, perception-privileged agent. That change addresses PAIRED's compute overhead and gives the regret a meaning specific to vision.
Two adversarial active-tracking works motivate the gap. AD-VAT trains an evasive target against the tracker. It needed a "partial zero-sum" reward so that a target escaping early would not kill the tracker's learning signal. This is the same degenerate-adversary problem PAIRED's regret solves more generally. Towards Distraction-Robust Active Visual Tracking adds learned distractors that occlude and confuse, and trains the visual student by cloning a coordinate-level teacher. Both leave the architecture passive. Both rely on a recurrent convolutional memory (a ConvLSTM) that only bridges short occlusions. The 2021 paper also reports a persistent student–teacher gap. Our regret targets that gap directly.
The passive long-term tracking literature constrains the design of the student. GlobalTrack and Siam R-CNN show that full-frame re-detection recovers lost targets, but they run at about 6 and 4.7 FPS, too slow for closed-loop camera control. RTracker keeps separate memories of verified target views and known distractors so that it halts template updates when the target is lost. DTNet learns when to switch between a fast and a robust tracker at 36 FPS. We take these as design options for the student's recovery module. The contribution under test is the training distribution, not a new tracker head.
Research Program
The environment is built on UnrealCV. UnrealCV lets us move and resize objects programmatically and returns exact object masks for every frame.
The designer outputs a layout on a coarse floor grid. The layout includes wall segments, pillars, and partitions with heights, textures from a fixed pool, and a sequence of waypoints for a scripted target, with optional static look-alike distractors. The designer is trained with the regret $$R_{\text{design}} = S_{\text{ref}}(\theta) - S_{\text{student}}(\theta),$$ where $\theta$ is the layout and $S$ is the fraction of frames the target is inside the agent's view and correctly localized. The score is computed from UnrealCV instance masks so that it is identical for both agents.
The reference agent receives the student's camera frame plus two extras. It gets the ground-truth image box of the target whenever the mask is non-empty, and it gets a top-down occupancy map of the layout. It never receives the target's position while the target is hidden. This agent learns quickly because its input is small. In a sealed-off layout it also fails, so the regret there is zero, exactly as in PAIRED.
The student is a pixel-based policy with three parts: a fast local tracker, a positive/negative appearance memory in the style of RTracker, and a learned switch that decides when to invoke a slower global re-detector. The switch is charged the re-detector's latency in simulated time, so it cannot run the re-detector every frame for free.
We train the student with the student's own RL reward plus imitation of the reference agent's actions. Imitating a reference agent that has no through-wall knowledge is feasible for a vision agent. Imitating a fully privileged teacher is not. This difference is where we expect the sample-efficiency gain.
Baselines
The naive reference point is a student with the same architecture, trained on hand-built occluder layouts plus the Unreal maps used by Towards Distraction-Robust Active Visual Tracking. This matches the current practice of fixed scenes. The strongest existing method is that paper's full system: learned distractors, a ConvLSTM tracker, and cross-modal distillation from a coordinate teacher, retrained in our layouts. It is the closest prior system and the only one that already uses an adversarial curriculum for occlusion. Its reported number is a success rate of 0.39 in the Parking Lot scene with identical distractors, versus 0.15 for DiMP. We will reproduce that scene first, to check that our reimplementation is faithful. Beating this baseline means a higher success rate and a shorter post-occlusion recovery time on held-out occlusion layouts, at equal environment steps. For grid-world context, PAIRED reports 40% versus 0% for domain randomization on Labyrinth. No reported result was found on Lacuna for regret-based design in 3D visual tracking.
We add three more comparisons: domain randomization over the same layout parameters, unconstrained minimax (the designer rewarded simply for student failure), and filtered random search (described below). Two ablations isolate the key ingredient. The first swaps the perception-privileged reference for a fully privileged one that knows the target's position through walls. The second removes the recovery module, to test whether the curriculum only helps a student that is able to use it.
All methods share the student architecture, the pool of texture and object assets, and the random seeds (five per method). They also share one total budget of simulator frames. That budget counts the reference agent's and designer's rollouts against the method using them, and it counts DR's extra student rollouts against DR. Each method gets the same number of hyperparameter trials.
Why not just…?
Why not just randomize occluder layouts and distill as before? This is the most serious threat. PAIRED's evidence that domain randomization fails comes from grid mazes, and occlusion-rich 3D scenes might be dense enough that random layouts already contain many useful cases. The DR baseline, run at matched frames, settles this. If DR matches our method on held-out layouts, the claim is wrong. We will also report the fraction of DR layouts in which the reference agent succeeds while the student fails. That fraction measures how much of the DR budget is informative.
Why not just search random layouts and keep the ones with large reference–student gaps, without training a designer? This is filtering by intermediate success, as in IFDecorator. It could plausibly capture most of the benefit. We therefore treat it as a first-class variant, not a strawman. If it matches the learned designer, we will report that the contribution is the perception-gap regret signal rather than the generator. We still expect a learned designer to help on compositional layouts, such as a long corridor leading to a turn beside a distractor, that random sampling rarely hits. We will measure this with an unfilterable held-out set of multi-part occlusion layouts.
Why not just bolt a global re-detector onto the tracker and skip the curriculum? Frame-by-frame re-detection runs at 6 FPS (GlobalTrack) and 4.7 FPS (Siam R-CNN). In a latency-charged loop the camera falls behind the target. A learned switch, as in DTNet, still has to learn when the target is lost and where to look, and fixed layouts rarely exercise this. The "fixed scenes plus recovery module" baseline tests exactly this alternative.
Why not an adversary that is simply rewarded when the tracker loses the target? Both PAIRED (0% on Maze, zero reward on Hopper) and AD-VAT's need for a partial zero-sum reward show that such an adversary drifts toward impossible scenes. Here that means sealing the target in a room. The minimax baseline reproduces this, and we will log the fraction of its layouts that are unsolvable even for the reference agent.
Proposed Experiments
Evaluation uses three sets of layouts, all unseen during training: held-out hand-designed occlusion layouts, the Parking Lot and other Unreal scenes from the 2021 paper, and a compositional set of layouts that chain several occlusions together. We report success rate, episode length, and accumulated reward, following the prior active-tracking papers. We add two occlusion-specific measures, both computed from instance masks. The first is recovery time, the frames from the target's reappearance to its re-localization, a metric RTracker reports for passive video. The second is drift time, the frames spent locked on a distractor.
Sample efficiency is shown as learning curves against total simulator frames. The key quantity is the number of frames each method needs to reach the DR baseline's final held-out success rate.
The decisive checks are the following. The method must beat both DR and filtered random search on the compositional set at matched frames. The perception-privileged reference must outperform the fully privileged one; otherwise our feasibility argument is unnecessary. Over training, the regret should concentrate on layouts with occlusions longer than the ConvLSTM baseline's typical bridging horizon, which we can verify by logging occlusion durations of the generated layouts.
As a secondary check on transfer, we will freeze the student's perception and recovery modules and run them passively on VideoCube (Global Instance Tracking) and LaSOT, comparing against the same modules trained without the curriculum. We will not claim competitiveness with RTracker's 63.2% VideoCube AUC unless the numbers show it.
References
- AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking
- Towards Distraction-Robust Active Visual Tracking
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
- GlobalTrack: A Simple and Strong Baseline for Long-Term Tracking
- Siam R-CNN: Visual Tracking by Re-Detection
- RTracker: Recoverable Tracking via PN Tree Structured Memory
- Online Decision Based Visual Tracking via Reinforcement Learning
- UnrealCV: Connecting Computer Vision to Unreal Engine
- IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards
- Global Instance Tracking: Locating Target More Like Humans