Multi-object trackers lose people when they walk behind each other. When a target disappears, the tracker has to guess where it went and then recognize it when it comes back. That guess comes from a motion model, a small piece of logic that predicts the next position. In practice this is usually a hand-written constant-velocity Kalman filter, and it breaks on dancers and athletes who turn and accelerate. The alternative is a learned sequence network, which needs large amounts of annotated video and gives no account of why it fails. We propose a third option. A language model writes the motion-and-occlusion logic as a short, readable program, and each occlusion where the tracker lost a target becomes a failing unit test. The language model then debugs the program against those tests, the way an automated repair agent fixes a bug against a test suite. The language model is used only offline, so the deployed tracker runs a plain program at classical-filter speed.
The bet is that the most informative training signal for occlusion handling is the specific occlusion where the tracker lost someone: the trajectory before the gap, the predicted path, and the actual point of return. Execution-feedback code repair already knows how to use that kind of signal. If the bet holds, we get motion models that match learned predictors on non-linear-motion benchmarks, generalize better when annotated data is scarce, and can be inspected line by line.
Core Claim
Our claim is about the refinement signal. Suppose a language model refines a structured motion program (prediction, update, re-acquisition gating, and appearance-freeze rules) against episode-level counter-examples of occlusion failure, with automatic rollback when changes cause regressions. We claim this produces trackers that beat both hand-crafted filters and tuned parametric filters on DanceTrack and SportsMOT. We further claim that, under limited training data, these trackers match or beat a learned state-space motion model trained on the same data. The ingredient we expect to matter is the episode-level test, not the language model. Refining against a single aggregate score should plateau early, as unstructured compiler feedback does in CodeTransOcean, where gains flatten after two to three rounds.
Where This Sits
The tracking side gives us both the problem and the ingredients the program can draw on. Joint Detection and Online Multi-Object Tracking shows that trackers without motion prediction fragment trajectories and switch identities when targets re-emerge, which motivates the problem. Several papers contribute the building blocks our program space must be able to express:
- Kinematic extrapolation through data loss. Tracking Known Three-Dimensional Objects extrapolates through sensor dropouts using models with higher-order time derivatives.
- Freezing appearance during occlusion. CFME stops appearance updates when the correlation peak falls below a fixed 0.3 threshold, so the model does not learn the occluder. It then relies on Kalman motion alone.
- Reachability-based search. Occlusion Geodesics re-associates targets along minimal-cost paths over an occlusion map. It assumes static cameras, flat ground, and known occluders.
Each of these is a hand-written rule tuned by a human for one domain. Our system is meant to rediscover and combine such rules per domain from failure evidence. For example, it could replace CFME's fixed threshold with a gate that widens as the occlusion lengthens.
The learned alternative is MambaTrack. It replaces the Kalman filter with a bi-directional state-space network and patches trajectories through missed detections autoregressively. It documents the failure of linear filters on erratic motion, and it is our strongest competitor. InstMove predicts motion from instance masks with a recurrent memory network. It warns that learned predictors inherit errors from their inputs. Neither can change its structure after training. Neither tells a practitioner which occlusions it handles badly.
The code-refinement side supplies the method:
- Counter-examples. From TELPA we take the idea of feeding concrete counter-examples, there the uncovered branches and failing executions, back into the prompt.
- Rollback. From MEIC we take automatic rollback to the last stable version whenever an edit degrades the score. This is our main defense against the regressions MEIC and TELPA both report.
- Trace-style feedback. From AutoSD we take the presentation of feedback as intermediate state traces rather than bare pass/fail.
The closest prior work in spirit is Jin et al., which refines robot control code against physics-simulator feedback. Its programs act on a known world; ours must estimate a world it cannot see during the gap. We deliberately avoid multi-agent debate. Adversarial Review shows that unstructured debate drifts toward false consensus. Our oracle is execution against ground truth, which cannot be argued with.
Research Program
The program interface. The tracking pipeline is held fixed: the same detector, the same appearance embedding, and the same matching solver. Only a motion module is editable. Its interface has four functions:
predict(state, dt), which advances a target's state;update(state, detection), which incorporates a new observation;gate(state, gap_length), which returns a re-acquisition region and a cost adjustment;freeze(state, appearance_score), which decides whether appearance memory may update.
The initial program is a constant-velocity Kalman filter. Edits may add state variables, introduce switching between motion regimes, or change gating and freeze rules. Edits may not import learned weights, since a learned predictor is a separate baseline. A length budget keeps programs readable.
The test suite. On the training split, we run the current tracker and extract every occlusion episode. An episode is a ground-truth track with a detection gap or a lost track, followed by re-emergence. Each episode becomes a unit test. The program receives the pre-gap history and runs forward through the gap. The test passes if, at re-emergence, the association cost built from gate ranks the correct detection first among all candidates. A secondary check asks that predicted positions stay within a tolerance of the ground truth wherever ground-truth boxes exist during the gap. Failing episodes are grouped by gap length and by the motion pattern before the gap, such as turning, accelerating, or crossing another target. They are shown to the language model as compact trajectory traces of predicted versus actual positions. The feedback says, in effect, "this dancer reversed direction in frame 12 of the gap and your gate pointed the wrong way."
The loop. Each round has five steps:
- Sample failing episodes, weighted toward the largest failure groups.
- Ask the language model to propose an edit.
- Re-run the full episode suite.
- Re-run full-pipeline tracking on a held-out portion of the training sequences.
- Accept the edit only if the episode pass rate rises and the held-out association score does not fall. Otherwise roll back.
Because rollback is checked against held-out training sequences, the loop is pushed toward rules that generalize rather than rules that memorize individual episodes. The final program is frozen and evaluated once on the validation and test splits.
Analysis. We will read the final programs directly. Did the loop rediscover gap-dependent gating, regime switching, or appearance freezing? Are these rules specific to each domain? We will check whether a program refined on DanceTrack transfers to SportsMOT, and the reverse.
Baselines
The naive reference point is the initial program alone: the constant-velocity Kalman filter in the same fixed pipeline, with no refinement. Every gain is measured against it, and it fixes what "no motion intelligence" costs on each benchmark.
The strongest classical competitor is OC_SORT, an observation-centric Kalman tracker. We run it with our detector so detection quality is identical. MambaTrack reports beating OC_SORT on DanceTrack by 2.2 HOTA points (MambaTrack). HOTA, Higher Order Tracking Accuracy, is a combined detection-and-association score.
The strongest existing method is MambaTrack's learned motion predictor with its trajectory-patching module. The reported numbers are:
- DanceTrack: HOTA 56.8 for its motion predictor, and a peak IDF1 (identity F1) of 57.8 (MambaTrack).
- SportsMOT: HOTA 72.6 (MambaTrack).
- Tracking latency: 11.37 ms per frame for the motion component (MambaTrack).
For a fair comparison, we retrain MambaTrack's motion module on exactly the training sequences our loop sees, with the same detector, and give it a tuning budget matched in GPU-hours to our refinement budget. This matters most in the low-data runs. Beating it means one of two outcomes. The first is a higher HOTA and AssA (association accuracy) on the DanceTrack and SportsMOT test splits at full data. The second is matching it within one HOTA point at full data while exceeding it when only a small fraction of training sequences is available. Either must come with motion-module latency at or below 11.37 ms per frame on comparable hardware. If MambaTrack wins clearly at every data scale, the claim fails.
Three ablations isolate the proposed ingredient:
- Aggregate-only feedback. The same language model and the same number of rounds, but the only feedback is a held-out HOTA score with no episode traces. This tests whether counter-examples matter.
- Parameter-only edits. The language model may change only constants in the Kalman filter, not structure. This tests whether structural edits matter.
- No rollback. This measures how much the regression trap described in MEIC costs here.
We found no reported result on Lacuna for any language-model-refined tracker, so all three ablations are new measurements.
Why not just…?
Why not just train the learned motion model, as MambaTrack does? This is the serious threat, and we do not argue around it. With the full DanceTrack training set, a learned state-space predictor may simply win. Our evidence against it being sufficient is indirect. MambaTrack offers no structural adaptation after training and no interpretability. InstMove shows that learned temporal predictors inherit errors from their inputs (InstMove). The data-scaling experiment settles the question. If the refined program does not beat the matched-data learned model in the low-data regime or in cross-dataset transfer, the proposal's practical value reduces to interpretability alone, and we will report that.
Why not just run Bayesian optimization over a Kalman filter's hyperparameters? Much of a tracker's error may be poorly tuned noise covariances and gate sizes, so plain black-box tuning against HOTA could capture most of the gain. We test this directly. We give a hyperparameter optimizer the same number of full-pipeline evaluations as our loop, and we compare it against the parameter-only language-model ablation. If structural edits add nothing beyond tuned constants, the structural-refinement claim is refuted. The fixed-threshold failures CFME exhibits when target dynamics change (CFME) suggest constants alone are not enough. That suggestion is a hypothesis, not a result.
Why not just prompt a language model once, or let it iterate on the overall score? One-shot generation is weak on real code. Standalone GPT-4 resolves 1.74% of SWE-bench issues (MAGIS). Score-only iteration plateaus after a few rounds (CodeTransOcean). The aggregate-only ablation tests whether the same holds for motion programs. If it matches episode-level feedback, the counter-example mechanism is unnecessary.
Why not just use multi-agent review instead of tests? Debate without a ground-truth oracle drifts toward agreement on wrong answers (Adversarial Review). Moreover, tracking has a cheap, exact oracle in the replayed episodes, so we spend tokens on execution rather than argument.
Proposed Experiments
The main experiment runs on DanceTrack and SportsMOT with a single shared detector. We report HOTA, AssA, IDF1, identity switches, and motion-module latency per frame. We break down the association pass rate by gap length, because the method should help most on long occlusions and erratic motion.
Data-scaling sweep. We train or refine on increasing fractions of the training sequences, from a handful up to all of them. The same fractions are given to MambaTrack's predictor and to the hyperparameter optimizer. This produces data-efficiency curves for all three methods.
Transfer test. We refine on DanceTrack and evaluate on SportsMOT unchanged, and the reverse, against the same transfer for the learned model.
Overfitting check. Within the training sets, we compare the episode pass rate on episodes the loop saw against episodes it never saw. A large gap would show that the test suite is being memorized.
Cost reporting. We report the language-model tokens and wall-clock time spent offline, since agent loops are expensive. AutoSD averages about nine minutes per bug (AutoSD), and our cost is paid once per domain, not per frame.
Decisive outcomes. The result that would justify the approach is this combination: refined programs that beat the matched-data learned predictor at low data, transfer better across datasets, beat equally budgeted hyperparameter tuning, and lose substantially when episode traces are replaced by an aggregate score. If the aggregate-only ablation or the hyperparameter optimizer matches the full method, the central claim is wrong.
References
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Joint Detection and Online Multi-Object Tracking
- Tracking Known Three-Dimensional Objects
- Object Tracking in Satellite Videos by Improved Correlation Filters With Motion Estimations
- Occlusion Geodesics for Online Multi-object Tracking
- MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Model
- InstMove: Instance Motion for Object-centric Video Segmentation
- Advancing Code Coverage: Incorporating Program Analysis with Large Language Models
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Explainable automated debugging via large language model-driven scientific debugging
- RobotGPT: Robot Manipulation Learning From ChatGPT
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution