Autonomous camera systems now steer the drone and the lens together. They move the camera body while also adjusting zoom, aperture, and focus distance, and they do this from what a perception network sees in each frame. Such controllers are only ever compared in hand-built simulator setups on whatever machine the authors had. We argue this is a scientific problem, not just an engineering nuisance. The main thing that separates a good controller from a bad one is often timing: how stale the target estimate is when the lens is commanded. Timing is exactly what an uncontrolled host machine randomizes. The proposal is a benchmark harness that makes timing a recorded, replayable experimental variable. It runs the simulator in lockstep with the controller and injects measured latency traces deterministically. Each run is scored by executable checks computed from a frozen log of camera state, not by human or vision-language-model judgment.
The two-sentence version is as follows. Camera controllers that coordinate focus with motion are unusually sensitive to perception delay, because a shallow depth of field turns a few hundred milliseconds of lag into visible blur. So their benchmark scores and rankings should depend on host load unless latency is controlled. We will build a harness that freezes and replays latency, and use it to measure how much of the reported differences between such controllers is algorithm and how much is execution timing.
Why timing, and why this domain
The thin-lens camera model replaces the pinhole camera of standard robotics. It is what CineMPC optimizes and what CinemAirSim renders in Unreal Engine. Under this model, a point at distance $d$ seen through a lens of focal length $f$, f-number $N$, and focus distance $s$ blurs into a disk of diameter
$$ b = \frac{f^2\,|d - s|}{N\,(s - f)\,d}. $$
Suppose the controller sets focus from a target estimate that is $\tau$ seconds old, and the target moves toward or away from the camera at speed $v$. Then the focus error is roughly $|d-s| \approx v\tau$. Blur therefore grows linearly in latency and in $f^2/N$. The linear dependence on $\tau$ comes from that first-order lag, and the $f^2/N$ factor comes directly from the lens formula. Long lenses at wide apertures, the shots cinematographers actually want for subject separation, are precisely where delay matters most.
CineMPC reports a full perception-to-control loop of about 631 ms on an embedded Jetson board. That is the same order of magnitude as plausible variation from GPU contention on a shared machine. Pose-only controllers degrade gracefully under delay. Controllers that also drive focus should not, and this is the regime the literature has not measured.
The reproducibility literature motivates the concern but does not address it. The flaky-tests study interviewed practitioners about non-deterministic tests. It notes that CPU throttling, memory contention, and GPU driver differences produce nondeterminism even inside containers, and that the most common remedy, rerunning, hides real bugs. This finding constrains the proposal: it rules out "just containerize it" as a sufficient design. Bencher supplies the method we reuse for isolation. It wraps benchmarks in Docker or Singularity and talks to optimizers over remote procedure calls, but its benchmarks are static function queries with no rendering or closed-loop time. SciAgent supplies the auditing pattern. It writes an append-only log of every tool call and hands it to a verifier that never saw the agent's reasoning, but it audits batch scientific outputs, not continuous trajectories. TerminalTraj validates terminal-agent work with executable tests run inside the container, rather than with language-model judges. We adopt that stance but replace the test files with optical and compositional checks. The Agentic Benchmark Checklist audited ten agent benchmarks and found that 70% violated both task and outcome validity. It also found that a do-nothing agent scored 38% on τ-bench. This motivates our insistence on trivial-controller reference points.
Core Claim
Under controlled latency, we expect three things. First, the run-to-run variance of focus and framing metrics for intrinsic-aware camera controllers on an unmanaged host will be large enough to reorder controllers. Second, containerization alone will not remove it. Third, deterministic latency replay will both remove it and reveal a latency-sensitivity curve that is a property of the controller and worth reporting. The novel part is not isolation, which Bencher already provides. It is treating perception delay as a logged, injectable variable for a vision-in-the-loop controller, and scoring with thin-lens checks derived from that log.
Research Program
The harness has three parts. The first is a containerized CinemAirSim service with pinned engine assets and graphics stack, exposing lens and pose control over a Bencher-style remote-procedure interface.
The second part is a stepping layer. It pauses the simulator after each rendered frame and advances simulated time only when the controller returns a command. On top of this it adds a delay schedule $\tau_t$ that holds the command back by a specified amount of simulated time. Schedules can be constant, drawn from a seeded distribution, or replayed from traces recorded on real hardware under named load conditions, for example an idle Jetson, a Jetson with a concurrent pose estimator, or a desktop GPU shared with a training job. Lockstep alone removes the problem entirely; the delay schedule puts it back in a form that can be reproduced.
The third part is an append-only log of every frame's camera pose, focal length, aperture, focus distance, ground-truth target depth and pixel location from the simulator, and the controller's commands and estimates. Executable checkers then compute from this log the blur diameter on the designated sharp subject, whether the subject lies within the depth-of-field limits, framing error against the requested composition rule, actuator rate violations, and, for dolly zooms, drift in the subject's screen size. A fresh-context audit in SciAgent's style is kept only as a secondary layer, for violations that the checkers cannot express.
Baselines
The naive reference points come first, following the checklist's advice. One is a camera frozen at its initial pose and lens settings. Another is a camera that tracks pose but holds the lens fixed. These tell us how much of any score is achievable without coordination, and an evaluation is suspect if either scores well.
The strongest existing closed-loop method is CineMPC, a nonlinear model-predictive controller that jointly optimizes drone pose, zoom, focus, and aperture against depth-of-field, composition, relative-pose, and focal-length costs. It is the strongest competitor because it is the only cited system that closes the loop over all three lens parameters on moving targets. The file contains its 631 ms loop latency and the user-study result that it reached compositions in 81 s versus 224 s for non-expert manual operators. No reported result was found on Lacuna for CineMPC's focus or framing error under repeated runs or varying host load, so we have no published accuracy number to beat. The comparison is between conditions, not against a paper's figure.
We also run CineTransfer, which extracts framing and focus style from a reference clip and hands it to CineMPC. It reports a focus-style error reduced from 27.5% to 5.7% and a chest-joint tracking error of 0.065 pixels in simulation, both measured without any timing control. Our reproduction checks whether these hold across load conditions.
As an open-loop learned generator we use ChatCam. It turns prompts into tokenized trajectories over rotation, translation, focal length, and velocity, and reported translation mean squared error of 5.3 against 17.7 for LERF on static neural scenes. We execute its trajectories in the dynamic simulator to see how a static-scene planner degrades with moving targets. Spline interpolation between keyframes is the simplest directed trajectory. Finally, the key ablation is CineMPC with the depth-of-field cost removed and focus slaved to the latest detection. It isolates whether the latency sensitivity we predict comes from the coupled focus objective or from tracking in general.
Fairness is enforced by running every controller in the same container image, on the same scenes and seeds, with the same delay schedules and the same compute budget. Each method's tuning gets a fixed number of trials on a held-out scene. A controller "wins" only if its advantage on a metric survives across all replayed load conditions, with confidence intervals over seeds that do not overlap.
Why not just…?
Why not just put CinemAirSim in a Docker container? This is necessary and we do it, but the flaky-tests study reports that host throttling and GPU driver differences leak through containers. In a vision-in-the-loop controller that leakage lands directly on $\tau$. Our first experiment directly compares container-only runs against replayed-latency runs. If container-only variance is already negligible, we will say so, and the stepping layer becomes optional.
Why not just run the simulator in lockstep with zero delay? This is the real threat, because it is simpler and fully deterministic. Its cost is that it erases the effect the blur equation predicts. A controller with a 631 ms loop would look identical to one with a 30 ms loop, so benchmark winners might fail on hardware. The latency-sweep experiment settles this. If rankings under zero delay match rankings under realistic replayed traces, lockstep suffices and our delay layer adds nothing. If they diverge, zero-delay lockstep is measurably misleading.
Why not just rerun each configuration many times and report variance? Rerunning is what half of the interviewed practitioners do, and the same study warns it masks real nondeterministic bugs. Variance from uncontrolled load also cannot be attributed to a cause or reproduced elsewhere. We include many-seed reruns on an unmanaged host as a condition and test whether their spread is explained by the logged latency. If it is, a recorded trace is strictly more informative than repetition.
Why not just have a vision-language model judge the footage? SciAgent notes that language-model verifiers have uncalibrated confidence and residual hallucination, and the checklist documents a 33% overestimation from naive scoring on CVE-Bench. Blur on a subject and depth-of-field membership are computable exactly from logged depth and lens state, so judging them by eye throws away ground truth. We test this rather than assume it, using injected faults.
Proposed Experiments
Scenes are the CinemAirSim recreations of The Revenant and Road to Perdition and CineTransfer's football-player-in-mountains scenario, extended with scripted target motion at several radial speeds. The roughly 1,000 Blender-keyframed prompt-trajectory pairs released with ChatCam supply directed shots, including dolly zooms, for the open-loop methods.
The first experiment measures nondeterminism. It runs each controller 30 times under four conditions: unmanaged host, container only, zero-delay lockstep, and replayed traces. It reports per-metric spread and rank stability across conditions, measured as rank correlation between controller orderings.
The second experiment is a latency sweep. It holds the scene fixed and varies constant $\tau$ and aperture $N$, testing whether subject blur scales as $v\tau f^2/N$ for the coupled controller but not for the pose-only ablation. Agreement would confirm that coupling is what makes these systems timing-fragile. Disagreement would mean predictive focus models already compensate.
The third experiment compares verifiers. We plant faults in otherwise clean runs: focus frozen mid-shot, a dolly zoom with mismatched retreat speed, and a silent fallback of the detector from GPU to CPU that raises latency. This last fault is the camera analogue of the cluster-to-local fallback SciAgent caught. We then measure detection rate and false alarms for the executable checkers, a fresh-context language-model auditor reading the log, and a vision-language judge reading only the video, against human annotation.
The decisive check is the second experiment's ranking result. If controllers swap order between zero-delay and realistic replayed latency, the community's current comparisons are measuring hosts as much as algorithms, and replayable timing should become standard. If they do not, we will report that simple lockstep is adequate and that the remaining contribution is the executable thin-lens scoring.
References
- CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene Composition
- CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes
- A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests
- Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example
- ChatCam: Empowering Camera Control through Conversational AI