Semantic Importance Weights for Asynchronous GUI-Agent Reinforcement Learning

Phone and web agents trained with reinforcement learning (RL) are slow to train because each episode runs on a real emulator. The fix that systems researchers favor is to decouple acting from learning: many emulators collect episodes with a slightly old copy of the policy while a GPU learner keeps updating. The cost of that fix is "policy lag," and it is usually handled by reweighting old data by how likely the current policy is to produce the same output. This proposal argues that for GUI agents this ratio is measured over the wrong object. A modern agent writes a reasoning paragraph and then emits something like click(412, 880). The probability ratio over that whole string falls toward zero after a few learner updates, even when the new policy would still tap the very same "Add to cart" button. Off-policy correction then discards exactly the data that asynchrony was meant to provide.

The core idea is to compute importance weights over what an action does, not over how it was written. Every executed action is mapped to a semantic class: action type, the target UI element, and the normalized text typed. The off-policy ratio becomes the ratio of the current and behavior policies' probabilities of choosing that class, estimated by sampling. If the hypothesis holds, an asynchronous learner can tolerate much more lag before its corrections cut off useful experience. Throughput can then rise without the instability that pushes current systems back to synchronous training.

Core Claim

Let $s$ be a screen state, $a$ the full model output (reasoning plus action string), and $\phi(s,a)$ the semantic class of the executed action. Let $\mu$ be the actor's stale behavior policy and $\pi$ the learner's current policy. Standard corrections use the token-level ratio $\pi(a\mid s)/\mu(a\mid s)$. We instead use

$$ \rho^{\text{sem}}_t = \frac{\pi(c_t \mid s_t)}{\mu(c_t \mid s_t)}, \qquad \pi(c \mid s) = \sum_{a:\,\phi(s,a)=c} \pi(a \mid s), $$

where $c_t=\phi(s_t,a_t)$ is the class actually executed. Each marginal is estimated by drawing $K$ samples from the relevant policy at $s_t$ and counting how many fall in $c_t$.

The ratio is unbiased for a well-defined problem: if all actions within a class produce the same distribution over next screens, the environment depends on the class alone. Retrace-style returns computed with $\rho^{\text{sem}}$ are then valid corrections for a policy over classes. The token-level log-likelihood still carries the gradient, so the model keeps learning how to write. The prediction is concrete. Under realistic lag, the token-level trace coefficient $\min(1,\rho_t)$ collapses on most transitions, while the semantic coefficient stays near one whenever the policy's choice of target has not changed. More of each trajectory therefore survives correction.

This targets Gap 1 in the research file and is distinct from every intersecting system. DistRL is the direct predecessor and supplies the method we modify. It runs an asynchronous host learner with remote phone or emulator workers and corrects lag with Retrace($\lambda$) plus distributed prioritized replay ("A-RIDE"), but its ratios are over raw action outputs. Action Semantics Learning motivates the proposal. It shows that rewarding actions by the state transition they cause, rather than by matching the literal string, improves robustness. It uses that idea only as a reward and training signal in synchronous pipelines, never inside off-policy correction.

FLIN and AutoRPA give the design intuition for the class map. FLIN ranks "concept-level" web actions. AutoRPA rewrites hard-coded clicks into element-level procedures such as find_element(text="Submit"). Both show that element-level descriptions are more stable than raw coordinates.

Research Program

The first piece is the class map $\phi$. When an accessibility tree or DOM is available, a tap's coordinates resolve to the smallest enclosing interactive element. That element is identified by its role, text, and resource ID rather than its position. Typed text is normalized, and swipes are bucketed by direction and scroll container. When metadata is missing, as with canvas views or unlabeled icons, we fall back to Set-of-Mark visual grounding: candidate elements are drawn as numbered marks on the screenshot, in the style of Ferret-UI 2. The last resort is a coarse screen-grid cell. The map is deliberately cheap and deterministic, with no world-model call on the critical path.

Whether $\phi$ actually captures "same effect" is tested, not assumed. AndroidWorld emulator snapshots let us execute pairs of actions that $\phi$ calls equivalent from the same state and compare the resulting screens and app databases.

The second piece is estimating the marginals. Actors already query an inference server. At collection time the server returns $K$ extra samples for $\mu(c_t\mid s_t)$, drawing them in the same batch, which costs throughput but not latency. At learning time a separate sampling pool computes $\pi(c_t\mid s_t)$ on replayed states without blocking the gradient step. This follows the pattern of Online Intrinsic Rewards from LLM Feedback, which keeps slow LLM queries off the fast RL update loop. Small $K$ makes the estimate noisy, so we smooth both marginals with a floor of $1/(K+1)$ and study $K\in\{4,8,16\}$. The extra sampling compute is charged to our method in every comparison.

The third piece is the learner. It is DistRL's A-RIDE with its prioritized replay unchanged and $\rho^{\text{sem}}$ substituted into the Retrace traces and the policy-gradient weight. This makes the comparison a one-line change in the correction rather than a new system.

Baselines

The naive reference point is synchronous on-policy training with no lag and no correction, run at the same model size and environment-step budget. It defines the ceiling on sample efficiency and the floor on throughput. The synchronous-group design of MobileRL is the strongest representative and a serious competitor. It reports an 80.2% success rate on AndroidWorld with a 9B model, but its distributed emulator cluster must synchronize each group of trajectories. We run a GRPO variant of it on identical emulator counts, compared under matched GPU-hours and matched environment steps.

The strongest existing asynchronous method is DistRL's A-RIDE with token-level Retrace. DistRL reports a 73.2% success rate on general Android tasks, 2.4x faster data collection, and 3x better training efficiency than DigiRL. We reimplement it on the same backbone, worker count, replay settings, and hyperparameter search budget, using an open-weight task judge so neither method depends on an external API. Beating it means one of two outcomes. The first is higher final success on AitW General and Web Shopping at matched wall-clock. The second is equal success reached in at least 25% less wall-clock, with stability maintained at lags where A-RIDE degrades.

Three ablations isolate the key ingredient. The first computes the token ratio over the action tokens only, ignoring reasoning tokens. The second replaces $\phi$ with a random partition of matched class sizes, which tests whether any marginalization helps or only meaningful marginalization does. The third varies the granularity of $\phi$, from element-level to element plus typed text to coarse grid cells. A V-trace learner as in IMPALA, and an uncorrected stale learner in the style of decoupled LLM post-training systems such as AREAL, complete the set.

Why not just…?

Why not just sync weights more often so lag stays small? This is the cheapest fix and a real threat. Frequent synchronization reintroduces waiting on the heavy-tailed episode lengths that DistRL documents, where one task takes seconds and another minutes. It also costs transfer time for multi-billion-parameter weights. We settle this by sweeping the synchronization interval for both corrections and plotting success against wall-clock. If token-level A-RIDE at short intervals matches our method at long intervals for the same wall-clock, the proposal's value reduces to a convenience, and we will report that.

Why not just compute the ratio over the action tokens and drop the reasoning? This removes the main source of ratio collapse at almost no cost, so it is the strongest simple alternative. It is biased, however: it ignores that reasoning changes which action gets chosen. It also still separates syntactically different outputs with the same effect, such as click(410,882) versus click(415,876) on one button, or two phrasings of the same search query. It is included as an explicit ablation. If it matches semantic weights, the honest conclusion is that reasoning tokens, not syntactic variety, were the problem.

Why not just train synchronously as MobileRL does, given its 80.2% on AndroidWorld? Synchronous training clearly reaches high accuracy. The open question is cost: what it takes to get there, and whether asynchrony with a better correction reaches the same point faster. FlexRL measured data loading alone at over 57% of iteration time in a synchronous multimodal pipeline. The matched-wall-clock comparison decides this. A loss to synchronous training at equal GPU-hours would be a negative result worth publishing.

Why not just train offline, as Digi-Q does? Digi-Q reaches 71.2% on AitW General from static data with a frozen backbone and a small Q-value head. DigiRL, however, documents that frozen policies decay as live apps and websites drift, and offline data cannot cover new error states. Offline training is therefore complementary. We report Digi-Q as an initialization-quality reference, not as a competitor to an online method.

Proposed Experiments

The first experiment decides whether the project should continue. Using the DistRL setup on AitW General and Web Shopping from AndroidInTheWild, we inject controlled lag of 0, 2, 4, 8, and 16 learner updates. For each lag we measure the fraction of transitions whose truncated trace coefficient falls below 0.1, and the effective sample size of each replay batch, under token-level and semantic ratios. If the token-level traces are not being cut at realistic lags, the premise fails and we stop. This could happen for agents that emit short outputs with no reasoning, as some DigiRL-style models do. We therefore run this both with a short-output backbone and with a reasoning-emitting backbone, and expect the effect to concentrate in the latter.

The second experiment validates $\phi$. On AndroidWorld we restore emulator snapshots and execute pairs of actions $\phi$ calls equivalent. We report how often the resulting screen and the ADB-checked app state agree, split by whether accessibility metadata was available. Low agreement on the visual fallback path would bound where the method applies.

The main experiment trains each baseline to convergence on AitW General, AitW Web Shopping, and AndroidWorld. It reports success rate, success against wall-clock, success against environment steps, and the variance of training curves across three seeds. Sampling overhead for $K$ is included in every wall-clock figure.

A final stress test applies to AndroidWorld apps a layout perturbation of the kind that motivated ASL: elements are reordered and restyled while their function is preserved. We ask whether replay collected before the perturbation is still usable under semantic weights. The method predicts it should be, because element identity, not position, defines the class.