Automated tests and screen-controlling AI agents often act before the app on screen is ready. They click a button that has not been drawn yet, or type into a form that is still loading. Two families of fixes exist. One watches the app's internal structure: the page's element tree, or Android's list of on-screen widgets. The other watches the pixels with a small image classifier. Each fails in a characteristic way. The structural signal reports "ready" before pixels are drawn and is blind to content it cannot see, such as canvases and third-party plugins. The visual signal mistakes looping GIFs and videos for ongoing loading. This note proposes treating the two signals the way composite-materials engineers treat hybrid fibers. Fine and coarse fibers each stop cracks at a different size, and the benefit of mixing them is measured explicitly against what each fiber does alone. We will build a hybrid readiness oracle and quantify whether the combination is superadditive. The test is whether it removes more failures, at lower waiting cost, than the sum of what each signal removes alone.
The practical stakes are concrete. Waiting problems cause 45.1% of flaky user-interface tests (UI-Based Flaky Tests Dataset). Current phone-control agents have no real "wait" action and fall back on fixed pauses between steps (Hi-Agent, AndroidLab). The contribution is twofold. The first part is a measured map of which signal catches which failures. The second part is an oracle and an agent "wait" action built on that map, with a pre-stated outcome that would show the hybrid idea is not worth it.
Core Claim
Asynchronous UI events unfold on different time scales. Element-tree mutations take milliseconds. Rendering and animation take hundreds of milliseconds. Network and third-party work can take seconds. The structural and visual signals cover different but overlapping parts of this range.
We claim that once failures are labelled by which signal could have detected readiness, the overlap is small enough that a combined oracle beats both component oracles. It should beat the stronger of the two on failures fixed at matched waiting overhead, and not merely match it. The combined oracle uses a learned arbiter that looks at where the signals disagree. Simple AND or OR rules over the same signals should not capture this gain, because each rule inherits one component's characteristic error. AND inherits false pauses from looping media. OR inherits premature firing from the gap between "the widget exists" and "the widget is drawn."
Borrowing the hybrid-effect index used for hybrid fiber concrete (Research on mechanical properties and prediction methods of hybrid fiber concrete for airport pavements, which reports $S_{GF}>1$ for most fiber pairs), we define a synergy index on failure reduction:
$$S = \frac{F_0 - F_{AB}}{(F_0 - F_A) + (F_0 - F_B)}$$
Here $F_0$ is the failure count under a fixed-delay reference. $F_A$, $F_B$ and $F_{AB}$ are the failure counts under the structural oracle, the visual oracle and the hybrid oracle, all measured at a matched average waiting overhead. $S>1$ means the hybrid fixes failures that neither component fixes. $S\approx$ (best single)/(sum) means the signals are redundant.
The materials side supplies a second, cautionary analogy. Hybrid concrete degrades when fiber dosage passes a threshold and fibers clump together (same paper). We therefore test whether adding a third signal, network idleness, raises or lowers $S$ once its latency and false waits are counted. The materials literature serves here as an evaluation lens and a source of hypotheses, not as a mechanism. We do not claim UI timing obeys fracture mechanics.
Research Program
The first stage is a complementarity audit. We take the async-wait subset of the 235-test UI-Based Flaky Tests Dataset and the 122-test WEFix benchmark (WEFix). We re-run each test under instrumentation that records three streams together:
- element-tree mutation events, captured as in WEFix's mutation recorder;
- screen frames scored by a MobileNetV2 rendering classifier of the kind used in AdaT (Efficiency Matters);
- network activity.
For each flaky failure we label which streams first showed readiness, and when. This yields an empirical overlap matrix, which is the central diagnostic.
The second stage builds the hybrid oracle. The arbiter is a small classifier that takes recent features from each stream and outputs "fire now" or "keep waiting." It is trained on frames where the streams disagree, with ground truth taken from whether the subsequent action succeeded on repeated runs. Two disagreement cases matter most. If the element tree is stable but pixels change in a bounded region, the arbiter should treat that as media rather than loading. If pixels are stable but mutations continue, the arbiter should treat that as invisible churn.
The third stage exposes the oracle to agents as a wait primitive. The action blocks until the oracle fires or a cap is reached. We integrate it into the AndroidLab harness in place of its fixed inter-action delay (AndroidLab).
Baselines
The naive reference is a fixed delay between actions. We sweep the delay from 0.2 to 2 seconds, matching the range WEFix reports for static waits. That range costs 2.33x to 8.84x runtime overhead (WEFix). Each delay setting gives one point on a failures-versus-overhead curve.
The strongest structural method is WEFix itself, run with the original tool on the same 122 tests and the same browsers. It reports a 98.4% fix rate (120 of 122) at 1.25x average overhead (WEFix). This benchmark is nearly saturated, so beating WEFix on it means one of two things. Either we fix the remaining out-of-structure cases at no worse than 1.25x overhead, or we match 98.4% at clearly lower overhead.
To avoid overfitting to a ceiling, we also build a held-out stress set from WEFix's own stated blind spots: third-party widgets, canvas rendering and external services. We also include pages with looping media, which the WeChat study identifies as a source of false pauses (Towards Efficient Record and Replay).
The strongest visual method is an AdaT-style four-class classifier. On mobile it is complemented by WeReplay's two-class classifier. AdaT reports 99.8% state accuracy and reproduces 100% of target crashes versus 44% for a 200ms throttle (Efficiency Matters). WeReplay reports 92.1% precision, 93.3% recall and 100% replay success at 18.45 s per scenario (Towards Efficient Record and Replay). Where we cannot obtain the WeChat data, we retrain on Rico frames using the same architecture and the same training budget as our arbiter's visual branch.
The ablations that isolate the key ingredient are fixed AND and OR combinations of the identical signals, plus a hand-tuned weighted vote. All share the same feature streams. Only the learned disagreement-aware arbiter is withheld. All methods are compared at matched overhead along the failure-versus-overhead curve, not at a single operating point.
On the agent side, the reference is AndroidLab's fixed wait. The reported success figures are about 31% for GPT-4o zero-shot and 21.50% for fine-tuned open models (AndroidLab). We know of no reported result on Lacuna for any agent equipped with a dynamic wait action.
Why not just…?
Why not just lengthen the fixed timeout? Longer timeouts trade failures for time on a curve that dynamic methods dominate. Static waits cost 2.33x–8.84x overhead against WEFix's 1.25x (WEFix). Industrial "10x wait" replay took about 153 s per scenario against 18.45 s with visual inference (Towards Efficient Record and Replay). The delay sweep in our baselines reproduces this curve on our own tests.
Why not just use the structural oracle, since WEFix already fixes 98.4%? This is the most serious objection on web tests. If the audit shows that almost every async-wait failure is structurally detectable, then the hybrid is unnecessary on the web, and we will report that. The case for the hybrid rests on three settings: the out-of-structure stress set, mobile, where widget lists report readiness before drawing and code-level replay fell to 21.7% success on some devices (Towards Efficient Record and Replay), and agents.
Why not just use the visual oracle alone? Visual classifiers misread looping media as loading (Towards Efficient Record and Replay). They also cannot see invisible state changes that a later action depends on. The looping-media slice of the stress set measures exactly this gap.
Why not just AND or OR the two signals? This is a real threat. If a fixed rule reaches the arbiter's curve, the learned component adds nothing. The finding would then reduce to the audit plus a simple rule, which is still useful but a smaller claim. The rule ablations at matched overhead settle the question directly.
Why not just let the agent decide when to wait by adding "wait" to its action list? A language model choosing to wait still has to judge readiness from a screenshot, which is the visual oracle's job done worse and more slowly. We include this as an agent baseline: the same model, with a prompt-level wait action of fixed duration.
Proposed Experiments
The first decisive check is the overlap matrix from the audit. If fewer than about one in ten async-wait failures are caught by exactly one signal, the bound on $S$ is close to (best single)/(sum), and the program stops at a negative result. This threshold is fixed before the audit is run.
Second, we plot failures-versus-overhead curves for every baseline on the WEFix tests, the flaky-dataset subset and the stress set, and report $S$ at three overhead levels. Third, we add the network-idle stream and test whether $S$ rises or, as with fiber clumping, falls once its latency is counted.
Fourth, on AndroidLab we compare task success and wall-clock time per task under three conditions: fixed waits, the prompt-level wait action, and the oracle-backed wait primitive. We also inject artificial loading delays to mimic slow apps, the condition Hi-Agent names as unresolved (Hi-Agent). The hybrid succeeds if it holds success under injected delay at lower time cost than the best fixed wait, and if it beats both single-signal waits on that test.
References
- UI-Based Flaky Tests Dataset
- Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control
- AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
- Research on mechanical properties and prediction methods of hybrid fiber concrete for airport pavements
- WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests
- Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference
- Towards Efficient Record and Replay: A Case Study in WeChat