Scores That Can Say "Not Here": A Hybrid Ranking-Calibration Loss for Label-Efficient, Pixel-Only GUI Grounding

Software agents that operate computers from screenshots alone have to solve grounding: given an instruction like "open the layer blend mode menu," return the pixel to click. Today's best grounders print the coordinates as text, one digit at a time. This has two costs. First, the training loss treats a digit that is off by one the same as a digit that is off by five, so it has no sense of distance on the screen. Second, the model has no principled way to say that the target is not on screen at all, so it clicks something nearby. The fix people reach for is to score every screen location and apply a softmax, but a softmax has the same blind spot in a different form. Adding a constant to every score leaves its output unchanged, so the scores rank locations against each other while carrying no absolute meaning. A ranking-only model can pick the best location on the screen, but it cannot tell you whether that best location is any good.

We propose training location scores with two coupled terms. A listwise softmax over screen locations sharpens the choice between neighboring candidates. A pointwise, per-location calibration term pins the scores to an absolute scale, so a single threshold on the top score decides whether to refuse. This is the "global anchor plus local magnifier" pattern from hybrid ranking-regression losses, moved from one-dimensional scores to two-dimensional screen fields. We pair it with a pixel-only pretraining task that generates its own absence examples. The model learns to find a crop inside a screenshot, and crops taken from a different screenshot teach it to say "not here." Only then do we fine-tune on 1–5% of a labeled grounding set. The claim is that the hybrid loss, not either term alone and not a bolted-on presence classifier, is what gives both precise grounding of tiny targets and reliable refusal when labels are scarce.

Core Claim

Let $s_i$ be the score the model assigns to visual patch $i$ of a screenshot for a given instruction. Let $q_i$ be a soft target that spreads unit mass over the target element as a Gaussian. Its width follows the element's size, as in the adaptive variance of GUI-G². Let $m_i \in [0,1]$ be a per-patch "this is the target" value: the same Gaussian rescaled to peak at 1, and identically 0 when the target is absent. The loss is

$$ \mathcal{L} = -\sum_i q_i \log \frac{e^{s_i}}{\sum_j e^{s_j}} \;+\; \lambda \sum_i w_i\,\mathrm{BCE}\big(\sigma(s_i), m_i\big), $$

where $\sigma$ is the logistic function, BCE is binary cross-entropy, $\lambda$ is the trade-off weight, and $w_i$ upweights patches near the target so that background patches do not swamp the gradient.

The first term is invariant to adding a constant to all $s_i$. Its gradient therefore never fixes the overall level of the scores, and absence examples, where $q$ is undefined, cannot even be written into it. The second term fixes that level, so $\max_i \sigma(s_i)$ becomes a calibrated probability that the target is present. On its own, the second term is ordinary heatmap regression. We expect it to blur adjacent small targets, for the same reason pointwise regression fails to separate close competitors in Temporal Relational Ranking for Stock Prediction and Automated Essay Scoring. The testable prediction splits by failure mode. Listwise-only training should fail at refusal. Pointwise-only training should fail on the tiny, crowded targets of professional software. The hybrid should do both jobs.

Relation to Prior Work

Three grounders motivate the gap: SeeClick, UGround, and JEDI. Each trains a vision-language model to emit coordinates as text, from data built out of HTML attributes, LLM-written referring expressions over the page structure, or rendered React and Figma layouts. All three are pixel-only at test time, but their training data still depends on page metadata. JEDI reports that even with explicit refusal data its models still click neighboring widgets when the target is absent. We change the output form (a score field instead of digits) and the loss, and our pretraining uses only pixels.

GUI-G² supplies part of our method. It shows that Gaussian, distance-aware rewards beat hit-or-miss box rewards, and we reuse its size-adaptive Gaussians as soft targets. It differs from us in three ways: it uses these Gaussians only as rewards inside reinforcement learning (GRPO), it trains on 100K labeled examples, and it does not address refusal.

SafeGround constrains our design. It wraps a grounder at inference time, samples coordinates repeatedly, and abstains when the samples disperse. It explicitly fails when the model is consistently wrong. Our calibration is built into the scores and does not rely on sampling spread.

The loss design comes from the hybrid ranking literature. The Combo loss in Batch-Softmax Contrastive Loss adds mean squared error to an in-batch softmax ranking loss for sentence scoring. Improving Top-N Recommendation with Heterogeneous Loss proves, for linear recommenders, that the pointwise term suppresses ranking noise. Baidu Search ranking reports that rankers without pointwise anchors produce scores too unstable to combine with other signals. All of this work is one-dimensional and has no notion of absence. We carry the pattern into dense two-dimensional fields, where "uncalibrated" becomes "cannot refuse."

Self-supervised learning theory serves as our analytical lens. Contrastive Learning is Spectral Clustering on Similarity Graph shows that InfoNCE (the standard contrastive loss) performs spectral clustering on the augmentation graph. Contrastive and Non-Contrastive SSL Recover Spectral Embedding Methods shows that contrastive objectives cap representation rank at the rank of that graph. Our pretext task, which finds a crop among all screen locations, is exactly such a softmax over locations. The theory therefore predicts two things we can check. Repeated identical icons create tied edges that ranking alone cannot break. And the representations of small elements may lose rank. We measure both.

The label-efficiency results in radiology from MGCA and MRM motivate the low-label regime: domain-matched self-supervised pretraining lets 1% of labels rival full supervision. Those results are for classification and segmentation, not continuous point grounding.

Research Program

The backbone is Qwen2.5-VL-3B, the base model JEDI fine-tunes. We add a light scoring head: each visual token's final hidden state is dotted with a pooled instruction embedding to give $s_i$, then upsampled bilinearly for sub-patch precision.

Pretraining is pixel-only and uses unlabeled screenshots only. We take the screenshot images in the JEDI and UGround training pools and discard all their annotations. Queries are formed in two ways, both without HTML, accessibility trees, or LLMs. The first is a crop of the screenshot, which the model must localize. The second is OCR-read text from a region, which the model must localize. Negatives cost nothing: a query taken from a different screenshot has $m \equiv 0$. We also include hard absent cases from the same app, where the crop comes from a different screen state. Both loss terms apply during pretraining.

Fine-tuning uses 1%, 5%, and 100% random subsets of the JEDI grounding data, including its refusal examples in proportion. The weight $\lambda$ is chosen once on a held-out split at 5% and then frozen. We also report a sweep over $\lambda$, because the balancing coefficient is a known weak point of hybrid losses.

The analysis ties results back to the theory. We compute the effective rank of patch embeddings for small versus large elements under the listwise-only, pointwise-only, and hybrid objectives. We report the fraction of errors that land on a visually identical duplicate of the target. And we plot reliability diagrams for $\max_i \sigma(s_i)$.

Baselines

The naive reference is Qwen2.5-VL-3B used as-is, with text-coordinate output. Next to it is the same model fine-tuned with standard text-coordinate cross-entropy on each labeled subset. This is the SeeClick, UGround, and JEDI recipe at matched scale and data, and it is the baseline the loss change must beat. For reference points, JEDI reports 91.7% average accuracy on ScreenSpot-v2 at 7B (JEDI). It also reports 46.9% versus 36.9% for its base model on the fine-grained manipulation split of OSWORLD-G.

The strongest existing competitor is GUI-G²: GRPO with Gaussian point and coverage rewards, trained on the same backbone, the same subsets, and the same GPU-hour budget, starting from the cross-entropy fine-tuned checkpoint as in its paper. It is the strongest competitor because it already injects continuous spatial distance, which is half of our argument. At 7B with full data, GUI-G² reports 47.5% on ScreenSpot-Pro and 93.3% on ScreenSpot-v2. For refusal, the strongest competitor is SafeGround's dispersion-based abstention applied to the cross-entropy model. SafeGround reports that AUROC for Holo1.5-7B improved from 0.69 to 0.75. No reported result was found on Lacuna for refusal accuracy on OSWORLD-G by any grounder.

Four ablations isolate the key ingredient: listwise-only, pointwise-only, hybrid without pretraining, and hybrid with pretraining but without self-supervised absent negatives. Every method receives the same number of hyperparameter trials, the same data, and matched compute.

We will count the proposal as beating the baselines if both of the following hold. At 5% labels, the hybrid must match or exceed GUI-G² trained on the same 5% on ScreenSpot-Pro. At the same time, it must give higher refusal AUROC than SafeGround, with no loss in grounding accuracy.

Why not just…?

Why not just fine-tune the vision-language model with ordinary cross-entropy on coordinate text? Cross-entropy on digits has no notion of screen distance. GUI-G²'s gains over hit-or-miss training suggest that distance matters, and the large drop from ScreenSpot-v2 to ScreenSpot-Pro across models, documented in ScreenSpot-Pro, shows where it matters most. Still, at 100% labels this baseline may close most of the gap. The low-label curves are where we expect the difference, and they will show it.

Why not just use GUI-G²'s Gaussian rewards? This is the real threat for grounding accuracy. Reinforcement learning with good rewards may match our field loss on hit rate. Two things work against it. Its rewards say nothing about absence. And policy-gradient training on 1% of labels gives few distinct prompts to explore from. If GUI-G² matches us on both hit rate and refusal at matched compute, the loss contribution is not supported, and we would report that.

Why not just regress a Gaussian heatmap (the pointwise term alone)? This is standard keypoint practice, and it is calibrated. Our prediction is that it underperforms on crowded tiny targets, because background patches dominate its gradient. The pointwise-only ablation on ScreenSpot-Pro, together with the duplicate-error analysis, tests this directly. If it matches the hybrid, the listwise term is unnecessary and we drop it.

Why not just add a separate "is the target present?" classifier head? This is a simpler route to refusal, and we run it: listwise loss plus a presence head trained on the same absent examples. The argument for tying presence to the location scores is that a separate head can say "present" while the chosen location is wrong. Refusal AUROC restricted to confidently-wrong cases, which is exactly SafeGround's failure mode, decides between the two.

Proposed Experiments

The evaluation benchmarks are ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G, whose splits include refusal. Because refusal cases are scarce, we add a constructed absent-target test: each ScreenSpot-Pro instruction is paired with a different screenshot from the same application.

Metrics are click accuracy, refusal AUROC, accuracy at a fixed refusal rate, and expected calibration error of $\max_i \sigma(s_i)$. All are reported against label fraction (1%, 5%, 100%).

The decisive checks follow from the claim. The listwise-only model should show near-chance refusal AUROC despite competitive click accuracy. The pointwise-only model should lose accuracy on the smallest-element quartile of ScreenSpot-Pro. The hybrid should hold both. Removing the self-supervised absent negatives should hurt refusal mostly at 1% labels. If none of these separations appear, the central claim is false.