Inline autocomplete decides at each keystroke whether to show a suggestion or stay silent. That decision depends on two quantities. One is the chance the suggestion is right. The other is what it costs the user, in lost time, to look at a right or a wrong suggestion. The first quantity is cheap to get in a new domain: once the user finishes typing, the true continuation is known. The second is expensive. It comes from timed user studies, and every deployed system we know of reuses a handful of constants measured on casual text. We propose treating those timing measurements as the labeled data in a covariate-shift problem. "Covariate shift" means the inputs change between the data a model was built on and the data it is used on, while the input-to-output relationship stays fixed. In our setting, the inputs are cheap and plentiful: target-domain text such as medical sentences or chatbot prompts. The labels are scarce: milliseconds of pause measured on a source domain such as text messages. We reweight source timing observations so they describe target contexts, detect when that reweighting is too unstable to trust, and spend a small target user-study budget only on the contexts where transport fails.
The payoff is practical. The strongest simple policy for autocomplete is a confidence threshold, and its best value is set by the ratio of the benefit of a correct suggestion to the cost of a wrong one. If wrong suggestions are much costlier to dismiss in technical text than in text messages, a threshold tuned with text-message costs will over-suggest exactly where interruptions hurt most. The two-sentence version is this: correctness labels are free in autocomplete but cognitive-cost labels are not, so the part that needs transferring across domains is the cost. We propose to transfer it with importance-weighted, stability-gated estimation and targeted measurement, and to test whether this recovers the threshold a full target-domain user study would pick.
Why the weighting belongs on the cost, not the confidence
The obvious combination of these two literatures is to recalibrate the language model's confidence under shift, using importance-weighted temperature scaling as in Unsupervised Calibration under Covariate Shift. That paper uses a domain discriminator to reweight calibration losses so image classifiers stay calibrated on unlabeled target data, cutting ECE on Office-31 D→A from 14.7% to 6.0%. We deliberately do not do this. Its premise is that target labels are unavailable, and in autocomplete that premise fails. Every typed character reveals whether the previous suggestion would have been correct, so the threshold on correctness can be tuned directly on unlabeled-in-name-only target text.
The quantity that genuinely lacks target labels is human cost. Sequential Decision-Making for Inline Text Autocomplete motivates this proposal and supplies its cost model. That paper frames suggest-or-wait as a sequential decision problem and measured, with nine users on casual text, about 10ms to process a correct suggestion and about 50ms to recover from a wrong one, with reading time roughly independent of length. With those constants, reinforcement-learning agents (PPO, DQN, IQL) did not significantly beat a tuned threshold. The paper also assumes the constants are the same everywhere. Two other lines of work suggest they are not: the distraction penalty professional editors pay on bad corrections in Automatic Correction of Human Translations, and the difficulty of verifying medical terminology in AutoMeTS: The Autocomplete for Medical Text Simplification.
Core Claim
Write $p$ for the calibrated probability that a suggestion of length $L$ is correct, $\tau$ for the user's time per typed character, and $c_+(x)$ and $c_-(x)$ for the time cost of seeing a correct or wrong suggestion in context $x$. Showing the suggestion has positive expected value when
$$p > \frac{c_-(x)}{\tau L - c_+(x) + c_-(x)}.$$
So the right threshold depends on both length and context through the costs. Because reading time does not grow with length, longer suggestions tolerate lower confidence.
Our claim is that $c_\pm(x)$ varies across domains enough to move this threshold. We further claim it can be estimated for a target domain from source timing data $\{(x_i, c_i)\}$ and unlabeled target contexts using the self-normalized weighted mean
$$\hat c_T = \frac{\sum_i w(x_i)^\lambda c_i}{\sum_i w(x_i)^\lambda}.$$
Here $w(x) = P_T(x)/P_S(x)$ is the density ratio, and $\lambda \in [0,1]$ is the flattening exponent of adaptive importance weighting reviewed in Supervised Learning under Covarita Shift. In practice $\hat c_T$ is the output of a weighted regression over context features rather than a single mean.
The assumption doing the work is that the cost of a given kind of suggestion in a given kind of context does not depend on domain once the context features are rich enough. Those features are suggestion and prefix embeddings, token rarity, terminology density, and suggestion length. We test this assumption directly rather than taking it on faith.
Research Program
The first piece is a timing instrument. Participants compose or transcribe text while suggestions of controlled correctness, length, and content are injected. Inter-keystroke intervals, and gaze where available, give per-suggestion costs, following the measurement logic of the sequential-decision paper.
The source study runs on casual text drawn from SMS Spam (ham subset) and Webis-TLDR-17, the corpora that paper used. Three target domains are chosen for increasing distance from the source:
- chatbot prompts from the ChaI-TeA sub-datasets OASST and ShareGPT, as in ChaI-TeA;
- medical sentences from the AutoMeTS Wikipedia medical simplification data;
- technical English sentences from the ACED enterprise domains.
The second piece is the transport machinery. We estimate density ratios over joint context-and-suggestion features in two ways and compare them: a domain discriminator in the style of Pampari and Ermon, and direct least-squares ratio fitting (uLSIF) from Direct Density Ratio Estimation for Large-scale Covariate Shift Adaptation. We then fit a weighted cost regression and choose $\lambda$ by importance-weighted cross-validation.
Following ShiftBench, each target cohort of contexts, defined by terminology density or prompt type, passes a stability check. The check looks at effective sample size and uses the PSIS tail-index cutoff of 0.7. That paper uses such checks, together with empirical Bernstein bounds, to issue certify, abstain, or no-guarantee decisions on classifier precision. We repurpose the three-way decision for a different question: whether suggesting in a given cohort is certifiably net time-saving under the asymmetric cost.
The third piece handles cohorts that fail the check. For those, we choose which target contexts to time in a small follow-up study using importance-weighted active selection in the spirit of ALICE from Sugiyama's review, rather than sampling contexts at random.
Baselines
The naive reference is the constant-cost gate. It uses Chitnis et al.'s 10ms and 50ms constants, with the confidence threshold tuned on target text using revealed continuations. This is exactly what a careful practitioner would do today. Because correctness is tuned on target data, the only thing it gets wrong is the cost. That makes it the cleanest test of our central claim.
The second baseline is an unweighted cost regression trained on source timing data and applied to target contexts. It isolates the value of the density ratio. The main ablation keeps the weighting but removes flattening and the stability check. This shows whether stabilization, not weighting itself, is what makes transport usable. It matters because conformal coverage under shift can be restored with estimated weights, as in Conformal Prediction Under Covariate Shift, where coverage went from 82.2% to 91.0% against a 90% target on Airfoil, yet ShiftBench finds naive weighted procedures issuing false certifications 65% to 81% of the time.
The strongest competitor is a small supervised target study: time $m$ randomly sampled target contexts and fit costs on those alone. We run it and our transported-plus-active estimator at matched budgets of $m \in \{50, 200, 800\}$ timed suggestions. Both use the same features, the same regression family, and the same hyperparameter-search budget. The ceiling is an oracle gate built from a full target study.
On the strongest method's own terms, we can say little numerically. We found no reported result on Lacuna for net time saved or keystroke savings of threshold gates on SMS, Reddit, or ChaI-TeA. The closest figures are the 10ms and 50ms cost parameters, and ChaI-TeA's estimate that Phi-3 and Mistral-7B could save 35% to 45% of keystrokes if ranking were fixed.
Beating a baseline means lower regret in realized time saved per character on a held-out target session, relative to the oracle threshold. For the supervised study, it additionally means reaching the same regret with a smaller timing budget.
Why not just…?
Why not just keep the published 10ms/50ms constants and tune the threshold on target text? This is the most serious threat. If the costs of correct and wrong suggestions barely change across domains, this baseline is optimal and our machinery adds nothing. The first experiment settles it: we fit a domain indicator into the cost regression and test for shifts in $c_-$. If the medical or technical cost differs from the SMS cost by less than the amount needed to move the optimal threshold, we report a negative result. That outcome would still be useful, as evidence that Chitnis et al.'s constants transfer.
Why not just run a small target user study? Sometimes that is the right answer, and the stability check exists to say so. Our claim is narrower. On cohorts where source data overlap the target, transport lets a study of size $m$ perform like one of size several times $m$. On cohorts where overlap fails, active selection spends the budget better than random sampling. If random target sampling at matched budget matches our regret, the transport story fails, and we will say so.
Why not recalibrate the model's confidence with importance-weighted temperature scaling instead? As argued above, correctness labels are revealed by typing, so label-free recalibration solves a problem autocomplete does not have. We include a threshold-tuned-on-target-text arm precisely so that any gain we report cannot come from better correctness calibration.
Why not learn costs from deployment logs of pauses after suggestions? Passive logs record only the suggestions an existing policy chose to show, so they confound cost with the policy's own selection. Unbiased estimates require randomized exposure, which charges real users for wrong suggestions. Our method targets the decision made before deployment. We will compare it against a randomized-exposure log estimator at matched user exposure, to quantify how much user time each approach spends.
Proposed Experiments
The first check is whether the phenomenon exists. Using the source study plus full studies in two target domains, we test whether the cost of a wrong suggestion depends on domain after conditioning on context features. We also test whether any shift is large enough to move the optimal threshold in the equation above.
The second check is transport. We estimate target costs from source data alone and measure error against the full target study, the coverage of weighted conformal intervals on per-suggestion cost, and threshold regret. We track how each of these degrades as effective sample size falls. The prediction is that transport works for chatbot prompts, which overlap casual text, and triggers no-guarantee on dense medical terminology.
The third check is budget. On the cohorts that fail the stability check, we compare active selection against random sampling at matched timing budgets.
The final decisive check is live. New participants use the gate produced by each method on held-out target sessions, and we measure realized time saved and acceptance of wrong suggestions. The second measure matters because VIANA reports that fatigued users passively accept poor outputs, so a gate that looks efficient could be hiding degraded text quality.
References
- Unsupervised Calibration under Covariate Shift
- Sequential Decision-Making for Inline Text Autocomplete
- Automatic Correction of Human Translations
- AutoMeTS: The Autocomplete for Medical Text Simplification
- Supervised Learning under Covarita Shift
- ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots
- Direct Density Ratio Estimation for Large-scale Covariate Shift Adaptation
- ShiftBench: A Benchmark for Per-Cohort Certify-or-Abstain Decisions on Positive Predictive Value Under Covariate Shift
- Conformal Prediction Under Covariate Shift.
- VIANA: Visual Interactive Annotation of Argumentation