Clean Tickets, Wrong Orders: Catching Silent Errors with a Language Model in Your Browser Tab

Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad, Sergey Ivanov, Ivan KhodnenkoarXiv 2026 · arXiv

TL;DR

This paper introduces a web-based demo that runs a 0.8B parameter language model entirely within a browser tab to parse natural language pizza orders into structured data. By shifting inference to the client side via WebAssembly, the authors demonstrate a privacy-first approach that helps users catch "silent errors"—outputs that look perfectly valid but are factually incorrect.

What Problem Does This Paper Address?

When we use small language models (SLMs) to turn typed text into structured data (like a kitchen ticket), they can fail in two distinct ways.

  1. Detectable Errors: The model outputs gibberish or something that breaks the schema. These are easy to catch and reject.
  2. Silent Errors: The model outputs a clean, well-formed ticket that looks correct but doesn't match what the user actually said (e.g., forgetting a drink).

Silent errors are dangerous because they pass through automated checks unnoticed. Furthermore, traditional server-side processing of such data raises privacy concerns and incurs hosting costs. The authors explore whether moving these models directly into the user's browser can solve the privacy issue while providing an interactive environment for users to "hunt" for these elusive silent errors.

Methods

The researchers built a system that runs a fine-tuned Qwen3.5-0.8B model entirely on the visitor's device.

  • Runtime: They used llama.cpp compiled to WebAssembly (Wasm) via the wllama library. To keep the UI responsive, the model runs in a Web Worker.
  • Architecture: A Service Worker is employed to handle cross-origin isolation headers, enabling multithreaded performance in the browser.
  • In-Browser Scoring: They ported the official PIZZA benchmark's Python-based matcher to TypeScript. This allows the browser to judge the model's output against ground truth in real-time without ever talking to a server.
  • Guardrails: Instead of a separate filter, the model itself is trained to output a "?" when it receives "off-menu" input (like "get me two large horses"), turning a potential silent error into a detectable refusal.
Figure 2: What runs where. Everything in the grey box runs in the visitor’s tab. The model file arrives once from Hugging Face; no order text leaves.

Datasets

The model was trained and evaluated using the PIZZA benchmark, a dataset designed for complex, end-to-end task-oriented parsing. The authors used all 1,357 test orders from this benchmark to verify that the browser-based model performed identically to its server-side counterpart.

Key Insights and Ideas

  • Privacy by Construction: Because the inference happens locally, the page sends zero bytes of order text to any server. This is a massive win for privacy-sensitive applications.
  • The User as the Final Audit: In a serverless setup, there are no logs for developers to audit. The person at the screen becomes the final check, making interactive UI elements (like the "Spot It" game in the demo) crucial for error detection.
  • Quantization is "Free": Moving from full-precision server models to 8-bit quantized models in the browser resulted in identical output for 99.0% of test cases. The small accuracy trade-off is well worth the portability.
  • Interactive Evaluation: The demo includes a "Break It" mode where users actively try to fool the model. This "human-in-the-loop" approach is often more effective at finding edge-case silent errors than static benchmarks.
Figure 3: What a visitor does. (1) The ticket for a real test order streams in as they type. (2) A saved ticket that parses and is wrong: two pizzas became one, with “small” as a topping. (3) Their own attempt to fool the model, judged by them. (4) Off-menu input bounces off with “?”; an order passes.

Key Results

The browser-based deployment proved to be both accurate and surprisingly fast:

  • Accuracy: The deployed model achieved an exact match score of 0.882, slightly outperforming the mean of the original server-side study (0.875).
  • Silent Error Rate: Only 3.6% of orders resulted in silent errors, which the authors visualized as a grid of tiles to show the distribution of outcomes.
  • Latency: On modern hardware like an M3 MacBook Air, the median parse time was just 383ms, well under the 1-second threshold required to maintain a user's "flow of thought." Even on a mid-range Windows laptop, it remained usable at ~3 seconds.
  • Reliability: The browser and server agreed on the exact output for 99.0% of the 1,357 test orders.
Figure 4: All 1,357 PIZZA test orders, each a tile, run by the deployed model inside Chromium and scored in the tab. Tiles are grouped by outcome. The dark tiles on the right parse cleanly and are wrong.

Limitations and Future Work

  • The "Cold Start" Problem: The first visit requires downloading an 834 MB model file, which takes about 40-45 seconds. While cached for subsequent visits, this is a high barrier for casual users.
  • Hardware Sensitivity: While fast on Apple Silicon, performance on older Intel-based laptops was significantly slower, highlighting the "digital divide" in client-side AI.
  • No Centralized Learning: Since no data leaves the device, developers cannot see common failures to improve the model. Future work might explore Federated Learning or anonymous error reporting to bridge this gap without sacrificing privacy.

Summary updated

Abstract

A small language model that turns a typed order into structure can fail in two ways. It can print something broken, which any system rejects. Or it can print a clean, well formed order that nobody placed. Our companion work calls the second kind a silent error, and it is the dangerous one. We present a web demo in which a fine-tuned 0.8B parameter model runs entirely inside the visitor's browser tab, through llama.cpp compiled to WebAssembly. No order text leaves the device. The page parses orders as the visitor types, scores every ticket with the PIZZA benchmark's own matcher ported to TypeScript, refuses anything off the menu with "?" so the model can face the open web, and lets visitors hunt for silent errors or try to cause one. Run in the browser on all 1,357 PIZZA test orders, the deployed model writes the same output as the server evaluation for 99.0% of them and reaches the same exact match (0.882); we also report its latency on 3 consumer laptops. Demo: https://wrong.cuyor.com
BibTeX
@misc{ahmed2026clean,
  author = {Umar Ahmed and Maxim Dunaev and Mohammad Nour Al Awad and Sergey Ivanov and Ivan Khodnenko},
  title = {{Clean Tickets, Wrong Orders: Catching Silent Errors with a Language Model in Your Browser Tab}},
  year = {2026},
  eprint = {2611.18284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18284},
}
Figure