TL;DR
This paper introduces a web-based demo that runs a 0.8B parameter language model entirely within a browser tab to parse natural language pizza orders into structured data. By shifting inference to the client side via WebAssembly, the authors demonstrate a privacy-first approach that helps users catch "silent errors"—outputs that look perfectly valid but are factually incorrect.
What Problem Does This Paper Address?
When we use small language models (SLMs) to turn typed text into structured data (like a kitchen ticket), they can fail in two distinct ways.
- Detectable Errors: The model outputs gibberish or something that breaks the schema. These are easy to catch and reject.
- Silent Errors: The model outputs a clean, well-formed ticket that looks correct but doesn't match what the user actually said (e.g., forgetting a drink).
Silent errors are dangerous because they pass through automated checks unnoticed. Furthermore, traditional server-side processing of such data raises privacy concerns and incurs hosting costs. The authors explore whether moving these models directly into the user's browser can solve the privacy issue while providing an interactive environment for users to "hunt" for these elusive silent errors.
Methods
The researchers built a system that runs a fine-tuned Qwen3.5-0.8B model entirely on the visitor's device.
- Runtime: They used
llama.cppcompiled to WebAssembly (Wasm) via thewllamalibrary. To keep the UI responsive, the model runs in a Web Worker. - Architecture: A Service Worker is employed to handle cross-origin isolation headers, enabling multithreaded performance in the browser.
- In-Browser Scoring: They ported the official PIZZA benchmark's Python-based matcher to TypeScript. This allows the browser to judge the model's output against ground truth in real-time without ever talking to a server.
- Guardrails: Instead of a separate filter, the model itself is trained to output a "?" when it receives "off-menu" input (like "get me two large horses"), turning a potential silent error into a detectable refusal.
Datasets
The model was trained and evaluated using the PIZZA benchmark, a dataset designed for complex, end-to-end task-oriented parsing. The authors used all 1,357 test orders from this benchmark to verify that the browser-based model performed identically to its server-side counterpart.
Key Insights and Ideas
- Privacy by Construction: Because the inference happens locally, the page sends zero bytes of order text to any server. This is a massive win for privacy-sensitive applications.
- The User as the Final Audit: In a serverless setup, there are no logs for developers to audit. The person at the screen becomes the final check, making interactive UI elements (like the "Spot It" game in the demo) crucial for error detection.
- Quantization is "Free": Moving from full-precision server models to 8-bit quantized models in the browser resulted in identical output for 99.0% of test cases. The small accuracy trade-off is well worth the portability.
- Interactive Evaluation: The demo includes a "Break It" mode where users actively try to fool the model. This "human-in-the-loop" approach is often more effective at finding edge-case silent errors than static benchmarks.
Key Results
The browser-based deployment proved to be both accurate and surprisingly fast:
- Accuracy: The deployed model achieved an exact match score of 0.882, slightly outperforming the mean of the original server-side study (0.875).
- Silent Error Rate: Only 3.6% of orders resulted in silent errors, which the authors visualized as a grid of tiles to show the distribution of outcomes.
- Latency: On modern hardware like an M3 MacBook Air, the median parse time was just 383ms, well under the 1-second threshold required to maintain a user's "flow of thought." Even on a mid-range Windows laptop, it remained usable at ~3 seconds.
- Reliability: The browser and server agreed on the exact output for 99.0% of the 1,357 test orders.
Limitations and Future Work
- The "Cold Start" Problem: The first visit requires downloading an 834 MB model file, which takes about 40-45 seconds. While cached for subsequent visits, this is a high barrier for casual users.
- Hardware Sensitivity: While fast on Apple Silicon, performance on older Intel-based laptops was significantly slower, highlighting the "digital divide" in client-side AI.
- No Centralized Learning: Since no data leaves the device, developers cannot see common failures to improve the model. Future work might explore Federated Learning or anonymous error reporting to bridge this gap without sacrificing privacy.
Summary updated