Can LLMs Test Terminal User Interfaces?

Chao Peng, Ruida Hu, Ajitha Rajan, Tegawendé F Bissyandé, Jacques Klein, Cuiyun GaoarXiv 2026 · arXiv

TL;DR

Terminal User Interfaces (TUIs) are booming in developer tools but remain severely under-tested due to a lack of dedicated automation methods. This paper introduces the first open, multi-language TUI testing benchmark and evaluates frontier LLMs against random exploration. The surprising verdict: while LLMs are vastly more efficient per interaction, simple random testing often wins under fixed time budgets due to its massive throughput advantage.

What Problem Does This Paper Address?

Terminal User Interfaces (TUIs)—think of rich, text-based applications like system monitors (bottom), git clients, or LLM coding agents—occupy a strange middle ground between Command-Line Interfaces (CLIs) and Graphical User Interfaces (GUIs).

As shown in the paper's comparison, they defy existing testing paradigms:

  • Unlike CLIs, which are stateless input/output transducers, TUIs maintain complex, screen-oriented states over time.
  • Unlike GUIs, TUIs do not expose a structured accessibility tree, DOM, or addressable widget IDs. The entire observable state is just a two-dimensional grid of raw characters and style attributes.
A technical comparison of CLI, GUI, and TUI interaction models.

Because no dedicated testing methodology exists, TUI testing in the wild is abysmal. The authors surveyed 197 real-world TUI applications and found that only 12% of test code exercises the interface layer at all. Worse, nearly half (45%) of those UI tests are completely static—they render a single frame and check it, never sending a single keystroke or mouse event.

Methods

To bridge this gap, the authors built a complete automated testing ecosystem for TUIs consisting of two core tools:

  1. tuicov: A language-agnostic coverage toolkit. It statically detects widget construction sites across different programming languages and injects lightweight runtime probes to measure "widget coverage" alongside native line coverage. Crucially, it is designed to flush coverage data even when an interactive application is forcefully terminated.
  2. tuibot: An automated testing framework that drives TUIs headlessly via a pseudo-terminal (PTY).

Using tuibot, they evaluated four exploration strategies:

  • random: Model-free random keystrokes (the baseline).
  • llm-guided: An LLM observes the text-grid screen at each step and selects the next action.
  • llm-guided-derived: The LLM-guided explorer is seeded with automatically derived command-line arguments and input fixtures.
  • llm-generated: An LLM reads the source code offline to generate natural-language test scenarios that steer the explorer.

The authors also developed a content-aware crash oracle. Instead of relying on non-zero exit codes (which are highly noisy), it parses the final rendered screen buffer for explicit tracebacks, panics, or fatal signals.

Figure 1: Control flow of the released tuibot and tuicov toolchain.

Datasets

The authors introduced a new, open benchmark derived from awesome-tuis. They curated 197 headlessly runnable applications spanning the four dominant TUI frameworks and languages:

  • ratatui (Rust)
  • bubbletea (Go)
  • textual (Python)
  • ink (TypeScript)

Each application is packaged as an instrumented Docker image, creating a standardized testbed for cross-language TUI evaluation.

Key Insights and Ideas

  • The Throughput vs. Efficiency Trade-off: LLM-guided testing is incredibly smart but slow. Because of API round-trip latencies, LLMs only execute a dozen or so steps within a 10-minute window. Random testing, by contrast, can fire off hundreds of keystrokes.
  • Launch Inputs are the Ultimate Bottleneck: Many TUIs crash or exit immediately if they aren't provided with valid files, arguments, or mock environments. Automatically deriving these launch inputs (llm-guided-derived) yielded the largest practical boost in code coverage, proving more critical than smart step-by-step navigation.
  • Coverage Decouples from Fault-Finding: In traditional software testing, higher code coverage usually means more bugs found. In interactive TUIs, this relationship breaks down due to a session-truncation confound: a successful crash immediately terminates the test run, capping the coverage it can achieve and penalizing effective bug-hunters.
  • Raw Exits are Mostly Noise: Relying purely on process exit codes to detect crashes is a mistake. 82% of non-zero exits were benign noise, such as an app exiting gracefully after printing a --help message.
Triage of 1,033 raw crash events, showing that 82% are noise.

Key Results

  • Random Wins on Time, LLMs Win on Keystrokes: Under an equal 600-second wall-clock budget, random achieved the highest per-run crash yield (15.2%), compared to llm-guided-derived (9.0%). However, when normalized per 1,000 steps, LLM guidance is ~13× more efficient at finding bugs and uniquely uncovers deep, state-gated faults (like completing an interactive OAuth setup wizard).
  • No Model Dominates: When comparing frontier models (Claude-Opus, GPT-5-preview equivalent, Gemini-Flash, DeepSeek-V4-Pro), no single model consistently outperformed the others in bug discovery or coverage. However, their token costs and latencies varied by an order of magnitude.

Limitations and Future Work

  • Oracle Scope: The current crash oracle only catches catastrophic failures (panics, segfaults, tracebacks). It cannot detect silent logical errors, performance regressions, or visual misrenderings.
  • Environment Constraints: The benchmark excludes applications requiring live network services, complex user credentials, or specialized hardware.
  • Future Directions: The authors suggest that the optimal TUI testing tool should be a hybrid system: utilizing LLMs for semantic reasoning (generating launch arguments, filling out forms) and switching to high-throughput random fuzzing to stress-test UI transitions. They also highlight multimodal models (processing actual terminal screenshots rather than text grids) as a promising next step.

Summary updated

Abstract

Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.
BibTeX
@misc{peng2026can,
  author = {Chao Peng and Ruida Hu and Ajitha Rajan and Tegawendé F Bissyandé and Jacques Klein and Cuiyun Gao},
  title = {{Can LLMs Test Terminal User Interfaces?}},
  year = {2026},
  eprint = {2608.03743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.03743},
}
Figure