TL;DR
Terminal User Interfaces (TUIs) are booming in developer tools but remain severely under-tested due to a lack of dedicated automation methods. This paper introduces the first open, multi-language TUI testing benchmark and evaluates frontier LLMs against random exploration. The surprising verdict: while LLMs are vastly more efficient per interaction, simple random testing often wins under fixed time budgets due to its massive throughput advantage.
What Problem Does This Paper Address?
Terminal User Interfaces (TUIs)—think of rich, text-based applications like system monitors (bottom), git clients, or LLM coding agents—occupy a strange middle ground between Command-Line Interfaces (CLIs) and Graphical User Interfaces (GUIs).
As shown in the paper's comparison, they defy existing testing paradigms:
- Unlike CLIs, which are stateless input/output transducers, TUIs maintain complex, screen-oriented states over time.
- Unlike GUIs, TUIs do not expose a structured accessibility tree, DOM, or addressable widget IDs. The entire observable state is just a two-dimensional grid of raw characters and style attributes.
Because no dedicated testing methodology exists, TUI testing in the wild is abysmal. The authors surveyed 197 real-world TUI applications and found that only 12% of test code exercises the interface layer at all. Worse, nearly half (45%) of those UI tests are completely static—they render a single frame and check it, never sending a single keystroke or mouse event.
Methods
To bridge this gap, the authors built a complete automated testing ecosystem for TUIs consisting of two core tools:
tuicov: A language-agnostic coverage toolkit. It statically detects widget construction sites across different programming languages and injects lightweight runtime probes to measure "widget coverage" alongside native line coverage. Crucially, it is designed to flush coverage data even when an interactive application is forcefully terminated.tuibot: An automated testing framework that drives TUIs headlessly via a pseudo-terminal (PTY).
Using tuibot, they evaluated four exploration strategies:
random: Model-free random keystrokes (the baseline).llm-guided: An LLM observes the text-grid screen at each step and selects the next action.llm-guided-derived: The LLM-guided explorer is seeded with automatically derived command-line arguments and input fixtures.llm-generated: An LLM reads the source code offline to generate natural-language test scenarios that steer the explorer.
The authors also developed a content-aware crash oracle. Instead of relying on non-zero exit codes (which are highly noisy), it parses the final rendered screen buffer for explicit tracebacks, panics, or fatal signals.
Datasets
The authors introduced a new, open benchmark derived from awesome-tuis. They curated 197 headlessly runnable applications spanning the four dominant TUI frameworks and languages:
ratatui(Rust)bubbletea(Go)textual(Python)ink(TypeScript)
Each application is packaged as an instrumented Docker image, creating a standardized testbed for cross-language TUI evaluation.
Key Insights and Ideas
- The Throughput vs. Efficiency Trade-off: LLM-guided testing is incredibly smart but slow. Because of API round-trip latencies, LLMs only execute a dozen or so steps within a 10-minute window. Random testing, by contrast, can fire off hundreds of keystrokes.
- Launch Inputs are the Ultimate Bottleneck: Many TUIs crash or exit immediately if they aren't provided with valid files, arguments, or mock environments. Automatically deriving these launch inputs (
llm-guided-derived) yielded the largest practical boost in code coverage, proving more critical than smart step-by-step navigation. - Coverage Decouples from Fault-Finding: In traditional software testing, higher code coverage usually means more bugs found. In interactive TUIs, this relationship breaks down due to a session-truncation confound: a successful crash immediately terminates the test run, capping the coverage it can achieve and penalizing effective bug-hunters.
- Raw Exits are Mostly Noise: Relying purely on process exit codes to detect crashes is a mistake. 82% of non-zero exits were benign noise, such as an app exiting gracefully after printing a
--helpmessage.
Key Results
- Random Wins on Time, LLMs Win on Keystrokes: Under an equal 600-second wall-clock budget,
randomachieved the highest per-run crash yield (15.2%), compared tollm-guided-derived(9.0%). However, when normalized per 1,000 steps, LLM guidance is ~13× more efficient at finding bugs and uniquely uncovers deep, state-gated faults (like completing an interactive OAuth setup wizard). - No Model Dominates: When comparing frontier models (Claude-Opus, GPT-5-preview equivalent, Gemini-Flash, DeepSeek-V4-Pro), no single model consistently outperformed the others in bug discovery or coverage. However, their token costs and latencies varied by an order of magnitude.
Limitations and Future Work
- Oracle Scope: The current crash oracle only catches catastrophic failures (panics, segfaults, tracebacks). It cannot detect silent logical errors, performance regressions, or visual misrenderings.
- Environment Constraints: The benchmark excludes applications requiring live network services, complex user credentials, or specialized hardware.
- Future Directions: The authors suggest that the optimal TUI testing tool should be a hybrid system: utilizing LLMs for semantic reasoning (generating launch arguments, filling out forms) and switching to high-throughput random fuzzing to stress-test UI transitions. They also highlight multimodal models (processing actual terminal screenshots rather than text grids) as a promising next step.
Summary updated