ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance

Qianli Wang, Junchao Wu, Binkai Ou, Lidia S. Chao, Derek F. WongarXiv 2026 · arXiv

TL;DR

Researchers have introduced ReconPotBench, the first multi-round benchmark designed to test how well Large Language Model (LLM) honeypots can fool hackers during Linux reconnaissance. The study reveals a critical "consistency gap": while LLMs produce much more realistic and useful data than traditional honeypots, they frequently "forget" details like IP addresses and process IDs between commands, potentially tipping off an observant attacker.

What Problem Does This Paper Address?

Honeypots are decoy systems designed to lure and study attackers. Traditionally, these were rigid, template-based systems (like Cowrie) that were easy for sophisticated hackers to identify because their responses were repetitive and lacked depth. LLMs offer a solution by generating dynamic, highly realistic terminal outputs.

However, existing evaluations of LLM honeypots usually look at a single command in isolation. In a real-world attack, reconnaissance is a multi-step process. If an attacker runs ifconfig in Round 1 and sees an IP address, then runs a routing command in Round 2 that shows a different IP, the illusion is shattered. This paper addresses the lack of benchmarks that evaluate contextual consistency and hacker utility across a sequence of dependent interactions.

Methods

The authors developed a unified testing protocol and a new evaluation framework called ReconPotBench. The core of their method is a three-round dependency chain that mimics a natural attack workflow:

  1. Network Layer (Round 1): Probing IPs, routes, and interfaces.
  2. Software Layer (Round 2): Examining services, packages, and configurations.
  3. Hardware Layer (Round 3): Probing CPU, memory, or device contexts.

To evaluate the systems, they used an LLM-as-a-judge (specifically GPT-4o-mini) to score interactions across three dimensions:

  • Authenticity: Does the output look like a real Linux system?
  • Consistency: Do entities (IPs, ports, PIDs) remain the same across all three rounds?
  • Hacker Utility: Does the information provided actually help an attacker plan their next move?

The researchers compared traditional honeypots (Cowrie), various closed-source LLMs (including Sonnet-4.6 and GPT-4o variants), and open-source models (Llama-3.1, Qwen, etc.).

Datasets

The authors introduced the ReconPotBench dataset, which consists of:

  • 1,032 three-round interactions (3,096 total command-output pairs).
  • Eight reconnaissance categories, such as DNS Configuration, ARP Cache, and Process Mapping.
  • Real-system references: Ground truth data was collected from a physical Kali Linux host to ensure the benchmark reflects genuine system behavior rather than simulated templates.

The dataset is publicly available on GitHub.

Key Insights and Ideas

  • The Consistency-Realism Trade-off: Traditional honeypots like Cowrie are incredibly consistent (scoring 0.45) because they use rigid templates, but they have very low authenticity and utility. LLMs are the opposite: they are highly realistic but struggle to maintain a coherent "state" across rounds.
  • Authenticity Decay: The study found that LLM performance often drops as the interaction deepens. A model might give a perfect response to the first command but start hallucinating or losing track of the system state by the third command.
  • Implicit vs. Explicit Entities: LLMs are relatively good at keeping track of "explicit" entities like port numbers. However, they struggle with "implicit" relations, such as the logical connection between a routing table and a specific network interface.
  • Thinking Mode Isn't a Silver Bullet: Testing with models like Qwen3 in "thinking mode" showed no consistent benefit for honeypot performance; in some cases, it actually decreased consistency.

Key Results

  • Top Performers: Claude Sonnet-4.6 and Gemini-3-Flash achieved the highest authenticity scores (~0.30), significantly outperforming the traditional Cowrie honeypot (0.17).
  • Utility Gap: LLMs provided far more value to the "hacker perspective," with Sonnet-4.6 scoring 3.24/5.0 in utility compared to Cowrie’s 0.66.
  • The Consistency Failure: Despite their realism, no LLM came close to Cowrie's consistency. Most LLMs scored between 0.07 and 0.21, meaning they frequently contradicted themselves during a three-round session.
  • Category Difficulty: "Connection-to-Process Mapping" was the easiest category for maintaining consistency, while "ARP Cache & MAC Addresses" was the hardest, exposing the difficulty LLMs have with low-level networking state.

Limitations and Future Work

  • Short Interaction Depth: The benchmark is currently limited to three rounds. Real-world reconnaissance can involve dozens of commands, which would likely further degrade LLM consistency.
  • Single-Goal Focus: The chains are designed around a single reconnaissance objective. The authors note that future work should explore "cross-goal" behavior where an attacker jumps between different types of tasks.
  • Platform Scope: The current benchmark is Linux-centric. Expanding to Windows or specialized IoT environments is a suggested next step.
  • State Management: The results point to a need for better state-tracking architectures (like external databases or memory modules) rather than relying solely on the LLM's context window to maintain honeypot believability.

Summary updated

Abstract

Large language models (LLMs) are increasingly used to build interactive honeypots, yet existing evaluations mainly assess isolated command outputs and cannot determine whether a system remains believable when later commands depend on earlier results. We present ReconPotBench, a multi-round benchmark for Linux reconnaissance honeypots. It contains 1{,}032 three-round reconnaissance interactions across eight reconnaissance categories, with reference outputs collected from a real Linux environment. Under a unified testing protocol, we compare Cowrie, open-source LLMs, and closed-source LLMs along three dimensions: round-level authenticity, cross-round contextual consistency, and hacker-oriented reconnaissance value. Results show that multi-round evaluation exposes failures hidden by single-round testing, as authenticity declines with interaction depth and both open-source and closed-source LLMs struggle to maintain coherent state across rounds. LLM-based honeypots nevertheless outperform Cowrie in authenticity and reconnaissance value, while Cowrie remains more consistent because of its rigid templated behavior. Category-level analysis further shows that tasks requiring systems to preserve network relationships across rounds are harder than tasks centered on explicit entities such as ports. ReconPotBench enables reproducible evaluation of LLM honeypots across authenticity, entity consistency, and hacker-perspective utility in dependent multi-round interactions. The dataset is available at: https://github.com/qianli-wang/ReconPotBench
BibTeX
@misc{wang2026reconpotbench,
  author = {Qianli Wang and Junchao Wu and Binkai Ou and Lidia S. Chao and Derek F. Wong},
  title = {{ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance}},
  year = {2026},
  eprint = {2611.20528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20528},
}
Figure