Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination

Kim Hammar, Tansu Alpcan, Emil C. Lupu

Abstract

Incident response (IR) is a critical yet manual process in cybersecurity, often plagued by long recovery times. While Large Language Models (LLMs) show promise for automation, issues with hallucinations, privacy, and cost hinder their deployment. This paper presents a framework utilizing a lightweight 14-billion parameter LLM (DeepSeek-R1-14B) optimized through instruction fine-tuning, Retrieval-Augmented Generation (RAG), and a novel decision-theoretic lookahead planning step. Our approach reduces hallucinations and achieves up to 22% faster recovery times compared to state-of-the-art frontier models, demonstrating that specialized, smaller models can outperform general-purpose giants in domain-specific security tasks.

Introduction

Cyber incident response—the containment and recovery from attacks—remains a bottleneck in organizational security, with full recovery taking an average of 73 days. Existing playbooks are often too static to handle the dynamic nature of modern threats. While frontier LLMs like GPT-4 offer reasoning capabilities, they suffer from hallucinations, high API costs, and privacy concerns regarding sensitive logs.

Effective IR requires minimizing the "recovery time" and "operational cost" associated with an attack event. As shown in Figure 3, the goal is to reduce the area of loss and accelerate the return to normal service levels.

Conceptual framework of incident response metrics, highlighting recovery time and operational cost.

Methodology

The proposed framework addresses these challenges through a three-pillar architecture designed to make LLMs reliable for front-line security operations.

1. Instruction Fine-Tuning (Offline)

The system utilizes a DeepSeek-R1-14B model, fine-tuned on a custom dataset of 68,000 instruction-answer pairs. This process aligns the model with specific IR phases—Containment, Assessment, Preservation, Eviction, Hardening, and Restoration—and enforces Chain-of-Thought (CoT) reasoning to improve logical consistency.

2. Information Retrieval (Online RAG)

To ground the model in real-world context, the system extracts Indicators of Compromise (IoCs) from live logs and queries external threat intelligence. This ensures the model's suggestions are based on current threat data rather than just its static training weights.

3. Lookahead Planning (Online)

The most significant innovation is the decision-theoretic planning step. Instead of executing the first suggested action, the system generates $N$ candidate actions and uses the LLM to simulate their future trajectories. By selecting the action that minimizes the estimated "recovery time-to-go," the system effectively filters out hallucinations—actions that appear plausible but fail to advance the system state toward recovery.

The system architecture illustrating (a) the offline fine-tuning process and (b) the online loop for log analysis, retrieval, and response planning.

Evaluation and Results

The framework was evaluated against 25 incidents across four public security datasets (CTU-Malware-2014, CIC-IDS-2017, AIT-IDS-V2-2022, and CSLE-IDS-2024), covering attacks such as ransomware and DDoS.

Key findings include:

  • Performance: The 14B parameter model achieved up to 22% shorter recovery times than frontier models like Gemini 2.5 and OpenAI o3.
  • Reliability: The "ineffective action" rate was reduced to 8.4%, with a "failed recovery" rate of only 3.0%, outperforming much larger general-purpose models.
  • Efficiency: The lightweight nature of the model allows for local deployment on commodity hardware, preserving data privacy and reducing latency.

Discussion and Conclusion

This research demonstrates that domain specialization can allow smaller, more efficient models to surpass the performance of massive frontier models in specialized tasks like incident response. The "lookahead planning" step provides a mathematically grounded method to bound hallucination probabilities.

Future work will focus on improving the accuracy of the LLM's internal "world model" and incorporating "conformal abstention" to further enhance reliability. While currently a decision-support tool, this framework paves the way for more autonomous, high-speed cyber defense systems.

Summary updated

BibTeX
@inproceedings{hammar2026incident,
  author = {Kim Hammar and Tansu Alpcan and Emil C. Lupu},
  title = {{Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination}},
  booktitle = {NDSS},
  year = {2026},
}
Figure