Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats

Ryan Pavlich, Nima Ebadi, Richard Tarbell, Billy Linares, Adrian Tan, Rachael Humphreys, Jayanta Kumar Das, Rambod Ghandiparsi, Hannah Haley, Jerris George, Rocky Slavin, Kim-Kwang Raymond Choo, Glenn Dietrich, Anthony Rios

Abstract

Traditional Natural Language Interfaces to Databases (NLIDB) focus primarily on the translation of natural language to SQL. However, in high-stakes domains like cybersecurity, simply retrieving data is insufficient. This paper introduces a novel framework and the IoT-SQL dataset, which require models to not only generate SQL queries but also perform domain-specific reasoning to classify network traffic. By employing Multi-Task Learning (MTL), the researchers demonstrate that jointly training a model for both querying and classification significantly enhances SQL generation accuracy, enabling smaller models to outperform larger, single-task counterparts.

Introduction

The proliferation of Internet of Things (IoT) devices has led to an explosion of network data, making manual threat detection increasingly difficult. While text-to-SQL models have improved significantly, they remain "data-blind"—capable of fetching rows but unable to interpret the security implications of the retrieved information. Security analysts require systems that understand the context of the data, such as distinguishing between a routine update and a malicious attack.

Furthermore, existing benchmarks like Spider often lack the temporal complexity required for forensic analysis. This research addresses these gaps by proposing a system that integrates data retrieval with automated threat classification.

Methodology and Dataset

The proposed framework operates in two stages: first, translating natural language prompts into schema-aware SQL; and second, performing binary classification on the returned network traffic to identify malicious activity.

Figure 1: The proposed workflow transitioning from natural language query to SQL execution and subsequent malicious traffic inference.

The core innovation is the use of Multi-Task Learning (MTL), fine-tuning a single model (specifically Flan-T5) to handle both tasks simultaneously. To support this, the authors created the IoT-SQL dataset, featuring 10,985 unique text-SQL pairs and nearly 240,000 rows of network traffic from real-world Zeek logs (IoT-23) and smart building sensors. The dataset includes complex temporal queries involving JOINs and HAVING clauses, reflecting the dynamic nature of IoT environments.

Results and Key Insights

The experimental results highlight the benefits of joint training. A multi-task T5-base model achieved an execution accuracy of 95.6% on the test set, surpassing a larger T5-large model (92.8%) that was trained solely on text-to-SQL. This suggests that understanding the underlying meaning of the data (classification) reinforces the model's ability to locate it (SQL generation).

Figure 5: Performance comparison of various models on the IoT-SQL validation and test sets, showing the superiority of the fine-tuned T5-base with Multi-Task Learning.

Key findings include:

  • Joint Training Synergy: Learning to classify data improves the precision of the retrieval mechanism.
  • LLM Limitations: GPT-3.5 struggled with domain-specific tabular reasoning, often failing to identify malicious patterns despite few-shot prompting.
  • Temporal Reasoning: The inclusion of over 1,000 temporal queries addresses a critical void in current NLIDB benchmarks, which often treat databases as static snapshots.

Conclusion and Future Work

This work demonstrates that integrating domain-specific reasoning into the text-to-SQL pipeline is essential for practical IoT defense. While the current model is highly effective within the IoT domain, future research will focus on generalizing these capabilities to other sectors like finance and healthcare. Additionally, the researchers plan to expand the classification task from row-level analysis to multi-row session reasoning to detect more sophisticated, long-term attack patterns.

Summary updated

BibTeX
@misc{pavlich2024beyond,
  author = {Ryan Pavlich and Nima Ebadi and Richard Tarbell and Billy Linares and Adrian Tan and Rachael Humphreys and Jayanta Kumar Das and Rambod Ghandiparsi and Hannah Haley and Jerris George and Rocky Slavin and Kim-Kwang Raymond Choo and Glenn Dietrich and Anthony Rios},
  title = {{Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats}},
  year = {2024},
  eprint = {2406.17574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17574},
}
Figure