Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats

Ryan Pavlich, Nima Ebadi, Richard Tarbell, Billy Linares, Adrian Eng-Choon Tan, Rachael Humphreys, Jayanta Das, Rambod Ghandiparsi, Hannah Haley, Jerris George, Rocky Slavin, Kim‐Kwang Raymond Choo, Glenn Dietrich, Anthony Rios

Abstract: Ryan Pavlich, Nima Ebadi, Richard Tarbell, Billy Linares, Adrian Tan, Rachael Humphreys, Jayanta Das, Rambod Ghandiparsi, Hannah Haley, Jerris George, Rocky Slavin, Kim-Kwang Raymond Choo, Glenn Dietrich, Anthony Rios. Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025).

2025.

Introduction

As Internet-of-Things (IoT) devices become ubiquitous, the volume of network traffic they generate presents a significant challenge for security analysts. Traditional Natural Language Interfaces to Databases (NLIDB) have focused primarily on the "text-to-SQL" task—translating human questions into database queries. However, in the high-stakes domain of cybersecurity, simply retrieving data is insufficient. An analyst needs to not only fetch network logs but also determine if those logs indicate a threat, such as a DDoS attack or botnet activity.

This paper addresses the gap between data retrieval and data reasoning. Existing benchmarks like Spider often lack the domain-specific complexity and temporal reasoning required for cybersecurity. To bridge this gap, the researchers introduce a novel framework and dataset that requires models to both query and reason about IoT network traffic in a single, cohesive workflow.

The IoT-SQL Framework

The core contribution of this work is a framework that moves beyond simple query generation to include malicious traffic inference. When a user asks a question about network activity, the system generates the appropriate SQL, executes it, and then analyzes the returned results to identify potential threats.

The IoT-SQL framework: Translating natural language queries into SQL and performing subsequent malicious traffic inference.

The IoT-SQL Dataset

To support this research, the authors developed IoT-SQL, a specialized dataset containing 10,985 unique text-SQL pairs and 239,398 rows of network traffic activity. This dataset is unique in several ways:

  • Diverse Sources: It combines real-world malware traffic logs from IoT-23 with Smart Building Sensor Data (e.g., temperature, humidity, CO2 levels).
  • Temporal Complexity: It includes over 1,000 temporal queries that require the model to handle complex date and time logic.
  • Threat Annotation: The data is annotated for 10 distinct types of cyberattacks, including DDoS, Command and Control (C&C), and HeartBeat attacks.
  • Curation: The dataset was built using a hybrid approach of programmatic templates and manual curation to ensure linguistic diversity and technical accuracy.

Methodology: Multi-Task Learning

The researchers employed a Multi-Task Learning (MTL) approach, fine-tuning Flan-T5 (Base and Large) to handle two distinct tasks simultaneously. The first task is Text-to-SQL, where the model generates SQL from a natural language question and a database schema. The second is Malicious Traffic Detection, where the model classifies serialized rows of network traffic as malicious or benign.

To feed tabular data into a text-based transformer, the authors used a serialization technique that utilizes spaces as delimiters and includes schema headers to provide necessary context for the model's reasoning.

Results and Performance

The experimental results highlight a significant performance gap between general-purpose Large Language Models (LLMs) and fine-tuned domain-specific models. While GPT-3.5 is highly capable in general tasks, it struggled with the specific reasoning required for IoT threat detection.

Performance comparison between prompt-based LLMs and fine-tuned multi-task models for SQL generation and malicious traffic detection.

Key findings include:

  • SQL Accuracy: The fine-tuned T5-Large achieved 92.8% execution accuracy. Interestingly, the T5-Base model using multi-task training outperformed its larger counterpart, reaching 95.6% execution accuracy.
  • Threat Detection: The T5-Large model led the field in identifying malicious traffic with an F1-score of 0.802, significantly higher than traditional machine learning baselines like Random Forest (0.710) and SVM (0.704).
  • LLM Limitations: GPT-3.5 achieved an F1-score of only 0.54 for malware detection, which is only marginally better than a random guess, underscoring the difficulty LLMs face with tabular reasoning without fine-tuning.

Discussion and Insights

One of the most compelling insights from the study is the "Joint Training Bonus." Training a model to detect malware actually improved its ability to write SQL queries. The authors hypothesize that by forcing the model to classify data from specific columns (such as orig_bytes), the model develops a deeper "schema grounding"—a better understanding of what the data represents, which reduces errors in query generation.

However, a trade-off was observed: while joint training helped SQL performance, it slightly decreased the accuracy of malicious traffic detection compared to models trained solely on classification. Furthermore, current models still struggle with "multi-row reasoning," such as detecting slow-rate attacks that occur over extended periods rather than in single packets.

Conclusion

This research demonstrates that multi-task learning is a powerful approach for building intelligent cybersecurity interfaces. By combining data retrieval with domain-specific reasoning, the IoT-SQL framework provides a path toward more autonomous and capable IoT defense systems. Future work will focus on expanding these models to handle multi-row session context and reducing the computational costs associated with high-scale model deployment.

Summary updated

BibTeX
@misc{pavlich2024beyond,
  author = {Ryan Pavlich and Nima Ebadi and Richard Tarbell and Billy Linares and Adrian Eng-Choon Tan and Rachael Humphreys and Jayanta Das and Rambod Ghandiparsi and Hannah Haley and Jerris George and Rocky Slavin and Kim‐Kwang Raymond Choo and Glenn Dietrich and Anthony Rios},
  title = {{Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats}},
  year = {2024},
  eprint = {2406.17574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17574},
}
Figure