Abstract
The proliferation of Video Internet of Things (VIoT) sensors has created a massive influx of data that traditional vision models struggle to process in a unified, context-aware manner. We present VIoTGPT, a framework that transforms Large Language Models (LLMs) into intelligent agents capable of scheduling specialized vision tools. By employing ReAct instruction tuning on a curated dataset of 200,000 instruction pairs, VIoTGPT learns to autonomously decompose complex surveillance queries into executable sequences of vision algorithms. Our results demonstrate that fine-tuned open-source models can significantly outperform general-purpose LLMs like ChatGPT in multi-step tool-chaining tasks, achieving high decision accuracy and robust tool selection.
Introduction
The Video Internet of Things (VIoT) generates staggering amounts of data from ubiquitous sensors. While general-purpose "foundation models" have made strides in computer vision, they often lack the domain-specific expertise required for high-stakes surveillance, such as gait recognition or precise license plate identification. Furthermore, real-world queries are often complex and interrelated, requiring the output of one tool to inform the use of another. Existing LLMs frequently struggle with these multi-step reasoning tasks, often "hallucinating" tool names or failing to maintain context across a chain of operations.
Methodology: The VIoTGPT Framework
VIoTGPT positions the LLM as an "intelligent agent" that bridges the gap between human users, a video knowledge base, and a suite of specialized vision tools. The core of this approach is ReAct Instruction Tuning, which trains the model to follow a "Reason + Act" loop.
ReAct Instruction Tuning
Instead of providing a direct answer, the model is trained to generate a structured sequence:
- Thought: The model articulates the reasoning behind selecting a specific tool.
- Action: It selects a tool from a predefined set (e.g., Face Recognition, Anomaly Detection).
- Action Input: It identifies the necessary parameters, such as specific video files or geographic locations.
- Observation: It processes the tool's output and determines if further action is required or if a final answer can be formulated.
To ensure computational efficiency, the researchers utilized LoRA (Low-Rank Adaptation). This allowed for the fine-tuning of 7-billion-parameter models, such as Llama and Vicuna, on consumer-grade hardware like NVIDIA RTX 4090s.
Experimental Setup and Datasets
The backbone of VIoTGPT is the VIoT-Tool dataset, a meticulously crafted collection of 200,000 instruction pairs totaling 2.79 billion tokens. The dataset covers 11 representative vision tools categorized into:
- Human-centric: Face recognition, Person Re-ID, Gait recognition, and Crowd counting.
- Vehicle-centric: Vehicle Re-ID and License plate recognition.
- Event-related: Fire/smoke detection, Anomaly detection, and Action recognition.
The knowledge base simulates a real-world distributed sensor network using a collection of videos associated with 125 different city-level names.
Key Results
The evaluation of VIoTGPT reveals significant improvements over base models and general-purpose LLMs:
- Decision Accuracy: VIoTGPT (based on Vicuna) achieved a 94.13% accuracy in determining when to invoke a tool, whereas the base Llama model failed almost entirely.
- Superiority in Interrelated Tasks: In complex scenarios requiring multiple tools, VIoTGPT (Vicuna) reached a "Whole Response" accuracy of 52.00%. In contrast, ChatGPT (few-shot) struggled with hallucinations and redundant tool calls, scoring only 28.25%.
- Robust Tool Selection: The model demonstrated high proficiency in distinguishing between semantically similar tools, such as differentiating between License Plate Recognition and Vehicle Re-identification based on the user's specific intent.
Conclusion and Future Work
VIoTGPT demonstrates that fine-tuning smaller, open-source LLMs on specialized instruction sets can create highly reliable schedulers for complex IoT environments. While the model excels at tool invocation, challenges remain regarding the generalization of diverse human queries and occasional "silence" where the model fails to provide final natural language feedback. Future research will focus on scaling the framework with more powerful LLMs and expanding the toolset to incorporate a wider array of environmental sensors.
Summary updated