Introduction
The Video Internet of Things (VIoT) has led to an explosion of surveillance data across smart cities. While general-purpose foundation models like Segment Anything (SAM) are powerful, they often lack the domain-specific precision required for tasks like biometric identification or license plate recognition. Conversely, specialized "expert" models exist but lack a unified interface for complex reasoning.
VIoTGPT addresses this by transforming Large Language Models (LLMs) into intelligent agents capable of orchestrating these specialized vision tools. By fine-tuning models on a custom "VIoT-Tool" dataset using ReAct-style instructions, the system can autonomously schedule fragmented expert models to answer multi-step queries about surveillance footage.
System Architecture and Methodology
VIoTGPT treats the LLM as a central "brain" connected to a "Knowledge Base" of video data and a "Tool Set" of specialized vision algorithms.
The framework relies on three core technical pillars:
- Hierarchical Taxonomy: Vision tools are organized into three primary categories: human-centric, vehicle-centric, and event-related analysis.
- ReAct Instruction Tuning: The model is trained using the Reason + Act (ReAct) method, fostering a "Thought-Action-Observation" loop. This allows the LLM to "think" about the user's query, select the appropriate tool ("Action"), process the tool's output ("Observation"), and iterate until a final answer is reached.
- Parameter-Efficient Fine-Tuning: Using Low-Rank Adaptation (LoRA), the researchers fine-tuned Llama-7B and Vicuna-7B models. This approach enables high-performance tool scheduling without the prohibitive costs of full-scale model training.
The VIoT-Tool Dataset
A major contribution of this work is the VIoT-Tool dataset, designed to bridge the gap between natural language queries and vision tool execution.
- Scale: The training set includes 200,000 instruction-response pairs, totaling 2.79 billion tokens.
- Tool Coverage: It integrates 11 representative vision models, including Face Recognition, Person Re-ID, Gait Recognition, and Anomaly Detection.
- Diversity: Data is sourced from 125 city-level locations, combining public datasets with real-world surveillance footage.
Key Insights
The research highlights that vision tools in a VIoT context are uniquely interrelated. Unlike a calculator tool, the output of a "person detection" model often serves as the direct input for a "gait analysis" model. VIoTGPT proves that specialized, lightweight models often outperform general-purpose foundation models for these fine-grained biometric tasks. Furthermore, smaller 7B models, when properly fine-tuned, can surpass larger models like ChatGPT in domain-specific tool scheduling by avoiding formatting errors and hallucinations.
Performance and Results
VIoTGPT demonstrated significant improvements in decision-making and reasoning:
- Decision Accuracy: The Llama-based VIoTGPT achieved 87.68% accuracy in tool selection, whereas the base model failed to follow the required execution format entirely.
- Complex Reasoning: For multi-step queries, the Vicuna-based version achieved a 52% "Whole Response" accuracy, significantly outperforming zero-shot ChatGPT, which struggled with tool chaining and input consistency.
Conclusion and Future Directions
While VIoTGPT marks a significant step toward intelligent VIoT, challenges remain. A "semantic gap" exists between diverse human phrasing and structured tool inputs. Additionally, while 7B models are efficient, more complex multi-step reasoning may require larger base models to maintain context over long conversations. Future work will focus on closing the "last mile" problem—ensuring that even after successful tool invocation, the final natural language synthesis is as robust as the technical execution.
Summary updated