TL;DR
ValQID is a new framework that identifies networked devices (like routers and cameras) by intelligently analyzing their web interfaces. By using a cost-aware policy to select only the most informative "groups" of semantic questions, it achieves state-of-the-art accuracy—especially for rare device brands—while running up to 8.5x faster than general-purpose language models.
What Problem Does This Paper Address?
Identifying the specific vendor and type of a device (e.g., a "Hikvision NVR" or a "Cisco Router") just by looking at its web management page is a cornerstone of network security and asset management. However, this is harder than it sounds for two main reasons:
- Dispersed Cues: Identity information is rarely in one place. You might find a "Save" button in one corner and a "Channel 01" label in another. To understand that this is a video recorder, a system needs to link these objects together and understand their relationship.
- The Efficiency Paradox: While Large Language Models (LLMs) are great at understanding these semantic cues, they are computationally expensive. Running a full LLM analysis on every single web response during a large-scale network scan is prohibitively slow and costly.
Traditional methods either rely on rigid "fingerprints" that break when a device's software updates (see Figure 1a) or use expensive learning-based models that treat every page with the same heavy-handed analysis (see Figure 1b).
Methods
The authors introduce ValQID, which treats device identification as a sequential decision-making problem. Instead of analyzing everything, it asks: "Which questions will give me the most 'net value' for my computational buck?"
Key components include:
- Relation Graphs: ValQID parses a web page into a graph where nodes are objects (text, forms, buttons) and edges represent their structural relationships. This provides the necessary context for semantic analysis.
- DSEJEV (Device Semantic Evaluation JEV): Instead of a standard "chatty" LLM, the authors developed a non-generative evaluator. It takes a question and context and directly outputs a probability distribution over predefined answers. This avoids the slow process of generating text and parsing it.
- Complementary Question Groups: The system organizes questions into groups that are more informative when answered together (e.g., asking about a form's purpose and its submission button's function simultaneously).
- Net Value Estimation: A policy model weighs the expected "identification benefit" (how much closer we get to the right answer) against the "execution cost." It selects the best group to execute next and knows when to stop once additional analysis no longer provides enough value.
Datasets
The researchers evaluated ValQID on two primary benchmarks:
- DevTag: A large public dataset for IoT device fingerprinting containing over 75,000 unique web responses covering 80 vendors and 20 device types.
- BLMProbe: A dataset of protocol responses collected from active internet probing, featuring 67 vendor classes.
Key Insights and Ideas
- Context is King: A single text fragment like "Admin" is ambiguous. By using a relation graph, ValQID understands that "Admin" inside a "Login Form" linked to a "Firmware Update" page is a much stronger signal for specific device types.
- The Power of "No": One of ValQID's biggest strengths is knowing when not to compute. By estimating the net value, it can skip semantic analysis for pages that are either very obvious (explicit brand names) or completely irrelevant (generic error pages).
- Long-Tail Success: Most identification models struggle with "long-tail" vendors—brands that appear rarely in the training data. ValQID excels here because its semantic questions focus on functional cues (what the device does) rather than just textual patterns (what the device says).
Key Results
ValQID significantly outperforms existing state-of-the-art (SOTA) methods like BLMProbe and RoBERTa-based classifiers:
- Accuracy Gains: It achieved relative gains of up to 47.1% in joint macro-F1 (correctly identifying both vendor and type).
- Long-Tail Performance: For rare vendors, it saw a massive improvement of up to 82.1% in macro-F1 compared to the strongest baseline (see Figure 3).
- Efficiency: DSEJEV provided speedups of 3.27× to 8.58× over general-purpose LLMs like Qwen3.5 or DeepSeek (see Figure 4).
- Query Reduction: The value-based selection policy reduced the number of semantic queries needed by over 53% while maintaining high accuracy.
Limitations and Future Work
While highly effective, ValQID currently relies on a fixed library of question templates and relation types discovered during an offline screening phase. If a completely new type of web interface architecture emerges, the current library might need manual or semi-automated updates.
The authors suggest that future work could involve dynamically adapting question groups and their value estimates in real-time as firmware and web interfaces evolve, further reducing the need for human intervention in the "cat-and-mouse" game of device identification.
Summary updated