No article text was recorded for this version.
metadata dependency (vision-only UI grounding)
Version 2 of 2179 papers286 conceptsfrozen 2026-10-04
Papers in this version
- Blog: IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation
- Blog: User Interface Softbots
- Blog: From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
- Blog: Cradle: Empowering Foundation Agents towards General Computer Control
- Blog: Surfer 2: The Next Generation of Cross-Platform Computer Use Agents
- Blog: Enhancing Virtual Assistant Intelligence: Precise Area Targeting for Instance-level User Intents beyond Metadata
- Blog: AndroidInTheWild: A Large-Scale Dataset For Android Device Control
- Blog: Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
- Blog: Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
- Blog: NiCro: Purely Vision-based, Non-intrusive Cross-Device and Cross-Platform GUI Testing
- Blog: HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices
- Blog: Large Language Model-Brained GUI Agents: A Survey
- Blog: CogAgent: A Visual Language Model for GUI Agents
- Blog: Computer User Interface Understanding. A New Dataset and a Learning Framework
- Blog: OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
- Blog: AssistGUI: Task-Oriented PC Graphical User Interface Automation
- Blog: OmniParser for Pure Vision Based GUI Agent
- Blog: WinSpot: GUI Grounding Benchmark with Multimodal Large Language Models
- Blog: AXNav: Replaying Accessibility Tests from Natural Language
- Blog: Image-Based Table Recognition: Data, Model, and Evaluation
- Blog: DroidRetriever: An Autonomous Navigation and Information Integration System Facilitating Mobile Sensemaking
- Blog: Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images
- Blog: Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
- Blog: GUICourse: From General Vision Language Model to Versatile GUI Agent
- Blog: ScreenQA: Large-Scale Question-Answer Pairs Over Mobile App Screenshots
- Blog: META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
- Blog: WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions
- Blog: {A New Hope}: Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation
- Blog: Avgust: Automating Usage-Based Test Generation from Videos of App Executions
- Blog: A data-driven approach for learning to control computers
- Blog: OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
- Blog: GUI Agents with Foundation Models: A Comprehensive Survey
- Blog: InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
- Blog: AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants
- Blog: Grounding Multimodal Large Language Model in GUI World
- Blog: Towards Automated Accessibility Report Generation for Mobile Apps
- Blog: SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Blog: Android in the Wild: A Large-Scale Dataset for Android Device Control
- Blog: Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
- Blog: AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
- Blog: Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery
- Blog: Owl Eyes: Spotting UI Display Issues via Visual Understanding
- Blog: Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
- Blog: K²-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control
- Blog: Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
- Blog: UFO: A UI-Focused Agent for Windows OS Interaction
- Blog: Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
- Blog: Avgust: automating usage-based test generation from videos of app executions
- Blog: AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
- Blog: TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
- Blog: VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
- Blog: Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing
- Blog: SpiritSight Agent: Advanced GUI Agent with One Look
- Blog: Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
- Blog: SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Blog: VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft
- Blog: Translating Video Recordings of Mobile App Usages into Replayable Scenarios
- Blog: A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions
- Blog: Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
- Blog: Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Showing 60 of 179 papers.
History & citation
Current · Permanent link
- Vision Based Mobile Application Testing merged into this direction (90% overlap)
- metadata dependency (structural metadata dependency) split from this direction (57% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_metadata_dependency_vision_only_ui_groun_367e84c6,
title = {metadata dependency (vision-only UI grounding)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/metadata-dependency-vision-only-ui-grounding/rdr_367e84c668434b7eb628368dc2310294},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 2, frozen text):
@misc{lacuna_metadata_dependency_vision_only_ui_groun_367e84c6_v2,
title = {metadata dependency (vision-only UI grounding)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/metadata-dependency-vision-only-ui-grounding/rdr_367e84c668434b7eb628368dc2310294/v/2},
note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}