Visual Element Tagging for Multimodal Agent Grounding

33 concepts29 papers1 proposal

Modern Vision-Language Models (VLMs) are remarkably capable of reasoning about the world, yet they often struggle with the "last mile" of interaction: precisely identifying and acting upon specific objects in a visual scene. Whether navigating a smartphone interface or directing a robot to pick up a tool, models frequently fail to translate high-level intent into exact pixel coordinates.

Visual element tagging (often referred to as "Set-of-Mark" prompting) has emerged as a powerful solution to this grounding gap. By overlaying unique visual identifiers—such as numbers, bounding boxes, or colored markers—directly onto an image before the model sees it, researchers have transformed a difficult continuous coordinate prediction problem into a much simpler discrete referencing task.

The Grounding Gap: From Pixels to Actions

For an AI agent to be useful, it must bridge the gap between semantic understanding ("click the search button") and spatial execution (clicking at exactly x=450, y=320). As Jing Yu Koh et al. demonstrate, even powerful models like GPT-4V struggle with this, failing nearly 84% of the time on visual web tasks compared to a human success rate of 89%.

The core difficulty lies in the representation. Standard VLMs are trained primarily on text and images, not on precise coordinate geometry. Asking a model to output raw pixel coordinates often leads to "hallucinations," where the model suggests an interaction point that is slightly off-target or entirely irrelevant. Ruofan Liu et al. and the authors found that constraining the model's reasoning space through visual prompts significantly reduces these errors, forcing the model to choose from a set of valid, pre-identified interaction points.

Discretizing the Visual Space

The intuition behind visual element tagging is to provide the model with a shared reference system. Instead of predicting coordinates, the model simply identifies a tag.

  1. Detection: A specialized module (like an object detector or a UI parser) identifies interactable elements.
  2. Tagging: Unique numerical identifiers or markers are overlaid on these elements. Bryan Wang et al. showed that even simple numeric identifiers in a structural representation allow models to reference UI components efficiently.
  3. Reasoning: The VLM receives the "marked" image. When it decides to act, it outputs the ID of the target (e.g., "Click 5").
  4. Execution: The system maps ID 5 back to the original coordinates to perform the action.

This technique, popularized as "Set-of-Mark" (SoM) prompting by Jianwei Yang and Mu Cai, simplifies the action space. By using discrete labels rather than continuous coordinates, Xueyu Hu et al. argue that agents can operate more reliably in high-resolution environments where precision is critical.

Versatility Across Domains

While highly effective for digital interfaces, the benefits of visual tagging extend to physical and complex semantic environments:

  • Mobile and Desktop UI: In frameworks like UFO, Chaoyun Zhang et al. use numbered markers to bridge high-level reasoning with Windows OS execution. Similarly, Lawrence Jang et al. use tagging to help agents ground actions in long-context video web tasks.
  • Robotics: For physical manipulation, Alice Fasoli et al. use SoM prompting to help robots reason about cluttered bins, identifying which obstructing items (labeled by ID) must be moved to reach a target.
  • General Computer Control: Even in complex AAA games like Red Dead Redemption II, Weihao Tan et al. demonstrate that semantic tagging of interactable objects allows agents to navigate and play using only raw pixels and keyboard/mouse outputs.
  • Specialized Reasoning: In medical contexts, Qingqiu Li et al. use visual prompts like bounding boxes to guide a model's attention to specific anatomical regions in X-rays, mimicking the systematic workflow of a radiologist.

Why It Works: The Power of Visual Anchors

The success of this approach stems from its ability to provide visual anchors. As Hongliang He et al. note in their work on WebVoyager, these markers provide a discrete reference system that allows the agent to map textual commands to spatial coordinates.

By offloading the "where" to a specialized detection tool and letting the VLM focus on the "what" and "why," visual element tagging enables generalist models to outperform specialized systems. As Aditya Bharat Soni et al. argue, equipping a single LLM with these general-purpose grounding tools allows it to solve problems across software engineering, web research, and corporate work more effectively than narrow, over-optimized specialists.

Key Foundational Work

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments (2020) — contributed 1 concept elements to this direction.
Enabling Conversational Interaction with Mobile UI using Large Language Models (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visual_element_tagging_3ef2553a,
  title = {visual element tagging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-element-tagging/rdr_3ef2553a3dba4d3aa98b83260235201f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_visual_element_tagging_3ef2553a_v1,
  title = {visual element tagging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-element-tagging/rdr_3ef2553a3dba4d3aa98b83260235201f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake