Abstract
While AI has made significant progress in understanding mobile and web interfaces, the desktop computer environment remains a challenging "black box" due to its visual complexity and lack of structural metadata like the Document Object Model (DOM). This paper introduces DataVisualWorkflow, a novel dataset of high-resolution desktop workflow videos, and UIMTCon, a multi-task contrastive learning framework. By utilizing synthetic data augmentation and a hierarchical loss function, UIMTCon learns to identify UI states—such as active applications, specific views, and open context menus—directly from pixels. Our approach achieves a significant performance boost in UI state retrieval and classification accuracy.
Introduction
The ability of AI agents to automate complex workflows, such as transferring data between disparate desktop applications, depends on a deep understanding of the user interface (UI) state. Unlike web browsers, where structural information is readily available, many desktop applications are opaque, requiring models to rely solely on visual information. Desktop environments are characterized by overlapping windows, varying resolutions, and significant visual clutter. To address these challenges, we propose a new dataset and a learning framework designed for robust UI understanding in unconstrained desktop environments.
The DataVisualWorkflow Dataset
To facilitate research in this area, we present DataVisualWorkflow. This dataset consists of 13,894 frames extracted from 88 videos of users performing realistic tasks at 1920x1080 resolution. It covers 26 software and view classes, expanding to 78 combined classes when including contextual states like open menus or text selection.
Unlike previous datasets that often use simplified or "clean" interfaces, DataVisualWorkflow captures the typical clutter of a productive desktop environment, providing a more realistic benchmark for UI understanding models.
Methodology
Our approach, UIMTCon (UI Multi-task Contrastive Learning), addresses the scarcity of labeled data and the inherent hierarchy of UI states through two primary innovations.
Synthetic Sample Generation
Labeling fine-grained UI states like "text selected" or "context menu open" is labor-intensive. We developed a pipeline that overlays synthetic UI elements onto real desktop images. This process increased our labeled instances for these specific states by 66%, providing the model with a richer training signal without the cost of manual annotation.
UIMTCon and Split Hierarchy Loss
UI labels are naturally hierarchical: a frame might belong to the Browser software category, the Gmail view, and have a Context Menu open. UIMTCon exploits this structure using a Split Hierarchy Loss (SHL). The model employs multiple projection heads to learn representations at different levels of the hierarchy simultaneously.
This multi-task setup serves as a powerful regularizer. By forcing the backbone to learn broader categories (Software and View), the model becomes more robust to the potential "noise" or "unnaturalness" introduced by synthetic samples, leading to better generalization on real-world data.
Experimental Results
We evaluated UIMTCon against standard classification baselines and other hierarchical contrastive learning methods. Our framework demonstrated superior performance across several metrics:
- Retrieval Performance: The combination of synthetic data and the multi-task framework improved R-Precision by approximately 10 points compared to models trained on real data alone.
- Classification Accuracy: In the most challenging Software-View-Context classification task, UIMTCon achieved a Precision@1 of 78.16%, a substantial improvement over the 56.68% achieved by a standard Cross-Entropy baseline.
- Backbone Agnosticism: The framework showed consistent gains even with smaller backbones like ResNet18, indicating that the learning strategy itself is the primary driver of performance.
Conclusion
This work establishes a new benchmark for desktop UI understanding. By combining a realistic, high-resolution dataset with a hierarchical contrastive learning framework, we demonstrate that models can learn to navigate the visual complexity of desktop environments without access to underlying code. Future work will explore incorporating temporal dynamics to recognize multi-step actions and improving the model's ability to handle unseen, "out-of-distribution" software.
Summary updated