Abstract: The growing mobile application development market brings an urgent need for tools capable of evaluating and comparing similarities between screens of different applications. This article proposes a comparative analysis of methods for similarity detection on Android screens, integrating Information Retrieval (IR), embedding models (BGE-M3, Snowflake-Arctic-Embed2, Nomic-Embed-Text) and Vision Language Models (VLM) such as qwen2.5vl:7b and gemma-3:27b. The study addresses challenges like design redundancies and UI test inconsistencies by using combined pipelines: IR, embeddings, and VLM for screen identification. The results demonstrate the strengths and limitations of each approach, providing insights into their applicability in mobile UI similarity detection.
Introduction
As the mobile application market continues to expand, developers and QA engineers face a significant challenge: automatically determining if two screens are functionally or visually identical. This capability is essential for detecting design redundancies, ensuring UI test consistency across different versions, and improving accessibility.
Mobile screens present a unique challenge because they are a hybrid of visual layout and structured hierarchical data (XML). Traditional pixel-matching often fails due to minor layout shifts or dynamic content, while simple text-matching may ignore the structural context. This study evaluates which combination of Information Retrieval (IR), text embeddings, and Vision Language Models (VLMs) provides the most robust "fingerprint" for identifying Android screens.
Methodology
The researchers developed a multi-stage solution pipeline to process and compare Android screens, focusing on the Android Settings application as a controlled environment. The pipeline integrates both structural data and visual interpretation.
Data Extraction and Filtering
Using the Android Debug Bridge (ADB), the system captures both the XML hierarchy dump and a PNG screenshot of the device screen. To reduce noise, the XML is filtered to remove low-signal attributes (such as whether a node is "checkable"), retaining high-signal technical identifiers like resource-id, class, and text.
Visual Description and Vectorization
To capture the "human-like" understanding of the UI, screenshots are processed by VLMs—specifically Qwen2.5-VL-7B and Gemma-3-27B—to generate natural language descriptions. These descriptions are then combined with the filtered XML data.
The resulting dataset is vectorized using five distinct techniques:
- TF-IDF: A classic statistical method for determining word importance.
- Embedding Models: Modern neural models including BGE-M3, Snowflake-Arctic-Embed2, and Nomic-Embed-Text.
To simulate real-world variability, the researchers introduced 10% "noise" (random deletions, insertions, or substitutions) into the text data before calculating Cosine Similarity.
Results and Discussion
The primary metric for evaluation was the M1/M2 ratio, where M1 represents the similarity score of a screen compared to itself (assertiveness) and M2 represents the similarity score compared to a different screen (differentiation). A higher ratio indicates a model is more effective at distinguishing between unique screens.
The "Exact Match" Advantage
Surprisingly, the results showed that the "old-school" TF-IDF method outperformed modern embedding models. When using the "XML + description" data type, TF-IDF achieved the highest M1/M2 ratio of 5.50. While embedding models are superior at capturing semantic "vibes," they often smooth out the precise technical identifiers—like specific resource IDs—that serve as unique fingerprints in a UI. TF-IDF weights these unique technical strings more heavily, making it more sensitive to the specific structural markers of a screen.
VLM Integration and Robustness
Integrating VLM-generated descriptions significantly enhanced performance. Among embedding options, Snowflake-Arctic-Embed2 paired with Qwen2.5-VL descriptions emerged as the strongest modern alternative, with a ratio of 1.63. Furthermore, TF-IDF demonstrated remarkable consistency, maintaining high "Min" similarity scores (0.96–0.98) even in the presence of noise, ensuring that screens are almost always recognized correctly upon re-evaluation.
Conclusion
This comparative analysis reveals that for technical UI similarity detection, statistical methods like TF-IDF remain highly effective due to their sensitivity to unique technical identifiers. However, the addition of VLM-generated descriptions provides a valuable semantic layer that raw XML lacks.
Future research will focus on developing hybrid models that explicitly weight technical identifiers differently than semantic content. Additionally, expanding the dataset beyond the structured environment of the Android Settings app to more visually diverse applications (like games or social media) will be necessary to determine if the TF-IDF advantage persists in less structured UI environments.
Summary updated