Abstract
The process of developing a mobile application typically starts with the ideation and conceptualization of its user interface. This concept is then translated into a set of mock-ups to help determine how well the user interface embodies the intended features of the app. After the creation of mock-ups developers then translate it into an app that runs in a mobile device. In this paper we propose an approach, called GUIGLE, that aims to facilitate the process of conceptualizing the user interface of an app through GUI search. GUIGLE indexes GUI images and metadata extracted using automated dynamic analysis on a large corpus of apps extracted from Google Play. To perform a search, our approach uses information from text displayed on a screen, user interface components, the app name, and screen color palettes to retrieve relevant screens given a query. Furthermore, we provide a lightweight query language that allows for intuitive search of screens. We evaluate GUIGLE with real users and found that, on average, 68.8% of returned screens were relevant to the specified query. Additionally, users found the various different features of GUIGLE useful, indicating that our search engine provides an intuitive user experience. Finally, users agree that the information presented by GUIGLE is useful in conceptualizing the design of new screens for applications.
Introduction
The "blank canvas" problem is a significant hurdle in mobile development. Before a single line of code is written, designers must conceptualize a user interface (GUI) that is both intuitive and functional. While developers often look to existing apps for inspiration, finding relevant examples is surprisingly difficult. General tools like Google Image Search lack the domain-specific metadata (like "show me screens with a progress bar and a blue color scheme") needed for deep design research. Meanwhile, existing research tools like GUIFetch require a pre-existing mock-up to function, creating a "chicken-and-egg" problem where you need a design to find a design. Guigle fills this gap by providing a lightweight, natural language-based search for the collective design patterns found in thousands of production-ready Android apps.
System Architecture
The Guigle system is built upon a three-stage pipeline designed to transform raw application packages into a searchable index of design patterns.
- Data Collection: The system utilizes an automated execution engine to "crawl" Android apps. Using a Depth-First Search (DFS) strategy, the engine interacts with apps to trigger different screens, capturing both a screenshot and an XML dump of the GUI hierarchy via the
uiautomatortool. - Data Filtering: To ensure high-quality results, the system automatically discards "junk" data, such as Android home screens, system overlays, and empty layout containers that do not provide meaningful design information.
- Web Implementation & Indexing:
- Metadata Indexing: Guigle uses Apache Lucene to index app names, component types (buttons, text fields), and on-screen text.
- Color Indexing: The system extracts the top six dominant colors from every screen, converting them into the HSL (Hue, Saturation, Lightness) color space to allow for intuitive color-based filtering.
- Query Parsing: A custom parser translates natural language queries (e.g., "red edittext pizza") into structured searches across specific fields.
Dataset and Key Insights
The researchers leveraged the ReDraw dataset, which originally contained 7,654 apps. After filtering for non-native apps and removing inaccessible entries from Google Play, the final Guigle index consists of 5,416 unique apps and approximately 12,000 individual screenshots with associated XML metadata.
Several key insights drove the development of Guigle:
- Frequency as a Proxy for Importance: The authors only index the top six most "visited" screens during automated exploration, assuming these represent the core functionality of the app.
- Structural vs. Visual Search: By indexing the XML hierarchy alongside pixels, Guigle understands the structure of a screen. It can distinguish between a blue background and a blue button, a distinction general image search engines cannot easily make.
- Closed-Source Accessibility: Unlike tools requiring source code, Guigle operates entirely on compiled APKs, allowing it to index the vast ecosystem of the Google Play Store.
Evaluation and Results
The system was evaluated by 13 professional developers with an average of four years of experience. The results demonstrated strong performance and utility:
- Precision: On average, 68.8% of the returned screens were deemed relevant to the user's specific query.
- Usability: Users reported high satisfaction with the "search-by-color" feature and the ability to view "similar screens" from the same app, which helped them understand the flow of a design language.
- Confidence: Participants gave the search results an average confidence score of 3 out of 4, indicating the results were "mostly relevant" to their design tasks.
Limitations and Future Work
While successful, Guigle has areas for improvement. Currently, color ranking identifies the existence of a color but does not prioritize screens where that color is dominant. Furthermore, the search is currently "flat"; future iterations aim to support queries based on spatial hierarchy (e.g., "a button located below an image"). Finally, the authors envision Guigle as a front-end for "screenshot-to-code" tools, allowing designers to find a screen and immediately generate a skeletal code implementation.
Summary updated