Appstract: on-the-fly app content semantics with better privacy

Earlence Fernandes, Oriana Riva, Suman NathMobiCom 2016 · DOI

TL;DR

This paper introduces Appstract, a novel system that extracts semantic information from mobile app content directly on the device, bypassing the need to send sensitive user data to the cloud. By automatically generating "entity templates" offline, Appstract enables real-time, privacy-preserving understanding of app content for richer user experiences.

What Problem Does This Paper Address?

Modern mobile applications are increasingly offering context-aware features, exemplified by services like Google Now on Tap and Bing Snapp. These services analyze the content displayed on a user's screen to understand their intent and provide relevant information or actions. However, this process typically involves sending screen content to cloud servers, raising significant privacy concerns, especially when sensitive data like financial or medical information is present. Existing solutions, like disabling these features for entire apps, are too coarse-grained and limit functionality. Appstract aims to solve this by enabling on-device semantic understanding, preserving user privacy without sacrificing the benefits of context-aware features.

Methods

Appstract employs a two-phase approach:

  • Offline, User-Agnostic Phase (Cloud-based): This phase focuses on automatically generating "entity templates" for apps. It involves:
  • App Crawling: Using UI automation tools like Monkey to explore app pages and collect UI structure and content.
  • Entity Type Extraction: Leveraging knowledge bases (KBs) and a combination of features (boolean match, TF-IDF similarity, text length) to identify potential entity types within UI elements.
  • Structural Analysis: Analyzing UI tree structures to improve entity type accuracy and discover relationships between entities. This includes cross-page similarity (combining information from similar pages) and in-page pattern analysis (identifying recurring entity sequences within a page).
  • Relationship Identification: Clustering nodes based on their spatial locality in the UI tree to identify related entities.
  • The output of this phase is an entity template for each app page class, essentially an annotated UI tree.
  • Online, On-Device Phase: This phase uses the pre-generated entity templates to extract semantics from the user's current app content in real-time.
  • Template Matching: The system captures the current app screen content and matches it against the corresponding entity template.
  • Semantic Annotation: This matching process assigns entity types and relationships to the on-screen content.

This on-device approach ensures that sensitive data never leaves the user's device.

Datasets

The paper does not explicitly mention a single, large, publicly available dataset for training or evaluation. Instead, it describes the creation and use of datasets derived from:

  • App Crawling: Using UI automation tools (Monkey, PUMA) to crawl a corpus of mobile apps (specifically mentioning 100+ food, dining, and music apps, and a dataset of 14k top Android apps for page class analysis).
  • Knowledge Bases (KBs): The system relies on KBs relevant to app categories (e.g., song titles, artist names for music apps) for entity type recognition.
  • Ground Truth Templates: For evaluating template accuracy, the authors manually compiled ground truth templates by observing hundreds of UI elements from several Monkey runs on a subset of 17 WP apps.

Key Insights and Ideas

  • Mobile App Structure is Key: The paper's core insight is that the inherent structural properties of mobile apps (singleton entities in UI elements, few stable page classes) make on-device semantic analysis feasible. This contrasts with the more unstructured nature of web pages.
  • Offline Template Generation is Scalable: Automating the generation of entity templates offline, without user involvement, is crucial for scaling the system to a large number of apps and categories.
  • Structural Analysis Boosts Accuracy: Relying solely on KB matching for entity extraction is insufficient for mobile apps. Incorporating UI tree structure analysis significantly improves accuracy by leveraging patterns and relationships within the app's layout.
  • Privacy-Preserving by Design: The on-device processing model fundamentally addresses privacy concerns by keeping user data local, offering a significant advantage over cloud-based solutions.
  • Fine-Grained Control: Entity templates provide a mechanism for fine-grained control over what data can be shared, even if entity linking is required for some services.

Key Results

  • Template Generation Speed: Generating entity templates for apps takes an average of 10.3 seconds per app.
  • Template Accuracy: The Appstract approach (KB-only + structural analysis) achieved an average precision of 78%, recall of 87%, and F1 score of 81% for entity type extraction, significantly outperforming the KB-only baseline (57% F1) and StanfordNER (41% F1) on mobile app data.
  • Entity Relationship Accuracy: The clustering algorithm for identifying entity relationships achieved 100% precision and 91.5% recall.
  • On-Device Performance:
    • Notification Delay: Appstract incurs a notification delay of 126-150 ms (Pandora) and 242-350 ms (Urbanspoon) for 100 subscriptions, which is considered timely for typical UI event rates.
  • CPU Overhead: On Android, Appstract introduces a CPU overhead of only 1.7% on average, increasing to 4.7% with 100 subscriptions.
  • Energy Consumption: Appstract's energy consumption is comparable to or less than cloud-based approaches, with a Nexus 4 retaining 75% battery after 2 hours of use compared to 74% for a cloud-based approach.
  • Storage Overhead: Monthly storage overhead for 10 monitored apps is estimated to be around 1.2 MB for active users and 72 kB for occasional users.

Limitations and Future Work

  • Image-Based Entities: Appstract currently focuses on structured text and does not recognize entities within images. The authors suggest that a trained model for image entity recognition could be developed in the cloud and sent to the device, but this requires a large labeled training dataset.
  • Incomplete App Crawling: The accuracy of template generation relies on the coverage of UI automation tools like Monkey. While these tools achieve good page class coverage, instance coverage can vary.
  • Data Ownership and Access Control: The paper acknowledges that the "all-or-nothing" model for app access to semantic data is insufficient. They suggest that a more granular permission system, similar to current OS resource access controls, is needed. They also raise the question of data ownership, suggesting that users should own their app usage data.
  • App Scope: While the current work focuses on food, dining, and music apps, the authors believe Appstract is applicable to other categories with structured content like communication, travel, shopping, and entertainment. Games and productivity apps are considered less interesting for entity extraction.
  • Future Work: The authors suggest extending Appstract to handle image-based entities and developing more sophisticated privacy and access control mechanisms. They also propose that cloud-based services like Google Now on Tap could leverage Appstract's templates to improve their privacy, accuracy, and efficiency.

Summary updated

BibTeX
@inproceedings{fernandes2016appstract,
  author = {Earlence Fernandes and Oriana Riva and Suman Nath},
  title = {{Appstract: on-the-fly app content semantics with better privacy}},
  booktitle = {MobiCom},
  year = {2016},
  doi = {10.1145/2973750.2973770},
}
Figure