TL;DR
This paper introduces WHYPER, a framework that uses Natural Language Processing (NLP) to automatically identify sentences in mobile app descriptions that justify requested permissions. This helps bridge the gap between what users expect an app to do and what it actually requests access to, aiding in risk assessment and user understanding.
What Problem Does This Paper Address?
The proliferation of mobile apps on platforms like the Apple App Store and Google Play Store has created a vibrant ecosystem, but also a challenge in ensuring app security and privacy. While app stores have mechanisms to vet applications, it's difficult for end-users to truly understand what permissions an app requests and why. Google's approach, relying heavily on user-presented permission lists, assumes users can make informed decisions, which is often not the case. Existing work has focused on analyzing app behavior or code, but this paper tackles the crucial question: "What do users expect?" Specifically, it addresses the gap between the app's functionality and the user's understanding of why certain permissions are necessary, aiming to automate the process of finding justifications for these permissions within app descriptions.
Methods
WHYPER employs a multi-stage Natural Language Processing (NLP) pipeline.
- Preprocessor: This stage cleans and standardizes the app descriptions by identifying sentence boundaries and reducing lexical tokens. It handles tricky cases like periods in abbreviations or enumerations.
- NLP Parser: Using the Stanford Parser, this component annotates sentences with Part-of-Speech (POS) tags and, crucially, Stanford-typed dependencies. These dependencies capture grammatical relationships, providing a richer semantic structure than simple POS tags.
- Intermediate-Representation Generator: This component transforms the parsed sentences into a First-Order-Logic (FOL) representation. This structured format helps disambiguate keywords and distinguish between resources and actions.
- Semantic Engine (SE): The core of WHYPER, the SE uses "semantic graphs" of permissions. These graphs represent the resources governed by a permission (e.g., "CONTACTS" for READ_CONTACTS) and the associated actions (e.g., "read," "send"). The SE then matches the FOL representation of a sentence against these semantic graphs to determine if it describes the need for a permission.
- Analyzer: This component evaluates the output of the SE, calculating metrics like precision, recall, and F-score.
A key innovation is the use of semantic graphs derived from Android API documents, which allows WHYPER to understand the relationships between permissions, resources, and actions beyond simple keyword matching.
Datasets
The evaluation was performed on a snapshot of popular Android applications downloaded from the Google Play Store in January 2012. This included 16,001 unique applications. For the evaluation, the authors focused on applications requesting specific permissions: READ_CONTACTS, READ_CALENDAR, and RECORD_AUDIO. After filtering for applications with English descriptions, a total of 581 application descriptions, containing approximately 9,953 sentences, were analyzed. The authors manually annotated these sentences to serve as ground truth for evaluating WHYPER's performance.
Key Insights and Ideas
- Bridging the Semantic Gap: The fundamental idea is to use NLP to connect the technical details of app permissions with the natural language used by developers in descriptions, making it understandable for users.
- Beyond Keywords: Simple keyword matching is insufficient due to ambiguous terms (e.g., "contact" for a person vs. "contact me"). WHYPER's reliance on semantic graphs and typed dependencies allows for a deeper understanding of context.
- API Docs as a Knowledge Source: Leveraging Android API documentation to build semantic graphs for permissions is a novel and efficient way to capture the intended use of permissions.
- User Expectations as a Guiding Principle: The work is motivated by understanding what users expect an app to do, which is crucial for assessing the appropriateness of requested permissions.
Key Results
WHYPER achieved promising results in identifying permission sentences:
- Average Precision: 82.8%
- Average Recall: 81.5%
- Average F-Score: 82.2%
- Average Accuracy: 97.3%
Compared to keyword-based searching, WHYPER demonstrated significant improvements, particularly in precision. For example, for READ_CONTACTS, WHYPER had only 18 false positives compared to 265 for keyword search. While recall saw a slight decrease on average (1.2%), this was largely due to the RECORD_AUDIO permission, and the authors suggest this can be improved with better semantic graphs.
Limitations and Future Work
The authors acknowledge several limitations:
- Developer Deception: WHYPER assumes developers are honest in their descriptions; it cannot detect deliberate lies.
- NLP Infrastructure Limitations: Inaccuracies in the underlying NLP parsers (like the Stanford Parser) can lead to incorrect sentence boundary detection or dependency parsing, impacting WHYPER's performance.
- Modular Applications: The current framework doesn't handle descriptions that refer to other applications for functionality.
- Internet Permission: WHYPER is not expected to perform well for permissions like "Internet" due to their ubiquitous nature and lack of specific descriptive context.
Future work includes:
- Incorporating other input sources: Exploring application names, user reviews, and screenshots.
- Extending to API calls: Moving beyond just permissions to analyzing specific API calls.
- Automating semantic graph creation: Developing more robust techniques for building and refining semantic graphs, potentially by mining user comments and forums.
- Improving NLP infrastructure: Addressing limitations in parsing and sentence boundary detection.
- Developing better interfaces: Designing ways to present this information to users without causing fatigue from lengthy descriptions.
Summary updated