Abstract
Application markets streamline the end-users' task of finding and installing applications. They also form an immediate communication channel between app developers and their end-users in form of app reviews, which allow users to provide developers feedback on their apps. However, it is unclear to which extent users employ this channel to point out their security and privacy concerns about apps, about which aspects of apps users express concerns, and how developers react to such security- and privacy-related reviews. In this paper, we present the first study of the relationship between end-user reviews and security- & privacy-related changes in apps. Using natural language processing on 4.5M user reviews for the top 2,583 apps in Google Play, we identified 5,527 security and privacy relevant reviews (SPR). For each app version mentioned in the SPR, we use static code analysis to extract permission-protected features mentioned in the reviews. We successfully mapped SPRs to privacy-related changes in app updates in 60.77% of all cases. Using exploratory data analysis and regression analysis we are able to show that preceding SPR are a significant factor for predicting privacy-related app updates, indicating that user reviews in fact lead to privacy improvements of apps. Our results further show that apps that adopt runtime permissions receive a significantly higher number of SPR, showing that runtime permissions put privacy-jeopardizing actions better into users' minds. Further, we can attribute about half of all privacy-relevant app changes exclusively to third-party library code. This hints at larger problems for app developers to adhere to users' privacy expectations and markets' privacy regulations. Our results make a call for action to make app behavior more transparent to users in order to leverage their reviews in creating incentives for developers to adhere to security and privacy best practices, while our results call at the same time for better tools to support app developers in this endeavor.
Introduction
Most mobile security research focuses on the "user side"—helping people choose safer apps or manage complex permission settings. However, these are often short-term fixes that don't address the root cause: developers releasing apps with poor privacy practices. This study investigates whether the public feedback loop of app stores actually incentivizes developers to improve. Specifically, we ask: Do users actually complain about security in reviews? Do developers listen? And when they do update their apps, are they fixing their own code or the third-party libraries they've bundled in?
Methodology
To connect user feedback to code-level changes, we developed a multi-stage pipeline:
- Mining & Crawling: We collected the entire version history and review logs for 2,583 popular Android apps (each with 50M+ downloads), totaling 4.5 million reviews and 62,838 distinct app versions.
- NLP Classification: Using a Support Vector Machine (SVM) and n-gram features, we built a classifier to distinguish Security/Privacy Relevant reviews (SPR) from general noise. This model achieved a high mean AUC of 0.93, ensuring the reliability of our identified SPR dataset.
- Static Analysis: We performed static analysis on app binaries using tools like
axplorerto map API calls to permissions andLibScoutto identify third-party libraries. - S&P Mapping: We linked reviews to specific app versions based on release dates, tracking whether an SPR was followed by a Security/Privacy Update (SPU) in a subsequent version.
Key Results and Insights
The Impact of User Feedback
Our analysis shows that user reviews are a powerful driver for mobile security. In 60.77% of cases, a security-related review could be mapped to a subsequent privacy-friendly code change. Furthermore, the "SPR ratio" is a massive predictor of updates: an app version with a high ratio of security complaints is 13.04 times more likely to receive a security update than one without.
User Concerns and Runtime Permissions
Users are highly sensitive to specific types of data access. The most complained-about permissions were Storage, Contacts, and Location. Interestingly, the introduction of runtime permissions in Android 6.0 acted as a catalyst; apps using runtime permissions receive 1.46 times more SPRs, as these prompts put privacy-jeopardizing actions "top of mind" for users.
The Library Transparency Gap
One of the most striking findings is that 48.8% of privacy-relevant changes were attributed exclusively to third-party library code (such as ad networks). Developers are often held accountable by users for behavior they did not write themselves, highlighting a significant transparency gap in the library ecosystem.
Conclusion
Developers are responsive to the reputational and financial incentives created by user feedback. When combining actual code fixes with developer explanations in review replies, the "Developer Response Rate" to security concerns reached 75.68%. However, the high prevalence of library-driven privacy issues suggests that developers need better tools to vet third-party components. Future work should explore if these feedback loops are equally effective for smaller apps outside the "superstar" category.
Summary updated