An Empirical Study of Ai Techniques in Mobile Applications

Yinghua Li, Xueqi Dang, Haoye Tian, Tiezhu Sun, Zhi-Jie Wang, Лей Ма, Jacques Klein, Tegawendé F. Bissyandé

TL;DR

This paper presents the most extensive empirical study to date on AI techniques in mobile applications, analyzing over 56,000 apps. It reveals key trends in AI adoption, framework usage, and user attitudes, highlighting the growing importance of on-device AI and the need for better model protection and user privacy.

What Problem Does This Paper Address?

The rapid integration of Artificial Intelligence (AI) into mobile applications has transformed user experiences, but a comprehensive understanding of this landscape is lacking. Existing research often focuses narrowly on deep learning (DL) or specific aspects like model protection, with limited user-centric analysis and often smaller datasets. This paper aims to fill this gap by conducting a large-scale, multi-faceted empirical study of AI in mobile apps, covering on-device machine learning (ML), on-device DL, and cloud-based AI services. The goal is to provide a holistic view of AI app development, usage, and user perception.

Methods

The core of this study is the development of "AI Discriminator," an automated tool designed to identify AI applications from a massive repository of Android apps (AndroZoo). This tool employs a pipeline involving APK decompilation and keyword matching against a custom-built AI keyword dictionary. To evaluate its effectiveness, the authors used standard metrics like precision, recall, and accuracy, achieving high scores.

The subsequent analysis is structured around three main perspectives:

  1. Application Analysis: This involves analyzing AI app popularity (release trends, market distribution, provider preferences, category prevalence) and update status (frequency, correlation with model/framework updates).
  2. Framework and Model Analysis: This delves into the internal workings of AI apps, examining AI framework usage (single vs. multi-framework, popularity trends) and AI model protection (use of open-source models, encryption status).
  3. User Analysis: This perspective focuses on the user experience, investigating user privacy protection (data access patterns) and analyzing user reviews to understand attitudes towards AI techniques.

Datasets

The primary dataset used in this study is derived from AndroZoo, a large-scale repository containing over 7.2 million Android applications. The authors' automated tool, AI Discriminator, successfully identified 56,682 real-world AI applications from this pool, forming the largest AI app dataset to date for research purposes. This dataset, along with their scripts, is made publicly available.

For model encryption analysis, a subset of 90 open-source mobile models from the TFLite Hub was used for comparison. For user privacy analysis, privacy policies from 6,016 AI apps were crawled.

Key Insights and Ideas

  • AI in Mobile is Booming: The trend of integrating AI into mobile apps has seen a significant surge since 2018, indicating a growing market and developer interest.
  • On-Device DL Dominates: While cloud-based AI services are present, on-device deep learning apps represent the largest segment, highlighting a preference for local processing, likely driven by privacy and latency benefits.
  • Framework Ecosystem: Lite Deep Learning Frameworks (like TensorFlow Lite) are the most prevalent, followed by General Cloud AI Frameworks and Computer Vision Frameworks. This suggests a focus on efficient deployment on resource-constrained devices.
  • Single Framework Preference: The majority of AI apps utilize a single AI framework, with TFLite being the most popular choice among these. However, multi-framework combinations are also common, especially for more complex applications.
  • Model Protection is Lacking: A significant concern is the low encryption rate of AI models embedded in apps (only 0.25% considered encrypted). This, coupled with the common use of directly renamed open-source models, exposes apps to security threats.
  • Privacy is a Major User Concern: Users are highly sensitive to how their data is accessed, with names, addresses, and emails being the most frequently collected private attributes. Lack of transparency in privacy policies and third-party data sharing are key user complaints.
  • Users are Generally Positive, but Accuracy Matters: Despite privacy concerns, users generally have a positive attitude towards AI techniques. However, accuracy issues are the leading cause of negative reviews, underscoring the importance of reliable AI performance.

Key Results

  • AI App Growth: The proportion of AI apps among all released apps has grown sharply since 2018.
  • Market Distribution: Google Play is the dominant market for AI app releases, significantly outpacing others.
  • Category Preferences: Finance and Business categories lead in the number of AI apps developed, while Strategy Games and Comics apps receive the highest average scores.
  • Update Frequency: The majority of AI apps (80.3%) are updated 0-5 times.
  • Model Update Correlation: Among updated AI apps, 46.18% also had their AI models updated.
  • Framework Transitions: TensorFlow → TFLite and TFLite → TensorFlow are the most frequent framework transitions observed during app updates.
  • Open-Source Model Usage: 0.7% of AI apps utilize open-source models by simply renaming them.
  • Encrypted Models: Only 0.25% of AI app models were found to be encrypted.
  • Privacy Data Access: User names, addresses, and emails are the most frequently accessed private attributes across various app categories.
  • User Review Sentiment: 68.4% of technical reviews for AI apps are positive, while 31.6% are negative.
  • Negative Review Causes: Accuracy issues (22.6%), incomplete functionality (17.7%), and crashes/errors (16.1%) are the primary drivers of negative user feedback.

Limitations and Future Work

Limitations:

  • Model Protection Measurement: The standard entropy test for model encryption is a heuristic and not guaranteed to be perfectly accurate. Manual verification was performed on a subset to mitigate this.
  • Keyword Collection: The manual collection of keywords for AI app identification, while extensive, might miss some AI apps that don't rely on explicit framework keywords.
  • External Validity of Model Protection: The study acknowledges that the entropy test for model encryption might have external validity threats.

Future Work:

  • LLM Integration: Exploring the integration of Large Language Models (LLMs) into mobile applications for enhanced user interaction and advanced features.
  • User Privacy and Security: Further research into ensuring data privacy and security without compromising AI model performance and usability.
  • AI Model Security: Developing robust methods to protect AI models from adversarial attacks, especially in critical applications.
  • Lightweight Model Optimization: Designing AI models that achieve high accuracy while being computationally lightweight for mobile devices.
  • Real-time Processing: Enabling real-time AI capabilities on mobile devices while maintaining high accuracy and low power consumption.
  • AI App Recommendations: Analyzing high-scoring AI apps to identify success factors and apply them to other categories.
  • User Feedback Integration: Continuously leveraging user feedback, particularly negative reviews, to improve AI applications.

Summary updated

BibTeX
@article{li2025empirical,
  author = {Yinghua Li and Xueqi Dang and Haoye Tian and Tiezhu Sun and Zhi-Jie Wang and Лей Ма and Jacques Klein and Tegawendé F. Bissyandé},
  title = {{An Empirical Study of Ai Techniques in Mobile Applications}},
  journal = {J. Syst. Softw.},
  year = {2025},
  doi = {10.1016/j.jss.2024.112233},
}
Figure