Introduction: The Scaling Crisis in App Vetting
Mobile app stores are inundated with thousands of new submissions daily. While many are legitimate, a significant portion contains malicious elements such as adware, spyware, or counterfeit features. Currently, platform operators rely on a labor-intensive manual review process. Security experts must navigate a fragmented landscape of single-purpose automated tools—one for malware detection, another for ad fraud, and yet another for privacy violations. This patchwork approach is slow, expensive, and increasingly unable to scale with the volume of modern app ecosystems.
Large Language Models (LLMs) offer a potential solution due to their ability to synthesize messy, multi-source data. However, applying LLMs to cybersecurity presents two major challenges: high-dimensional data that exceeds standard context windows and the risk of "hallucinations," where models invent fake evidence. In the context of app vetting, a hallucinated risk could lead to wrongful delisting and significant legal liabilities. To address these issues, this paper introduces MARS, an automated system designed to provide scalable and interpretable risk analysis.
The MARS Framework: Bridging Heuristics and LLMs
MARS (Multi-agent App Risk analysis System) operates through a two-phase pipeline that restricts and guides LLM reasoning using expert-defined structures.
1. The Risk Identification Tree (Offline Phase)
Instead of feeding raw data directly into an LLM, MARS utilizes a pre-constructed hierarchical decision tree. This tree integrates two types of features:
- Knowledge-driven features: Heuristics derived from expert security engineers, such as specific patterns in user complaints or known malicious API call sequences.
- Data-driven features: Statistical anomalies identified through historical analysis, such as sudden, sharp drops in app ratings or unexpected spikes in background data usage.
2. Multi-Agent Evidence Filtering (Online Phase)
When a new app is submitted, MARS employs specialized, lightweight LLM agents assigned to specific data groups (e.g., a "User Feedback Agent" or a "Runtime Monitor Agent"). These agents distill noisy, voluminous data into concise semantic summaries. This "divide and conquer" strategy drastically reduces the context size for the main reasoning model, ensuring it stays within operational limits while maintaining focus on relevant risk indicators.
Generating Interpretable Evidence
A core contribution of MARS is its ability to produce human-readable evidence chains. To prevent false positives, the system requires at least two leaf nodes of the risk tree to trigger before flagging an application. It then uses Retrieval-Augmented Generation (RAG) to cross-reference the flagged behavior against a database of historically delisted apps, ensuring the current judgment aligns with established malicious patterns.
As shown in the example report, MARS outputs a comprehensive narrative explaining why an app was flagged. This includes specific evidence snippets from execution patterns and runtime monitors, a detailed risk description, and a list of similar risky apps to provide context for human reviewers.
Evaluation and Real-World Impact
The system was evaluated using a dataset of 2,232 real-world applications from a major Android app store. This included a labeled set of 136 apps for accuracy testing and an unlabeled set of 2,096 apps to simulate deployment conditions.
- High Accuracy: MARS achieved an F1-score of 0.838 for risk identification and 0.934 for evidence retrieval.
- Efficiency: The system analyzes an app in an average of 24.68 seconds, costing approximately $0.015 per application in compute resources.
- Expert Validation: In a study involving 20 security practitioners, MARS was found to reduce manual review time by 60% to 90% by automating the most tedious aspects of evidence gathering and report writing.
Conclusion and Future Directions
By forcing LLMs to route their reasoning through expert-guided decision trees and validating findings via RAG, MARS effectively neutralizes hallucinations while maintaining high throughput. While the system currently excels at identification and evidence gathering, future work will focus on automating behavior reproduction—the process of automatically triggering malicious behaviors in emulated environments to provide definitive proof for bans. Additionally, the team aims to optimize smaller, local LLMs to further reduce operational costs without sacrificing reasoning performance.