Abstract
E-commerce risk management is a complex domain requiring the integration of heterogeneous data and evolving business logic. General-purpose Large Language Models (LLMs) often struggle with domain-specific nuances, leading to high false-positive rates. This paper introduces SHERLOCK, a framework that combines a structured domain knowledge base with a multi-stage "Reflect & Refine" reasoning process. SHERLOCK incorporates a self-evolution mechanism through a data flywheel, allowing for rapid hotfixes and long-term model internalization of expert logic. Real-world deployment at JD.com demonstrates significant improvements in investigation efficiency and accuracy, with an 82% expert acceptance rate and a nearly fourfold increase in daily throughput.
1. Introduction
E-commerce risk investigation is a high-stakes task that involves connecting disparate dots across transaction logs, graph-based relationships, and internal business logic. Traditional automated systems often fail to capture the "long-tail" nuances of specific business domains. For instance, a general LLM might flag a high concentration of orders from a single IP address as a bot attack, failing to recognize it as benign behavior in a "Local Life Service" context, such as a large office building ordering lunch.
The researchers identify three primary challenges:
- Knowledge Fragmentation: Expert insights are often siloed in documentation or code.
- Contextual Mapping: Difficulty in applying the correct domain knowledge to specific case features.
- Adversarial Drift: The rapid evolution of fraud tactics renders static models obsolete.
2. The SHERLOCK Framework
SHERLOCK (Self-evolving Hierarchy for E-commerce Risk via LLM Orchestrated Contextual Knowledge) addresses these challenges through a three-pillared architecture.
2.1 Domain Knowledge Base (KB)
The foundation of SHERLOCK is a structured KB distilled from 1.2 million tokens of business documents, codebases, and expert records. It categorizes knowledge into domain terms, historical cases, risk associations, and "business priors"—legitimate justifications for suspicious-looking patterns.
2.2 Knowledge-Augmented Inference
The inference process follows a "Reflect & Refine" (R&R) cycle:
- Stage 1 (Input Augmentation): Relevant domain terms and historical cases are retrieved to provide initial context.
- Stage 2 (Reflect & Refine): After an initial analysis, the system uses preliminary findings to fetch specific business priors. It performs factual verification to mitigate hallucinations and applies expert logic to "rule out" false positives.
2.3 Self-Evolution and the STRO Framework
To combat adversarial drift, SHERLOCK employs a dual-track update system. Short-term "hotfixes" allow analysts to inject new rules immediately. For long-term adaptation, the Suspect-Then-Rule-Out (STRO) framework synthesizes high-quality training data from expert corrections, which is then used for Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).
3. Experimental Evaluation
SHERLOCK was evaluated using 5,000 real-world production cases from JD.com and a curated benchmark of 1,000 expert-annotated samples.
As shown in the results, SHERLOCK-enhanced models (e.g., JoyAI-10B) significantly outperform much larger vanilla models like DeepSeek-R1 and JoyAI-750B. Specifically, the Signal-to-Noise Ratio (SNR) and Core Detection Rate (CDR) show marked improvements, indicating that structured domain grounding is more effective than raw parameter scale for specialized industrial tasks.
3.1 Operational Impact
The deployment of SHERLOCK at JD.com led to transformative efficiency gains:
- Review Efficiency: Average case review time decreased by 79.5%, from 15.24 minutes to 3.13 minutes.
- Throughput: Daily investigation capacity increased by 386.7%, from 945 to 4,600 cases.
- Accuracy: The Expert Acceptance Rate (EAR) reached 82%, compared to just 31% for legacy systems.
4. Conclusion
SHERLOCK demonstrates that LLMs can achieve professional-grade performance in e-commerce risk management when integrated with structured domain knowledge and a dynamic self-evolution mechanism. The "Reflect & Refine" process effectively filters out business noise, while the STRO framework ensures the model adapts to shifting adversarial tactics. Future work will focus on automating knowledge distillation for complex edge cases and optimizing the reasoning engine for real-time applications.
Summary updated