SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management

Nan Lu, Yurong Hu, Jiaquan Fang, Yan Liu, Rui Dong, Yiming Wang, Rui Lin, Shaoyi XuarXiv 2025 · DOI · arXiv

Abstract

E-commerce risk management is a complex domain requiring the integration of heterogeneous data and evolving business logic. General-purpose Large Language Models (LLMs) often struggle with domain-specific nuances, leading to high false-positive rates. This paper introduces SHERLOCK, a framework that combines a structured domain knowledge base with a multi-stage "Reflect & Refine" reasoning process. SHERLOCK incorporates a self-evolution mechanism through a data flywheel, allowing for rapid hotfixes and long-term model internalization of expert logic. Real-world deployment at JD.com demonstrates significant improvements in investigation efficiency and accuracy, with an 82% expert acceptance rate and a nearly fourfold increase in daily throughput.

1. Introduction

E-commerce risk investigation is a high-stakes task that involves connecting disparate dots across transaction logs, graph-based relationships, and internal business logic. Traditional automated systems often fail to capture the "long-tail" nuances of specific business domains. For instance, a general LLM might flag a high concentration of orders from a single IP address as a bot attack, failing to recognize it as benign behavior in a "Local Life Service" context, such as a large office building ordering lunch.

The researchers identify three primary challenges:

  1. Knowledge Fragmentation: Expert insights are often siloed in documentation or code.
  2. Contextual Mapping: Difficulty in applying the correct domain knowledge to specific case features.
  3. Adversarial Drift: The rapid evolution of fraud tactics renders static models obsolete.

2. The SHERLOCK Framework

SHERLOCK (Self-evolving Hierarchy for E-commerce Risk via LLM Orchestrated Contextual Knowledge) addresses these challenges through a three-pillared architecture.

The SHERLOCK system architecture, illustrating the four-stage process from knowledge base construction to refined output and system evolution.

2.1 Domain Knowledge Base (KB)

The foundation of SHERLOCK is a structured KB distilled from 1.2 million tokens of business documents, codebases, and expert records. It categorizes knowledge into domain terms, historical cases, risk associations, and "business priors"—legitimate justifications for suspicious-looking patterns.

2.2 Knowledge-Augmented Inference

The inference process follows a "Reflect & Refine" (R&R) cycle:

  • Stage 1 (Input Augmentation): Relevant domain terms and historical cases are retrieved to provide initial context.
  • Stage 2 (Reflect & Refine): After an initial analysis, the system uses preliminary findings to fetch specific business priors. It performs factual verification to mitigate hallucinations and applies expert logic to "rule out" false positives.

2.3 Self-Evolution and the STRO Framework

To combat adversarial drift, SHERLOCK employs a dual-track update system. Short-term "hotfixes" allow analysts to inject new rules immediately. For long-term adaptation, the Suspect-Then-Rule-Out (STRO) framework synthesizes high-quality training data from expert corrections, which is then used for Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).

3. Experimental Evaluation

SHERLOCK was evaluated using 5,000 real-world production cases from JD.com and a curated benchmark of 1,000 expert-annotated samples.

Performance comparison of SHERLOCK against legacy and vanilla LLM methods across various risk metrics.

As shown in the results, SHERLOCK-enhanced models (e.g., JoyAI-10B) significantly outperform much larger vanilla models like DeepSeek-R1 and JoyAI-750B. Specifically, the Signal-to-Noise Ratio (SNR) and Core Detection Rate (CDR) show marked improvements, indicating that structured domain grounding is more effective than raw parameter scale for specialized industrial tasks.

3.1 Operational Impact

The deployment of SHERLOCK at JD.com led to transformative efficiency gains:

  • Review Efficiency: Average case review time decreased by 79.5%, from 15.24 minutes to 3.13 minutes.
  • Throughput: Daily investigation capacity increased by 386.7%, from 945 to 4,600 cases.
  • Accuracy: The Expert Acceptance Rate (EAR) reached 82%, compared to just 31% for legacy systems.

4. Conclusion

SHERLOCK demonstrates that LLMs can achieve professional-grade performance in e-commerce risk management when integrated with structured domain knowledge and a dynamic self-evolution mechanism. The "Reflect & Refine" process effectively filters out business noise, while the STRO framework ensures the model adapts to shifting adversarial tactics. Future work will focus on automating knowledge distillation for complex edge cases and optimizing the reasoning engine for real-time applications.

Summary updated

BibTeX
@misc{lu2025sherlock,
  author = {Nan Lu and Yurong Hu and Jiaquan Fang and Yan Liu and Rui Dong and Yiming Wang and Rui Lin and Shaoyi Xu},
  title = {{SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management}},
  year = {2025},
  eprint = {2510.08948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.08948},
}
Figure