Introduction
On massive e-commerce platforms like Alibaba’s Xianyu, maintaining a safe environment is a constant challenge. "Toxic" content—including fraud, prohibited items, and illegal services—often evades standard automated classifiers through "camouflage" techniques like coded language, rare word combinations, or intentional typos. When suspicious items slip through, human auditors must manually brainstorm search queries to find similar hidden violations. This process is mentally taxing and often inefficient, as human-selected keywords can be either too broad or too specific. QExplorer addresses this by leveraging Large Language Models (LLMs) to automate the "risk understanding" phase, generating precise search queries that effectively hunt down camouflaged toxic content.
Methodology
QExplorer treats query extraction as a sequence-to-sequence generation task, specifically optimized for real-world search system performance. The framework employs a multi-stage training pipeline to align the LLM with the goals of toxic content exploration.
The core innovation lies in its alignment strategy:
- Instruction Supervised Fine-Tuning (SFT): The model (e.g., Qwen or ChatGLM) is initially trained on historical human-annotated keywords to learn the fundamental task of identifying risky phrases within content.
- Direct Preference Optimization (DPO): To move beyond simple human mimicry, the model is fine-tuned using search engine feedback. By creating pairs of "preferred" (successful retrieval) and "dispreferred" (failed retrieval) queries from search logs, the model learns to prioritize keywords that actually yield results in the production environment.
- Long-Context Clustering: By clustering similar toxic items and providing the LLM with a "long context" of related violations, the system identifies rare co-occurrence patterns that individual items might not reveal.
- Efficient Adaptation: The researchers utilized Low-Rank Adaptation (LoRA) to fine-tune 7B-parameter models efficiently on standard hardware.
Datasets and Evaluation
The framework was developed and tested using real-world data from the Xianyu platform:
- Training Data: Over 25,000 pairs of toxic content and human-annotated phrases.
- Preference Data: Approximately 700 query pairs ranked by their actual "hit rate" in the live search system.
- Evaluation: A test set of 402 reported toxic items from late 2024 was used for offline benchmarking.
Key Results and Impact
QExplorer demonstrated significant improvements in both offline benchmarks and a 20-week online deployment. Offline, a Qwen1.5-7B based QExplorer achieved a Query Hit Rate of 0.561, outperforming human auditors (0.489) and GPT-4 Turbo (0.526).
In production, the impact was even more pronounced:
- Detection Increase: LLM-suggested queries led to a 59.9% increase in toxic item detection compared to the human-only baseline.
- Efficiency Gains: The "Hit Query Increment" rose by nearly 40%, indicating the model successfully identifies "pockets" of toxic content previously missed by humans.
- Human-in-the-loop Acceptance: Professional auditors accepted the model's suggestions 61.6% of the time, validating the practical utility of the generated queries.
Discussion and Future Work
A primary insight from this work is that search engine feedback is a superior signal for query optimization than human intuition alone. While humans may pick descriptive words, the LLM learns to select "surgical" keywords that minimize noise.
However, limitations remain. Currently, QExplorer is text-centric, whereas bad actors often hide information in images. Future iterations will focus on multimodal fine-tuning to incorporate visual data. Additionally, the researchers are exploring synthetic data generation techniques to overcome the scarcity of high-quality preference data for DPO.
Summary updated