QExplorer: Large Language Model Based Query Extraction for Toxic Content Exploration

Shaola Ren, Li Ke, Longtao Huang, Dehong Gao, Hui XuearXiv 2025 · DOI · arXiv

Introduction

On massive e-commerce platforms like Alibaba’s Xianyu, maintaining a safe environment is a constant challenge. "Toxic" content—including fraud, prohibited items, and illegal services—often evades standard automated classifiers through "camouflage" techniques like coded language, rare word combinations, or intentional typos. When suspicious items slip through, human auditors must manually brainstorm search queries to find similar hidden violations. This process is mentally taxing and often inefficient, as human-selected keywords can be either too broad or too specific. QExplorer addresses this by leveraging Large Language Models (LLMs) to automate the "risk understanding" phase, generating precise search queries that effectively hunt down camouflaged toxic content.

Methodology

QExplorer treats query extraction as a sequence-to-sequence generation task, specifically optimized for real-world search system performance. The framework employs a multi-stage training pipeline to align the LLM with the goals of toxic content exploration.

The three-step training pipeline for QExplorer, featuring SFT and DPO alignment with search engine feedback.

The core innovation lies in its alignment strategy:

  1. Instruction Supervised Fine-Tuning (SFT): The model (e.g., Qwen or ChatGLM) is initially trained on historical human-annotated keywords to learn the fundamental task of identifying risky phrases within content.
  2. Direct Preference Optimization (DPO): To move beyond simple human mimicry, the model is fine-tuned using search engine feedback. By creating pairs of "preferred" (successful retrieval) and "dispreferred" (failed retrieval) queries from search logs, the model learns to prioritize keywords that actually yield results in the production environment.
  3. Long-Context Clustering: By clustering similar toxic items and providing the LLM with a "long context" of related violations, the system identifies rare co-occurrence patterns that individual items might not reveal.
  4. Efficient Adaptation: The researchers utilized Low-Rank Adaptation (LoRA) to fine-tune 7B-parameter models efficiently on standard hardware.

Datasets and Evaluation

The framework was developed and tested using real-world data from the Xianyu platform:

  • Training Data: Over 25,000 pairs of toxic content and human-annotated phrases.
  • Preference Data: Approximately 700 query pairs ranked by their actual "hit rate" in the live search system.
  • Evaluation: A test set of 402 reported toxic items from late 2024 was used for offline benchmarking.

Key Results and Impact

QExplorer demonstrated significant improvements in both offline benchmarks and a 20-week online deployment. Offline, a Qwen1.5-7B based QExplorer achieved a Query Hit Rate of 0.561, outperforming human auditors (0.489) and GPT-4 Turbo (0.526).

Online performance trends showing significant increments in toxic item detection and successful query hits over 20 weeks.

In production, the impact was even more pronounced:

  • Detection Increase: LLM-suggested queries led to a 59.9% increase in toxic item detection compared to the human-only baseline.
  • Efficiency Gains: The "Hit Query Increment" rose by nearly 40%, indicating the model successfully identifies "pockets" of toxic content previously missed by humans.
  • Human-in-the-loop Acceptance: Professional auditors accepted the model's suggestions 61.6% of the time, validating the practical utility of the generated queries.

Discussion and Future Work

A primary insight from this work is that search engine feedback is a superior signal for query optimization than human intuition alone. While humans may pick descriptive words, the LLM learns to select "surgical" keywords that minimize noise.

However, limitations remain. Currently, QExplorer is text-centric, whereas bad actors often hide information in images. Future iterations will focus on multimodal fine-tuning to incorporate visual data. Additionally, the researchers are exploring synthetic data generation techniques to overcome the scarcity of high-quality preference data for DPO.

Summary updated

Abstract

Automatically extracting effective queries is challenging in information retrieval, especially in toxic content exploration, as such content is likely to be disguised. With the recent achievements in generative Large Language Model (LLM), we are able to leverage the capabilities of LLMs to extract effective queries for similar content exploration directly. This study proposes QExplorer, an approach of large language model based Query Extraction for toxic content Exploration. The QExplorer approach involves a 2-stage training process: instruction Supervised FineTuning (SFT) and preference alignment using Direct Preference Optimization (DPO), as well as the datasets construction with feedback of search system. To verify the effectiveness of QExplorer, a series of offline and online experiments are conducted on our real-world system. The offline empirical results demonstrate that the performance of our automatic query extraction outperforms that of several LLMs and humans. The online deployment shows a significant increase in the detection of toxic items.
BibTeX
@misc{ren2025qexplorer,
  author = {Shaola Ren and Li Ke and Longtao Huang and Dehong Gao and Hui Xue},
  title = {{QExplorer: Large Language Model Based Query Extraction for Toxic Content Exploration}},
  year = {2025},
  eprint = {2502.18480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18480},
}
Figure