A Domain-Adapted Pipeline for Structured Information Extraction from Police Incident Announcements on Social Media

Mengfan Shen, Kaixin Song, Xindi Wang, Weijia Jia, Tao Wang, Ziqiang HanarXiv 2025 · DOI · arXiv

Abstract

Structured information extraction from police incident announcements is crucial for timely and accurate data processing, yet presents considerable challenges due to the variability and informal nature of textual sources such as social media posts. To address these challenges, we developed a domain-adapted extraction pipeline that leverages targeted prompt engineering with parameter-efficient fine-tuning of the Qwen2.5-7B model using Low-Rank Adaptation (LoRA). This approach enables the model to handle noisy, heterogeneous text while reliably extracting 15 key fields, including location, event characteristics, and impact assessment, from a high-quality, manually annotated dataset of 4,933 instances derived from 27,822 police briefing posts on Chinese Weibo (2019-2020). Experimental results demonstrated that LoRA-based fine-tuning significantly improved performance over both the base and instruction-tuned models, achieving an accuracy exceeding 98.36% for mortality detection and Exact Match Rates of 95.31% for fatality counts and 95.54% for province-level location extraction. The proposed pipeline thus provides a validated and efficient solution for multi-task structured information extraction in specialized domains, offering a practical framework for transforming unstructured text into reliable structured data in social science research.

Introduction

In criminology and public policy research, high-quality data serves as the "ground truth" essential for evidence-based decision-making. However, in many regions, large-scale, publicly accessible crime datasets are scarce. In China, police departments frequently utilize Weibo (a major social media platform) to post incident briefings. While these posts are rich in information, the data is typically trapped in unstructured narrative formats—often embedded within text or images of text.

Extracting consistent variables such as injury counts, specific illegal behaviors, or precise locations from these posts is a significant challenge for traditional Natural Language Processing (NLP). Social media text is inherently noisy, informal, and heterogeneous. Previous attempts using classical machine learning (SVMs) or standard Deep Learning architectures (CNNs/RNNs) have struggled with complex semantic nuances and the strict formatting required for structured databases. This paper presents a domain-adapted pipeline designed to overcome these hurdles.

Methodology

The proposed extraction pipeline consists of three primary stages: Data Creation, Prompt Engineering, and Model Training.

Data Creation and Annotation

The researchers crawled over 27,000 Weibo posts from 2019 to 2020. Optical Character Recognition (OCR) was employed to extract text from image-based briefings. From this pool, a "gold standard" dataset of 4,933 instances was manually annotated. The extraction focuses on 15 key variables, including:

  • Geospatial Information: Province and city-level data.
  • Case Characteristics: Cybercrime status, completion status, and event types.
  • Impact Assessment: Casualty counts (fatalities and injuries) and economic losses.

Structured Prompt Engineering

To ensure machine-readability, the pipeline utilizes "Structured Prompts" rather than simple zero-shot queries. These prompts incorporate role-setting (e.g., "You are a professional assistant..."), explicit JSON schema requirements, and a standardized coding table for event types. This structure forces the model to align its output with the requirements of a relational database.

LoRA Fine-Tuning

The technical core of the pipeline is the application of Low-Rank Adaptation (LoRA) to the Qwen2.5-7B model. LoRA is a parameter-efficient fine-tuning (PEFT) technique that freezes the original model weights and trains only a small fraction of additional parameters (low-rank matrices). This allows for high-performance domain adaptation without the prohibitive computational costs associated with full-parameter fine-tuning. Sensitivity analysis determined that a LoRA rank ($r$) of 16 and a low decoding temperature ($T=0.1$) provided the optimal balance between efficiency, accuracy, and strict adherence to the JSON schema.

Experimental Results

The fine-tuned Qwen2.5-7B model was benchmarked against traditional baselines (SVM, BERT, BiLSTM-CRF) and several prominent Large Language Models (LLMs), including LLaMA3, Gemma2, and API-based models like GPT-5 and DeepSeek-V3.

Extraction Performance

The LoRA-tuned model demonstrated exceptional precision across critical fields:

  • Mortality Detection: 98.36% accuracy.
  • Fatality Counts: 95.31% Exact Match Rate (EMR).
  • Location Extraction: 95.54% EMR for province-level data.
  • Schema Compliance: While base models frequently failed to produce valid JSON, the LoRA-tuned versions achieved nearly 100% compliance.

Semantic Alignment

On descriptive fields, such as "Police Handling Actions," the fine-tuned model's Cosine Similarity scores improved significantly compared to standard instruction-tuned versions, rising from approximately 13% to 65%. This indicates a much higher degree of alignment with human-annotated ground truth.

Discussion: Specialization vs. Scale

A key insight from this research is that specialization often beats scale in domain-specific tasks. The fine-tuned 7B parameter model consistently outperformed massive "frontier" models (such as GPT-5 or Claude-4) in extracting precise numerical data and locations. While larger models remain superior at subjective reasoning—such as assessing the "social impact" of an event—the specialized model proved more reliable for structured data extraction.

The results suggest that prompt engineering and fine-tuning are complementary rather than redundant. Prompting effectively guides the output format (JSON compliance), while LoRA ensures domain alignment, allowing the model to interpret specific Chinese legal and administrative terminology correctly.

Limitations and Future Work

Despite its high performance, the pipeline faces certain limitations. The model still lags behind larger LLMs in tasks requiring professional human-like judgment, such as nuanced severity assessments. Additionally, inferring city-level locations can be difficult when the source text only mentions district names without broader context.

Future research will explore the integration of Reinforcement Learning from Human Feedback (RLHF) to refine subjective assessments and the use of AI Agent technologies to handle complex, multi-step reasoning tasks. This pipeline represents a significant step toward transforming unstructured social media narratives into reliable, structured data for social science research.

Summary updated

Abstract

Structured information extraction from police incident announcements is crucial for timely and accurate data processing, yet presents considerable challenges due to the variability and informal nature of textual sources such as social media posts. To address these challenges, we developed a domain-adapted extraction pipeline that leverages targeted prompt engineering with parameter-efficient fine-tuning of the Qwen2.5-7B model using Low-Rank Adaptation (LoRA). This approach enables the model to handle noisy, heterogeneous text while reliably extracting 15 key fields, including location, event characteristics, and impact assessment, from a high-quality, manually annotated dataset of 4,933 instances derived from 27,822 police briefing posts on Chinese Weibo (2019-2020). Experimental results demonstrated that LoRA-based fine-tuning significantly improved performance over both the base and instruction-tuned models, achieving an accuracy exceeding 98.36% for mortality detection and Exact Match Rates of 95.31% for fatality counts and 95.54% for province-level location extraction. The proposed pipeline thus provides a validated and efficient solution for multi-task structured information extraction in specialized domains, offering a practical framework for transforming unstructured text into reliable structured data in social science research.
BibTeX
@misc{shen2025domain,
  author = {Mengfan Shen and Kaixin Song and Xindi Wang and Weijia Jia and Tao Wang and Ziqiang Han},
  title = {{A Domain-Adapted Pipeline for Structured Information Extraction from Police Incident Announcements on Social Media}},
  year = {2025},
  eprint = {2512.16183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.16183},
  howpublished = {arXiv},
}
Figure