Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search

Zequn Xie, Guijin Luo, Chuxin Wang, Sihang Cai, Tao Jin, Zhou Zhao, Yixuan TangarXiv 2026 · arXiv

TL;DR

Text-based person anomaly search faces a challenging "Pose-Semantic Gap" where completely different behaviors (like falling down vs. doing push-ups) share nearly identical skeletal geometries. The proposed Structure-Semantic Decoupled Cascade (SSDC) framework bridges this gap by pairing a fast, lightweight structural filter with a multi-agent "Detective Squad" powered by a Multimodal Large Language Model (MLLM) for precise, efficient semantic verification.

What Problem Does This Paper Address?

Text-based person search allows operators to retrieve specific individuals from surveillance archives using natural language queries. However, extending this to anomaly search (e.g., identifying shoplifting, accidents, or fights) introduces a major hurdle: the Pose-Semantic Gap.

Traditional pose-aware models rely heavily on geometric alignment, but semantically distinct actions often look identical when reduced to a skeleton. For instance, a person reaching into a bag to steal looks structurally similar to someone putting an item into a shopping basket.

Figure 1: Illustration of the Pose-Semantic Gap. Tra- ditional pose-aware methods (left) fail to distinguish semantically distinct actions with similar skeletal ge- ometries. Our SSDC framework (right) bridges this gap through Multi-Round Interactive Verification using a collaborative agent workflow (Verify →Extract → Rewrite) to analyze fine-grained visual details and gen- erate semantically accurate captions.

While Multimodal Large Language Models (MLLMs) possess the advanced reasoning capabilities needed to tell these actions apart, deploying them directly across massive surveillance databases is computationally impossible due to high inference latency. This paper addresses the dilemma of choosing between fast but inaccurate lightweight models and accurate but prohibitively slow large models.

Methods

The authors introduce the Structure-Semantic Decoupled Cascade (SSDC) framework, which splits the retrieval process into a coarse-to-fine pipeline:

  1. Stage 1: Structure-Aware Coarse Retrieval
    A lightweight, pose-aware model acts as a high-speed filter. It extracts a pose map and fuses it with image features using cross-attention to quickly scan the entire database, filtering out over 99.9% of irrelevant candidates based on structural similarity.

  2. Stage 2: Detective Squad Interaction
    Instead of using a single, monolithic MLLM prompt on the remaining top-$K$ candidates, the authors orchestrate a collaborative multi-agent workflow using a fine-tuned Qwen3-VL-8B-Instruct model:

  3. The Detective (Filter): Acts as an efficient gatekeeper, performing rapid binary filtering ("Match" or "Discard") to throw out obvious false positives.

  4. The Analyst (Extractor): For surviving candidates, it conducts a "physical examination," checking fine-grained visual details against a 15-point checklist (gender, clothing, specific actions).
  5. The Writer (Integrator): Synthesizes the Analyst's fragmented evidence into a coherent, highly accurate new caption ($T_{new}$).
Figure 2: Overall architecture of the SSDC framework . The framework follows a coarse-to-fine pipeline : (1) Coarse Retrieval uses a lightweight model to filter the gallery based on structural similarity. (2) Semantic Verification introduces a specialized Detective Agent to scrutinize hard negatives. This agent employs Detective-style Prompting to resolve fine-grained ambiguities through multi-round reasoning and visual entailment. Finally, an Adaptive Fusion Mechanism integrates structural scores with the agent’s semantic verdicts to produce the final ranking.
  1. Final Ranking Generation
    An adaptive fusion mechanism calculates a text-to-text semantic similarity score between the user's original query and the Writer's generated caption, combining it with the structural score from Stage 1 to produce the final ranking.

Datasets

The framework was evaluated on two primary benchmarks:

  • Pedestrian Anomaly Behavior (PAB) Benchmark: A large-scale dataset containing over 1 million image-text pairs, covering 1,600 anomaly types and 1,000 normal action types. It includes a Multi-weather Setting to simulate 10 different weather conditions (rain, snow, dark, etc.).
  • UCC Test Set: Extracted from the real-world UCF-Crime dataset, containing 5,320 unseen image-text pairs. This was used strictly to evaluate out-of-distribution (OOD) generalization.

Key Insights and Ideas

  • Decoupling is Key to Efficiency: Trying to force a single model to handle both massive-scale structural alignment and deep semantic reasoning is a losing battle. Decoupling them allows expensive MLLM reasoning to be strictly rationed for ambiguous, high-value candidates.
  • The Power of Role-Playing Agents: Standard MLLM captioning is prone to hallucinations. Breaking the task into a structured chain of thought (Verify $\rightarrow$ Extract $\rightarrow$ Rewrite) forces the model to anchor its final summary in explicit visual evidence.
  • The Analyst is the Bottleneck Breaker: Ablation studies revealed that introducing the Analyst agent provided the largest single performance jump ($\uparrow$ 1.92% R@1). This proves that the core bottleneck in bridging the Pose-Semantic Gap is the lack of explicit visual evidence extraction.

Key Results

  • State-of-the-Art Performance: On the full PAB dataset, SSDC set a new SOTA by achieving 87.01% Recall@1 and 92.74% mAP, noticeably outperforming strong baselines like IRRA (78.67% R@1) and the standalone CMP model (84.93% R@1).
  • Robustness to Weather: SSDC demonstrated consistent performance gains across all 10 simulated weather degradation scenarios, proving that the Detective Squad's hierarchical reasoning provides resilience even when visual cues are compromised.
  • Strong Generalization (OOD): On the unseen UCC dataset, SSDC achieved 59.45% R@1 and 45.25% mAP, significantly outperforming specialized person retrieval baselines and proving it learns generalized anomaly intents rather than dataset biases.

Limitations and Future Work

While highly effective, the framework primarily operates on static image frames rather than continuous video streams. Many real-world anomalies are inherently temporal and cannot be fully understood from a single snapshot. The authors note that extending this multi-agent cascade framework to handle complex spatiotemporal reasoning across full video sequences is a promising direction for future work.

Summary updated

Abstract

Text-based person anomaly search retrieves specific behavioral events from surveillance archives using natural-language queries. Although recent pose-aware methods align geometric structures well, they face a fundamental Pose-Semantic Gap: semantically different actions can share similar skeletal geometries. While Multimodal Large Language Models (MLLMs) can reduce this ambiguity, using them for large-scale retrieval is computationally prohibitive. We propose the Structure-Semantic Decoupled Cascade (SSDC) framework, which decouples retrieval into two stages: (1) Structure-Aware Coarse Retrieval, where a lightweight model quickly filters candidates by skeletal similarity ; and (2) Detective Squad Interaction, a multi-agent semantic verification module. The squad consists of a Detective for fast binary filtering, an Analyst for evidence extraction, and a Writer for semantic synthesis. Finally, we re-rank candidates by fusing the synthesized captions with structural priors. Experiments on the PAB benchmark show that SSDC achieves state-of-the-art performance by balancing efficiency and semantic reasoning.
BibTeX
@misc{xie2026bridging,
  author = {Zequn Xie and Guijin Luo and Chuxin Wang and Sihang Cai and Tao Jin and Zhou Zhao and Yixuan Tang},
  title = {{Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search}},
  year = {2026},
  eprint = {2604.23282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.23282},
}

Loading related content...

Figure