CoT-PL: Visual Chain-of-Thought Reasoning Meets Pseudo-Labeling for Open-Vocabulary Object Detection

Hojun Choi, Youngsun Lim, Jaeyo Shin, Hyunjung ShimSubmission to ICLR 2026 · Rejected

TL;DR

Existing open-vocabulary object detectors rely on direct CLIP image–text matching to generate pseudo-labels, which breaks down in crowded or occluded scenes. CoT-PL rethinks pseudo-labeling as a multi-step visual chain-of-thought (CoT) reasoning pipeline (proposing candidate masks, verifying presence, naming classes zero-shot, and grounding background) combined with Contrastive Background Learning (CBL). This substantially reduces false positives and background collapse, achieving state-of-the-art novel-class detection on OV-COCO (+7.7 $\text{AP}_{50}$) and OV-LVIS (+2.9 mask $\text{AP}$).

What Problem Does This Paper Address?

Open-Vocabulary Object Detection (OVD) trains detectors on base categories while requiring them to detect unseen (novel) categories at inference time. Many state-of-the-art approaches automate pseudo-labeling for novel objects using Vision-Language Models (VLMs) like CLIP or image captions. However, this single-step image–text matching fails in visually complex, crowded, or occluded scenes due to three major failure modes:

  1. Noisy pseudo-boxes: VLMs trained on web image–text pairs exhibit strong co-occurrence biases (e.g., mislabeling a person's foot as a "skateboard").
  2. Caption dependency: Unannotated objects absent from human captions remain unlabeled, missing valid novel instances.
  3. Background collapse: Partially occluded or hard-to-detect instances are ignored and mistakenly treated as negative background signals during training.

Methods

CoT-PL decomposes pseudo-label generation into a three-step visual chain-of-thought (CoT) process using the Segment Anything Model (SAM) and a Multimodal Large Language Model (MLLM, default: Qwen2-7B), followed by contrastive representation learning:

[ Input Image ] ──► Step 1: SAM Region Proposals + MLLM Object Verification
                         │
                         ▼
                    Step 2: Soft Masking (Blur/Grayscale) + MLLM Zero-Shot Naming
                         │
                         ▼
                    Step 3: MLLM Background Grounding
                         │
                         ▼
            Contrastive Background Learning (CBL) with Bag-of-Regions
  • Step 1: Region Perception & Verification: SAM generates class-agnostic, whole-instance candidate masks. The MLLM acts as a ternary gate (Yes / No / Unsure) answering whether an identifiable object actually exists inside the box, pruning background noise.
  • Step 2: Caption-Free Zero-Shot Naming: The MLLM identifies the object within each proposal box. To prevent visual attention leakage and silhouette artifacts, non-target pixels are suppressed using soft masking (Gaussian blur and grayscaling) rather than hard black masks. Low-frequency outlier labels (below the minimum base-class frequency) are pruned to retain reliable semantic anchors.
  • Step 3: Contextual Background Grounding: The MLLM categorizes concepts as foreground or background (e.g., classifying grass or sky as background).
  • Contrastive Background Learning (CBL): Building upon the BARON framework, pre-computed anchors are cached and batched into bags of regions. The identified background concepts serve as explicit negative samples in an alignment InfoNCE contrastive loss, actively disentangling novel objects from background clutter.

Datasets

  • OV-COCO: Evaluated on the OVR-CNN split (48 base, 17 novel classes).
  • OV-LVIS: Evaluated on LVIS v1.0 following the ViLD protocol (1,203 total classes: 337 rare categories treated as novel, evaluated on mask AP).
  • Objects365: Used strictly for zero-shot cross-dataset transfer evaluation (trained on OV-LVIS, tested directly on Objects365 without fine-tuning).

Key Insights and Ideas

  • CoT beats single-step VLM alignment in cluttered scenes: Breaking perception into presence verification, naming, and background identification provides massive improvements in occluded (+168.4% relative gain in pseudo-label quality) and crowded (+103.4%) settings over previous pseudo-labelers.
  • Context suppression requires soft boundaries: Hard blacking-out of non-target regions introduces stark silhouette cues that hallucinate shapes (e.g., a background tree masked into a giraffe silhouette gets classified as a giraffe). Blurring and grayscaling retains subtle context while focusing MLLM attention, gaining +3.2 $\text{AP}_{50}^N$ over hard black masking.
  • Background as explicit negative contrast: Rather than letting unlabeled novel instances collapse into the background class, grounding generic background concepts (sky, water, vegetation) and using them as contrastive negatives forces the detector to cleanly separate objects from non-object features.

Key Results

  • OV-COCO: CoT-PL achieves 41.7% $\text{AP}_{50}^N$ on novel classes using a standard ResNet-50-FPN backbone, outperforming prior pseudo-labeling methods (SAS-Det at 37.4%, PB-OVD at 30.8%) and distillation baselines (BARON at 34.0%).
  • OV-LVIS: Reaches 23.0% $\text{AP}_r$, outperforming ViLD (16.6%), Detic (17.8%), and BARON (19.2%).
  • Complex Scene Pseudo-Label Quality: In dedicated evaluation on the COCO validation set, CoT-PL novel pseudo-label quality reaches 23.6% in crowded scenes (vs. 11.6% for SAS-Det) and 15.3% under heavy occlusion (vs. 5.7% for SAS-Det).
  • Cross-Dataset Generalization: LVIS-trained CoT-PL transfers zero-shot to COCO (36.8% AP) and Objects365 (13.9% AP), outperforming F-VLM and BARON.

Limitations and Future Work

  • Reliance on MLLM Capability: Performance scales with teacher model strength (BLIP-2 2.7B achieves 37.6% $\text{AP}_{50}^N$ vs. Qwen2-7B at 41.7%), making the pipeline dependent on capable vision-language frontends.
  • Irreversible False Negatives: The verification step discards all proposals marked "Unsure" by the MLLM, permanently losing difficult or long-tail instances.
  • Aggressive Frequency Filtering: Discarding pseudo-labels with counts below the minimum base-class threshold disproportionately eliminates true, highly long-tailed rare categories. Future work could investigate adaptive thresholding or soft pseudo-label weighting.

Summary updated

Abstract

Open-vocabulary object detection (OVD) seeks to recognize and localize object categories beyond those seen during training. Recent approaches typically leverage vision-language models (VLMs) to generate pseudo-labels using image-text alignment, allowing detectors to generalize to unseen classes without explicit supervision. However, these methods depend heavily on direct image–text matching, neglecting the intermediate reasoning steps essential for interpreting semantically complex scenes. This results in limited robustness when confronted with crowded or occluded visual contexts. In this paper, we introduce CoT-PL, a new framework that employs structured visual chain-of-thought (CoT) reasoning into the pseudo-labeling process. CoT-PL decomposes object understanding into three interpretable steps: (1) region perception even for unseen objects, (2) category recognition via zero-shot reasoning, and (3) background grounding to separate semantically complex objects. Crucially, the third step naturally motivates our contrastive background learning (CBL) that uses the pre-computed background cues as negatives to promote feature disentanglement between objects and background. In this way, CoT reasoning and CBL form an integrated pipeline tailored to robust pseudo-labeling in crowded or occluded scenes. Notably, in these two settings, our novel-class pseudo-label quality achieves relative improvements of 103.4% and 168.4% over the best prior, respectively. Our extensive experiments demonstrate that CoT-PL achieves +7.7 AP50 on open-vocabulary COCO and +2.9 mask AP on LVIS for novel classes, setting a new state of the art.
BibTeX
@misc{choi2025cot,
  author = {Hojun Choi and Youngsun Lim and Jaeyo Shin and Hyunjung Shim},
  title = {{CoT-PL: Visual Chain-of-Thought Reasoning Meets Pseudo-Labeling for Open-Vocabulary Object Detection}},
  year = {2025},
  eprint = {2510.14792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14792},
}
Figure