COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking

Shukun Jia, Shiyu Hu, Yipei Wang, Ximeng Cheng, Yichao Cao, Xiaobo LuarXiv 2026 · arXiv

TL;DR

COAL (Counterfactual and Observation-enhanced Alignment Learning) addresses the "Sparsity-Discriminability Paradox" in Referring Multi-Object Tracking (RMOT) by leveraging external knowledge from foundation models. By injecting explicit semantic priors from VLMs and generating counterfactual "hard negatives" with LLMs, the framework prevents models from taking shortcuts, leading to a significant 7.28% HOTA improvement on the challenging Refer-KITTI-V2 benchmark.

What Problem Does This Paper Address?

Referring Multi-Object Tracking (RMOT) requires a system to track multiple objects described by natural language (e.g., "the white car on the left"). The authors identify a fundamental structural contradiction they call the Sparsity-Discriminability Paradox.

In RMOT, scenarios are often visually homogeneous (many similar-looking cars) and descriptions use a restricted, repetitive vocabulary. To succeed, a model needs high discriminative power to spot subtle differences. However, the available training data is relatively sparse. This mismatch causes models to suffer from "shortcut learning"—they overfit to easy, salient cues (like just "car") and ignore the fine-grained attributes (like "on the left" or "silver") needed for true compositional understanding. This leads to "feature collapse," where the internal representations of different objects become indistinguishable.

The Sparsity-Discriminability Paradox in RMOT

Methods

The authors propose COAL, a framework that regularizes the learning process using external knowledge. It consists of three main pillars:

  1. Explicit Semantic Injection (ESI): Instead of relying solely on implicit visual features, COAL uses a frozen Vision-Language Model (VLM), specifically DINO-X, to generate dense object proposals and descriptive captions for every frame. This "densifies" the observation space, providing explicit labels for subtle visual cues that the main model might otherwise miss.
  2. Counterfactual Learning (CFL): To prevent shortcut learning, the authors use a Large Language Model (LLM) to generate "counterfactual" queries. If the original query is "white car on the left," the LLM might generate "red car on the left." The model is then trained with a "push" force to ensure it does not match the target object to these high-interference distractors, forcing it to learn attribute disentanglement.
  3. Hierarchical Multi-Stream Integration (HMSI): This is the core architecture that fuses visual features, referring queries, and VLM-generated captions. It uses a Pixel-Word Contextualization module for early cross-modal interaction and a Hierarchical Semantic Refinement module to denoise the potentially "hallucinated" or noisy captions from the VLM using the original referring query as a filter.
Overview of the COAL Framework

Datasets

The framework was evaluated on two primary RMOT benchmarks:

  • Refer-KITTI: The standard benchmark with 18 videos and 818 expressions.
  • Refer-KITTI-V2: A much larger and more complex extension featuring 9,758 diverse expressions across 21 videos. This dataset is designed to be significantly more challenging due to its complex compositional semantics.

Key Insights and Ideas

  • Knowledge as Regularization: The paper shifts the focus from "better fusion architectures" to "better supervision." By using VLMs and LLMs as offline "teachers," they provide the dense supervision needed to overcome data sparsity.
  • The Anchoring and Expansion Effect: Visualization of the feature space (via t-SNE) shows that ESI acts as a semantic "anchor," creating distinct clusters for different objects. Meanwhile, CFL acts as an "expansion" force, preventing these clusters from collapsing into single points and ensuring the model captures the full diversity of the data manifold.
  • Denoising External Priors: The authors acknowledge that foundation models aren't perfect (e.g., VLMs might hallucinate a car's color). Their HMSI module is specifically designed to treat VLM captions as noisy inputs that must be filtered by the user's actual referring query.
Visualization of t-SNE Embeddings

Key Results

COAL achieved state-of-the-art (SOTA) results across both benchmarks:

  • Refer-KITTI-V2: Achieved 43.46% HOTA, a massive 7.28% absolute improvement over the previous best method (HFF-Tracker). The authors note that the performance gap widens on this more difficult dataset, proving COAL's robustness in complex scenarios.
  • Refer-KITTI: Achieved 53.38% HOTA, outperforming the previous SOTA by 0.97%.
  • Ablation Studies: Removing ESI dropped HOTA by ~4%, and removing CFL dropped it by ~3.5% on Refer-KITTI-V2, confirming that both "observation enhancement" and "counterfactual supervision" are critical.

Limitations and Future Work

  • Offline Dependency: The VLM and LLM components are currently used offline to generate priors and counterfactuals. While this avoids online inference overhead, it means the model cannot dynamically generate new counterfactuals or captions for entirely novel, unseen environments during deployment.
  • VLM Noise: While the HMSI module helps, the system is still somewhat dependent on the quality of the VLM's initial detections and descriptions.
  • Future Directions: The authors suggest that moving toward a more tightly integrated, end-to-end trainable system that can leverage these foundation models online without massive computational costs is a promising next step.

Summary updated

Abstract

Referring Multi-Object Tracking (RMOT) faces a fundamental structural contradiction between the high-discriminability demand and the sparse semantic supervision. This mismatch is particularly acute in highly homogeneous scenarios that require fine-grained discrimination over complex compositional semantics. However, under sparse supervision, models overfit to salient yet insufficient cues, thereby encouraging shortcut learning and semantic collapse. To resolve this, we propose COAL (Counterfactual and Observation-enhanced Alignment Learning), a framework that advances RMOT beyond isolated structural optimization through knowledge regularization. First, we introduce Explicit Semantic Injection (ESI) via a VLM to densify the observation space and enhance instance discriminability. Second, leveraging LLM reasoning, we propose Counterfactual Learning (CFL) to augment supervision, enforcing strict attribute verification for robust compositional recognition. These strategies are unified within a Hierarchical Multi-Stream Integration (HMSI) architecture, which distills external knowledge into domain-specific discriminative representations. Experiments on Refer-KITTI and Refer-KITTI-V2 benchmarks validate COAL's efficacy. Notably, it surpasses the state-of-the-art by 7.28% HOTA on the highly challenging Refer-KITTI-V2. These results demonstrate the effectiveness of knowledge regularization for resolving the sparsity-discriminability paradox in RMOT.
BibTeX
@misc{jia2026coal,
  author = {Shukun Jia and Shiyu Hu and Yipei Wang and Ximeng Cheng and Yichao Cao and Xiaobo Lu},
  title = {{COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking}},
  year = {2026},
  eprint = {2605.14795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.14795},
}
Figure