LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency

Jiachen Guo, Xinshan Zhu, Boyun Wang, Yanyan Liang, Jun Wan, Feng Lin, Ajian LiuarXiv 2026 · arXiv

TL;DR

LocalAgent is a new framework that fixes a major blind spot in Multimodal Large Language Models (MLLMs): the inability to tell if two images show the exact same object. By using the MLLM as a "spatial controller" to identify suspicious regions and delegating the dirty work to specialized vision tools, LocalAgent achieves a 4.9% accuracy boost while remaining efficient enough to only call for help 12% of the time.

What Problem Does This Paper Address?

Current MLLMs (like GPT-4o or Qwen-VL) are surprisingly bad at "instance-level consistency." While they can easily identify that two images both contain a "white flower vase," they often fail to notice that one vase has a slightly different base or texture than the other.

The authors argue this stems from a semantic bottleneck: the vision encoders in MLLMs are optimized to align images with language. In that process, they prioritize global "concepts" and discard the fine-grained, non-semantic details (like specific scratches or geometric patterns) required to distinguish identical twins from mere lookalikes. As shown in the motivation figure below, standard models often over-rely on global similarity and miss the "smoking gun" local evidence.

Figure 1: The motivation of LocalAgent. Standard MLLMs over-rely on global similarity and fail on instance-level verification in both directions. LocalAgent uses the MLLM as a spatial controller to localize suspicious regions, delegates them to the LEV toolkit (LAC, LGC) for invariant matching, and integrates the returned structured scalar evidence into the final decision.

Methods

The researchers propose LocalAgent, which treats the MLLM as a high-level manager that delegates tasks to a Local Evidence Verifier (LEV) toolkit.

  1. MLLM as Spatial Controller: Instead of just guessing "yes" or "no," the MLLM first reasons globally. If it's unsure, it identifies "suspicious regions" and outputs their coordinates.
  2. The LEV Toolkit: These are lightweight, vision-only specialists that don't care about language.
  3. Local Appearance Consistency (LAC): Uses DINO features to compare textures and colors across multiple views (crops, flips, color-jittered versions).
  4. Local Geometric Correspondence (LGC): Uses XFeat to find matching keypoints, ensuring the physical structure of the object is consistent across different viewpoints.
  5. Structured Scalar Evidence: Crucially, the tools don't send images back to the MLLM. They return a small set of numbers (scores for matching, stability, and coverage). The MLLM then performs "evidential reasoning" on these numbers to reach a final verdict.
  6. Auxiliary-Capped Reinforcement Fine-Tuning (RFT): To teach the model when to call a tool, the authors used a specialized RL strategy. It rewards the model for being correct but "caps" the rewards for using tools, preventing the model from becoming "lazy" and calling tools for easy questions.
Overview of the LocalAgent framework, showing the loop from global reasoning to local verification.

Datasets

The model was evaluated across 12,000 queries from six diverse sources to ensure it wasn't just memorizing specific types of objects:

  • TWIN-Eval: The primary benchmark for fine-grained product verification.
  • ILIAS: Retail products.
  • Landmarks: Geographical sites.
  • MET: Artworks.
  • CUB: Bird species.
  • Inquire: Flora and fauna.

Key Insights and Ideas

  • Bypassing the Bottleneck: By using vision-only encoders (DINO/XFeat) for the local check, the system recovers the fine-grained details that the main MLLM's encoder typically "compresses" away.
  • Numbers over Pixels: Feeding scalar scores back to the MLLM is far more efficient than re-feeding high-resolution crops. It allows the MLLM to act as a "judge" of evidence rather than a raw sensor.
  • Learning Restraint: The RFT strategy is vital. Without it, models either call tools 100% of the time (slow and expensive) or never (inaccurate). LocalAgent learns to reserve tool use for the "hard" cases where global perception fails.

Key Results

  • Accuracy: LocalAgent achieved a mean accuracy of 80.7%, a 4.9% improvement over the strongest baseline (CoT-RFT).
  • Efficiency: It only invoked the LEV toolkit on 12.13% of samples, keeping the average inference latency low (336ms vs. 610ms for an "always-call" policy).
  • Generalization: Unlike many fine-tuned models, LocalAgent maintained its performance on general benchmarks (like MMLU and GSM8K), proving that specialized training didn't "break" its general intelligence.
Qualitative examples where LocalAgent correctly identifies mismatches that global reasoning missed.

Limitations and Future Work

  • Localization Dependency: If the MLLM fails to identify the correct "suspicious region" in the first step, the LEV toolkit is essentially looking in the wrong place.
  • Multiple Choice Bottleneck: The current policy doesn't aggregate evidence across all candidates in a multiple-choice setting simultaneously; it treats them as a series of pairwise checks.
  • Latency: While more efficient than "always-call" methods, it is still ~30% slower than direct inference without any tools.

Future work may involve more sophisticated "multi-turn" verification where the agent can look at multiple regions before deciding.

Summary updated

Abstract

Multimodal large language models (MLLMs) have made rapid progress in general visual understanding, yet they still struggle with fine-grained instance-level consistency verification, where the goal is to determine whether two images depict the exact same physical object under varying viewpoints, illumination, and backgrounds. This task requires both global semantic perception and precise examination of subtle local evidence, since semantically similar objects may differ only in small suspicious regions. To address this challenge, we propose LocalAgent, a collaborative agentic framework that integrates the complementary strengths of a generalist MLLM and lightweight specialist verifiers. The MLLM first reasons globally to identify suspicious local regions, which are then delegated to a Local Evidence Verifier (LEV) toolkit for fine-grained appearance and geometric matching. LEV comprises Local Appearance Consistency (LAC), which estimates appearance similarity under illumination and color changes, and Local Geometric Correspondence (LGC), which estimates spatial correspondence under viewpoint changes. LEV returns structured scalar evidence that the MLLM integrates for the final decision. We further introduce an auxiliary-capped reinforcement fine-tuning (RFT) strategy, in which answer correctness dominates auxiliary tool-use and format signals, enabling the policy to learn when verification is necessary. Experiments show that LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples and preserving the general multimodal capabilities of the underlying MLLM. The open-source repository is available at https://github.com/Firefly-zoo/LocalAgent-main.
BibTeX
@misc{guo2026localagent,
  author = {Jiachen Guo and Xinshan Zhu and Boyun Wang and Yanyan Liang and Jun Wan and Feng Lin and Ajian Liu},
  title = {{LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency}},
  year = {2026},
  eprint = {2610.29174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29174},
}
Figure