TL;DR
LocalAgent is a new framework that fixes a major blind spot in Multimodal Large Language Models (MLLMs): the inability to tell if two images show the exact same object. By using the MLLM as a "spatial controller" to identify suspicious regions and delegating the dirty work to specialized vision tools, LocalAgent achieves a 4.9% accuracy boost while remaining efficient enough to only call for help 12% of the time.
What Problem Does This Paper Address?
Current MLLMs (like GPT-4o or Qwen-VL) are surprisingly bad at "instance-level consistency." While they can easily identify that two images both contain a "white flower vase," they often fail to notice that one vase has a slightly different base or texture than the other.
The authors argue this stems from a semantic bottleneck: the vision encoders in MLLMs are optimized to align images with language. In that process, they prioritize global "concepts" and discard the fine-grained, non-semantic details (like specific scratches or geometric patterns) required to distinguish identical twins from mere lookalikes. As shown in the motivation figure below, standard models often over-rely on global similarity and miss the "smoking gun" local evidence.
Methods
The researchers propose LocalAgent, which treats the MLLM as a high-level manager that delegates tasks to a Local Evidence Verifier (LEV) toolkit.
- MLLM as Spatial Controller: Instead of just guessing "yes" or "no," the MLLM first reasons globally. If it's unsure, it identifies "suspicious regions" and outputs their coordinates.
- The LEV Toolkit: These are lightweight, vision-only specialists that don't care about language.
- Local Appearance Consistency (LAC): Uses DINO features to compare textures and colors across multiple views (crops, flips, color-jittered versions).
- Local Geometric Correspondence (LGC): Uses XFeat to find matching keypoints, ensuring the physical structure of the object is consistent across different viewpoints.
- Structured Scalar Evidence: Crucially, the tools don't send images back to the MLLM. They return a small set of numbers (scores for matching, stability, and coverage). The MLLM then performs "evidential reasoning" on these numbers to reach a final verdict.
- Auxiliary-Capped Reinforcement Fine-Tuning (RFT): To teach the model when to call a tool, the authors used a specialized RL strategy. It rewards the model for being correct but "caps" the rewards for using tools, preventing the model from becoming "lazy" and calling tools for easy questions.
Datasets
The model was evaluated across 12,000 queries from six diverse sources to ensure it wasn't just memorizing specific types of objects:
- TWIN-Eval: The primary benchmark for fine-grained product verification.
- ILIAS: Retail products.
- Landmarks: Geographical sites.
- MET: Artworks.
- CUB: Bird species.
- Inquire: Flora and fauna.
Key Insights and Ideas
- Bypassing the Bottleneck: By using vision-only encoders (DINO/XFeat) for the local check, the system recovers the fine-grained details that the main MLLM's encoder typically "compresses" away.
- Numbers over Pixels: Feeding scalar scores back to the MLLM is far more efficient than re-feeding high-resolution crops. It allows the MLLM to act as a "judge" of evidence rather than a raw sensor.
- Learning Restraint: The RFT strategy is vital. Without it, models either call tools 100% of the time (slow and expensive) or never (inaccurate). LocalAgent learns to reserve tool use for the "hard" cases where global perception fails.
Key Results
- Accuracy: LocalAgent achieved a mean accuracy of 80.7%, a 4.9% improvement over the strongest baseline (CoT-RFT).
- Efficiency: It only invoked the LEV toolkit on 12.13% of samples, keeping the average inference latency low (336ms vs. 610ms for an "always-call" policy).
- Generalization: Unlike many fine-tuned models, LocalAgent maintained its performance on general benchmarks (like MMLU and GSM8K), proving that specialized training didn't "break" its general intelligence.
Limitations and Future Work
- Localization Dependency: If the MLLM fails to identify the correct "suspicious region" in the first step, the LEV toolkit is essentially looking in the wrong place.
- Multiple Choice Bottleneck: The current policy doesn't aggregate evidence across all candidates in a multiple-choice setting simultaneously; it treats them as a series of pairwise checks.
- Latency: While more efficient than "always-call" methods, it is still ~30% slower than direct inference without any tools.
Future work may involve more sophisticated "multi-turn" verification where the agent can look at multiple regions before deciding.
Summary updated