Multimodal Large Language Models (MLLMs) are rapidly evolving from passive observers into active agents that can use external tools like web search, code interpreters, or image croppers to solve complex tasks. But what happens when a new, state-of-the-art specialist model—say, a better OCR engine or a 3D depth estimator—becomes available? Traditionally, you'd have to choose between expensive full retraining or "naive" fine-tuning that often causes the agent to forget how to use its original tools.
A new paper, "Continual Tool-Space Extension Learning for Multimodal Agents," introduces COTEX, a framework designed to add new tools to an agent's belt efficiently and without the "catastrophic forgetting" that plagues standard fine-tuning.
TL;DR
COTEX allows multimodal agents to acquire new specialist tools by learning low-rank adapters that are mathematically constrained to stay "out of the way" of existing tool-selection logic. By isolating a shared "tool-call subspace" and enforcing orthogonality, it enables agents to gain new expert capabilities with virtually zero performance loss on their original toolset.
What Problem Does This Paper Address?
The toolset for an AI agent is rarely fixed. As better specialist models emerge, we want to integrate them. However, integrating new tools into a pre-trained agent is a balancing act.
- Full Retraining: Preserves old skills but is prohibitively expensive as the number of tools grows.
- Naive Fine-tuning (PEFT/LoRA): Is efficient but often overwrites the parameter directions the agent uses for existing tools. This leads to "catastrophic forgetting," where an agent might learn to use a new depth estimator but suddenly lose its ability to use its original cropping tool.
Methods
The core of COTEX is the observation that tool use involves two distinct decisions: initiating a tool call (realizing that the current visual evidence is insufficient) and selecting which specific tool to use.
- Subspace Isolation: The authors use Singular Value Decomposition (SVD) on the weight changes from the agent's initial training to identify a "tool-call subspace." They found that tool-related sensitivity is concentrated in a very low-rank portion of the model's parameters.
- Orthogonal Adaptation: Within this subspace, they identify directions sensitive to the selection of existing tools and mark them as "protected." When a new tool is added, its adapter is learned only in directions that are orthogonal to these protected ones. This ensures the new tool's logic doesn't interfere with the old tools' selection process.
- Self-Verified Trajectories: Training agents to use tools usually requires expensive "teacher" models like GPT-4 to generate demonstrations. COTEX bypasses this by repurposing existing specialist datasets (e.g., OCR labels). The agent is prompted to solve a task using the new tool; if its final answer matches the ground truth label, that successful "trajectory" is used for training.
Datasets
The researchers evaluated COTEX by extending two representative agents (DeepEyesV2 and Metis) with three specialist tools: OCR (SVTRv2), Object Detection (GroundingDINO), and Depth Estimation (DepthAnything).
- Training Data: Generated from ICDAR 2015 (OCR), Ref-Adv-s (Detection), and UniQA-3D (Depth).
- Evaluation Benchmarks: A battery of 10 benchmarks, including:
- Visual Perception: V*Bench and HRBench.
- Text-Rich Understanding: SEED-Bench-2-Plus, TextVQA, and OCRBench.
- Spatial Reasoning: TreeBench, MultihopSpatial, and MC-Bench.
- Depth Perception: BLINK and HardBLINK.
Key Insights and Ideas
- Hierarchical Logic is Key: The "aha moment" is treating tool-calling as a shared skill and tool-selection as a specific one. By sharing the "trigger" parameters, the agent naturally knows when to use a new tool based on its general experience.
- Parameter Efficiency: At a rank of just 64 (less than 2% of the full rank), the identified subspace retains up to 97% of the tool-call gradient energy. This proves that agentic behavior is highly localized in the model's weight space.
- No Teacher Needed: The self-verification protocol proves that we don't need "frontier" models to teach smaller agents how to use tools. If you have a verifier (like a ground-truth label), the agent can teach itself through trial and error.
Key Results
- Massive Gains on New Tasks: On the BLINK depth perception benchmark, the agent's accuracy jumped from 41.1% to 87.1% after adding the depth tool.
- Preservation of Old Skills: On V*Bench, which tests the agent's original cropping tool, performance actually improved slightly from 81.7 to 83.3. Across all benchmarks, the drop in performance for existing tools was at most 0.3 points.
- Spatial Reasoning Boost: Interestingly, adding the Detection tool led to a 12.2-point gain on MC-Bench, showing that specialist tools can resolve challenging spatial samples that generalist agents typically fail.
Limitations and Future Work
- Scaling Limits: The paper demonstrates adding three tools sequentially. It remains to be seen how many tools can be added before the "orthogonal space" within the tool-call subspace becomes saturated.
- Verifier Dependency: The training protocol relies on having a clear, rule-based verifier for tool outputs. This works well for perception tasks (OCR, detection) but might be harder to implement for more abstract or creative tools.
- Order Sensitivity: The authors do not deeply explore whether the order in which tools are added affects the final performance of the "protected" subspace.
Summary updated