TL;DR
MarineEVT introduces the first comprehensive, event-centric marine video dataset featuring 20K multi-task video question-answering pairs to address the sparsity of underwater ecological data. Alongside it, the authors propose EVT-R1, a multi-turn reinforcement learning framework that trains Vision-Language Models (VLMs) to strategically call spatial and temporal localization tools, significantly outperforming state-of-the-art models in parsing complex marine dynamics.
What Problem Does This Paper Address?
While Vision-Language Models (VLMs) excel at static image analysis and everyday video summarization, they struggle in specialized domains like marine biology. Underwater footage is notoriously long, tedious, and visually redundant. Crucial ecological events—such as a predator-prey interaction or a rare species appearance—are typically sparse, ephemeral, and unevenly distributed across hours of static background.
Current general-purpose VLMs lack the domain expertise to recognize these fleeting moments and often fail to localize the specific visual tokens that carry scientific meaning. Instead of capturing the ecological significance of an event, they default to generic scene descriptions or rely too heavily on language priors, completely missing subtle behaviors like a reef fish defending its territory.
Methods
To enable models to focus on critical events while discarding irrelevant visual noise, the authors developed EVT-R1 (Event-centric Visual Tool-integrated Reasoning).
Instead of forcing a VLM to process an entire redundant video at once, EVT-R1 decomposes video understanding into a multi-turn sequential decision-making process:
- Multi-Turn Tool Invocation: At each reasoning turn, the VLM decides whether it needs to invoke external visual tools (such as SAM3 for spatial grounding or specialized toolkits for temporal localization).
- Dynamic Visual Updating: When a tool is called, it modifies the visual input (e.g., cropping a specific region or isolating a precise timeframe). This replaces the outdated, redundant video tokens in the dialogue history with condensed, highly relevant visual cues.
- Dual-Reward Reinforcement Learning: The framework optimizes this policy using a modified version of Group Relative Policy Optimization (GRPO). Rather than only rewarding the final answer, EVT-R1 implements a dual-reward system: a tool-reasoning reward ($R_{tool}$) that evaluates the validity and accuracy of tool calls at each turn, and a multi-task answer reward ($R_{ans}$) evaluating format compliance and semantic correctness.
Datasets
The paper introduces MarineEVT, a newly curated dataset specifically designed for event-level underwater video understanding.
- The Data: It consists of 20,000 richly annotated video question-answer pairs spanning 20 fine-grained dimensions, including species identification, human-species causal dynamics, and environmental conditions.
- The Pipeline: To build it, the authors segmented 7,300 raw marine videos into 97,284 scenes. They used GPT-5 to generate domain-specific scene descriptions, applied grounding toolkits (SAM3, DepthAnythingV2, OrientAnything) to extract 94,028 object-grounding annotations, and utilized QwenVL-Max to generate QA pairs. The entire dataset underwent a rigorous two-tier verification process involving automated LLM filtering and validation by expert human annotators.
Key Insights and Ideas
- Active Tool Use Beats Passive Compression: Standard video VLMs rely on passive token compression algorithms to handle long sequences, which frequently discards sparse, critical information. EVT-R1 proves that letting the model actively choose where and when to look acts as a superior, task-driven compression mechanism.
- SFT is Essential Before RL: The authors found that applying reinforcement learning (GRPO) directly to a base model for post-training caused instability, reward overfitting, and infinite reasoning loops. A brief Supervised Fine-Tuning (SFT) phase on the domain data is a strict prerequisite, giving the model the baseline domain knowledge required to make logical tool-use decisions.
- Language Priors Dominate in the Deep: Ablation tests using blank or adversarially inverted videos revealed that even advanced commercial models like GPT-5-Mini rely dangerously on language biases when confronted with complex marine queries, underscoring the necessity of explicit visual grounding.
Key Results
- State-of-the-Art Performance: EVT-R1 achieved an overall accuracy of 48.89% on the MarineEVT benchmark, outperforming the top open-source baseline (Qwen3-VL-8B-SFT) by 3.62 points and standard GRPO post-training by 13.31 points.
- Beating Commercial Models: EVT-R1 surpassed the best closed-source model evaluated (GPT-5-Mini with tool invocation) by +8.54 points.
- Outperforming Compression Baselines: When compared against state-of-the-art token compression methods like VisionZip and PVC, EVT-R1 demonstrated a massive performance leap of +18.66 points, validating the multi-turn localization approach.
Limitations and Future Work
While highly effective, EVT-R1's multi-turn reasoning loop introduces noticeable computational overhead and increased inference latency due to sequential tool execution. Furthermore, the system's performance is inherently bounded by the accuracy of its underlying visual toolkits (e.g., if SAM3 fails to segment a camouflaged marine animal, the VLM receives flawed visual cues).
Future work will focus on optimizing the inference speed of multi-turn tool reasoning and expanding the toolbox to include more diverse capabilities, such as motion tracking and underwater acoustic signal processing.
Summary updated