MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning

Tuan-An To, Yuk-Kwan Wong, Tuan-Anh Vu, Ziqiang Zheng, Sai-Kit YeungarXiv 2026 · arXiv

TL;DR

MarineEVT introduces the first comprehensive, event-centric marine video dataset featuring 20K multi-task video question-answering pairs to address the sparsity of underwater ecological data. Alongside it, the authors propose EVT-R1, a multi-turn reinforcement learning framework that trains Vision-Language Models (VLMs) to strategically call spatial and temporal localization tools, significantly outperforming state-of-the-art models in parsing complex marine dynamics.

What Problem Does This Paper Address?

While Vision-Language Models (VLMs) excel at static image analysis and everyday video summarization, they struggle in specialized domains like marine biology. Underwater footage is notoriously long, tedious, and visually redundant. Crucial ecological events—such as a predator-prey interaction or a rare species appearance—are typically sparse, ephemeral, and unevenly distributed across hours of static background.

Current general-purpose VLMs lack the domain expertise to recognize these fleeting moments and often fail to localize the specific visual tokens that carry scientific meaning. Instead of capturing the ecological significance of an event, they default to generic scene descriptions or rely too heavily on language priors, completely missing subtle behaviors like a reef fish defending its territory.

Fig. 2: An example question for evaluating event summarization tested on GPT-5.0 [53] with human response provided for comparison.

Methods

To enable models to focus on critical events while discarding irrelevant visual noise, the authors developed EVT-R1 (Event-centric Visual Tool-integrated Reasoning).

Instead of forcing a VLM to process an entire redundant video at once, EVT-R1 decomposes video understanding into a multi-turn sequential decision-making process:

  1. Multi-Turn Tool Invocation: At each reasoning turn, the VLM decides whether it needs to invoke external visual tools (such as SAM3 for spatial grounding or specialized toolkits for temporal localization).
  2. Dynamic Visual Updating: When a tool is called, it modifies the visual input (e.g., cropping a specific region or isolating a precise timeframe). This replaces the outdated, redundant video tokens in the dialogue history with condensed, highly relevant visual cues.
  3. Dual-Reward Reinforcement Learning: The framework optimizes this policy using a modified version of Group Relative Policy Optimization (GRPO). Rather than only rewarding the final answer, EVT-R1 implements a dual-reward system: a tool-reasoning reward ($R_{tool}$) that evaluates the validity and accuracy of tool calls at each turn, and a multi-task answer reward ($R_{ans}$) evaluating format compliance and semantic correctness.
Fig. 1: We propose MarineEVT, the first hierarchical and comprehensive event-centric marine video dataset. Based on MarineEVT, we propose EVT-R1, integrating visual tool reasoning into the VLM for more reliable marine video understanding.

Datasets

The paper introduces MarineEVT, a newly curated dataset specifically designed for event-level underwater video understanding.

  • The Data: It consists of 20,000 richly annotated video question-answer pairs spanning 20 fine-grained dimensions, including species identification, human-species causal dynamics, and environmental conditions.
  • The Pipeline: To build it, the authors segmented 7,300 raw marine videos into 97,284 scenes. They used GPT-5 to generate domain-specific scene descriptions, applied grounding toolkits (SAM3, DepthAnythingV2, OrientAnything) to extract 94,028 object-grounding annotations, and utilized QwenVL-Max to generate QA pairs. The entire dataset underwent a rigorous two-tier verification process involving automated LLM filtering and validation by expert human annotators.
Fig. 3: We propose a data construction pipeline that systematically transforms pub- licly available marine videos into high-quality, human-verified annotations, ensuring reliability and versatility for diverse VLM training tasks.

Key Insights and Ideas

  • Active Tool Use Beats Passive Compression: Standard video VLMs rely on passive token compression algorithms to handle long sequences, which frequently discards sparse, critical information. EVT-R1 proves that letting the model actively choose where and when to look acts as a superior, task-driven compression mechanism.
  • SFT is Essential Before RL: The authors found that applying reinforcement learning (GRPO) directly to a base model for post-training caused instability, reward overfitting, and infinite reasoning loops. A brief Supervised Fine-Tuning (SFT) phase on the domain data is a strict prerequisite, giving the model the baseline domain knowledge required to make logical tool-use decisions.
  • Language Priors Dominate in the Deep: Ablation tests using blank or adversarially inverted videos revealed that even advanced commercial models like GPT-5-Mini rely dangerously on language biases when confronted with complex marine queries, underscoring the necessity of explicit visual grounding.

Key Results

  • State-of-the-Art Performance: EVT-R1 achieved an overall accuracy of 48.89% on the MarineEVT benchmark, outperforming the top open-source baseline (Qwen3-VL-8B-SFT) by 3.62 points and standard GRPO post-training by 13.31 points.
  • Beating Commercial Models: EVT-R1 surpassed the best closed-source model evaluated (GPT-5-Mini with tool invocation) by +8.54 points.
  • Outperforming Compression Baselines: When compared against state-of-the-art token compression methods like VisionZip and PVC, EVT-R1 demonstrated a massive performance leap of +18.66 points, validating the multi-turn localization approach.

Limitations and Future Work

While highly effective, EVT-R1's multi-turn reasoning loop introduces noticeable computational overhead and increased inference latency due to sequential tool execution. Furthermore, the system's performance is inherently bounded by the accuracy of its underlying visual toolkits (e.g., if SAM3 fails to segment a camouflaged marine animal, the VLM receives flawed visual cues).

Future work will focus on optimizing the inference speed of multi-turn tool reasoning and expanding the toolbox to include more diverse capabilities, such as motion tracking and underwater acoustic signal processing.

Summary updated

Abstract

Recent Vision-Language Models (VLMs) have achieved remarkable success in visual understanding, driven by the growing availability of high-quality image-text pairs. However, the performance of VLMs often degrades in the video domain due to the essential need for temporal understanding and the scarcity of large-scale annotated video data. In this work, we focus on marine video understanding, which brings further challenges: first, it requires substantial domain expertise; and video VLMs usually struggle with localizing and interpreting critical information from marine videos, as the informative events are typically sparse, unpredictable, and unevenly distributed. To address these challenges, we carefully curate the first event-centric marine video understanding dataset called MarineEVT, which features 20K multi-task, video-level visual question-answering pairs spanning multiple dimensions of marine understanding and analysis. Meanwhile, based on MarineEVT, we decompose marine video understanding as an Event-centric Visual Tool-integrated Reasoning process EVT-R1 for short, where we leverage powerful visual tools to drive the model to localize and interpret critical information aligned with visual questions and human intent. To demonstrate its effectiveness, we compare EVT-R1 against 11 SOTA VLMs in different settings. EVT-R1 outperforms the top open-source and top commercial models by 5.22 and 11.09, respectively. MarineEVT and EVT-R1 lay the foundation for ecological discovery and marine education, fostering the development of VLMs capable of interpreting marine dynamics, reasoning about ecological interactions, and supporting sustainable ocean video understanding and analysis.
BibTeX
@misc{to2026marineevt,
  author = {Tuan-An To and Yuk-Kwan Wong and Tuan-Anh Vu and Ziqiang Zheng and Sai-Kit Yeung},
  title = {{MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning}},
  year = {2026},
  eprint = {2607.24064},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.24064},
}
Figure