ABRR: Transforming Temporal Sentence Grounding with Wasserstein-Adaptive Refinement and Precision Ranking

Xiang Fang, Wanlong Fang, Junhao Dong, Daizong LiuarXiv 2026 · arXiv

TL;DR

ABRR (Adaptive Boundary Refinement and Ranking) is a novel, model-agnostic post-processing framework that significantly improves the precision of Temporal Sentence Grounding (TSG) without requiring model retraining. By leveraging optimal transport theory to refine segment boundaries and a new quality-driven ranking strategy, ABRR corrects quantization errors and ensures that the most accurately localized video moments rise to the top.

What Problem Does This Paper Address?

Temporal Sentence Grounding (TSG) is the task of pinpointing the exact start and end times of a specific event in a video based on a text query (e.g., "the man begins laying plaster on the floor"). While modern transformer-based models are good at finding the general area of an event, they struggle with two major issues:

  1. Quantization Errors: To keep computation manageable, videos are often downsampled into coarse "snippets" (e.g., 1-second chunks). This means a predicted boundary might be "snapped" to a snippet edge, missing the true start or end of an action by a significant margin.
  2. Suboptimal Ranking: Most models rank their predictions based on semantic relevance—how well the video content matches the text. However, a segment might have a high semantic match but very sloppy boundaries. Current systems often rank these "imprecise but relevant" segments higher than "precise and relevant" ones.
The difference between coarse outputs from standard models and the refined, precise outputs from ABRR.

Methods

The authors propose ABRR, a three-stage pipeline that acts as a "plug-and-play" module for any existing TSG model.

  1. Temporal Distribution Smoothing: Predicted score distributions are often noisy. ABRR applies a Gaussian kernel convolution to smooth these scores, creating a cleaner signal for boundary refinement while preserving the core semantic information.
  2. Boundary Refinement via Optimal Transport: This is the mathematical heart of the paper. The authors treat boundary refinement as an optimization problem. They use the 2-Wasserstein distance to align the predicted (and now smoothed) evidence distribution with an ideal, boundary-centered target distribution. By using Sinkhorn iterations, they can efficiently calculate the "barycenter" of the transported mass, resulting in refined, continuous-valued timestamps that break free from the coarse snippet grid.
  3. Quality-Driven Re-Ranking: ABRR introduces a new quality score ($q_m$) that balances two factors:
  4. Temporal Precision: Derived from the Wasserstein distance (lower distance = higher precision).
  5. Semantic Relevance: Calculated using the cosine similarity between video features and the text query embedding.
Fig. 2: Overview of the ABRR framework. The pipeline includes temporal distribution smoothing, boundary refinement using optimal transport, and quality-driven re-ranking. Best viewed in color.

Datasets

The framework was evaluated on three major TSG benchmarks:

  • Charades-STA: Focused on daily indoor activities with an average video length of 30 seconds.
  • ActivityNet Captions: A large-scale dataset featuring diverse human activities (sports, cooking, etc.) with longer videos averaging 120 seconds.
  • QVHighlights: A recent benchmark for open-domain videos that includes both moment retrieval and highlight detection annotations.

Key Insights and Ideas

  • Model-Agnosticism is a Feature: Because ABRR is a post-processing step, it can be applied to any model (2D-TAN, VSLNet, UniVTG, etc.) without needing the original training data or expensive GPU retraining.
  • Geometry Matters: Using the Wasserstein distance is superior to standard heuristic shifts because it accounts for the temporal distance between snippets. It provides a geometrically meaningful way to "pull" boundaries toward the strongest evidence.
  • Precision as a Ranking Signal: The paper demonstrates that semantic confidence alone is a poor proxy for localization quality. Incorporating a precision metric into the final ranking significantly boosts performance, especially at strict evaluation thresholds.

Key Results

ABRR consistently improved the performance of every base model tested. The gains were most dramatic at high Intersection over Union (IoU) thresholds, where precision is most critical:

  • UniVTG + ABRR: On Charades-STA, it achieved a 34.2% Recall@[email protected], a significant jump from the baseline's 29.4% (+4.8%).
  • ActivityNet Captions: Improvements were consistent even on longer videos, with UniVTG seeing a +4.6% boost in Recall@[email protected].
  • Efficiency: Despite the advanced math, ABRR is lightweight. It adds only about 0.1 seconds of latency per video, making it viable for real-time applications.
TABLE I: Performance comparison on Charades-STA and ActivityNet Captions, averaged over 3 runs with standard deviations. Bold indicates the best results, and ↑denotes improvement with ABRR.

Limitations and Future Work

  • Base Model Dependency: ABRR is a refiner, not a miracle worker. If the base model completely misses the relevant event, ABRR has no "evidence" to refine.
  • Repeated Actions: In cases where a video contains multiple similar actions (e.g., someone clapping several times), the smoothing process might merge nearby peaks, potentially shifting the boundary to an incorrect intermediate point.
  • Future Directions: The authors suggest extending ABRR to other tasks like Video Question Answering (VideoQA) and exploring adaptive hyperparameter tuning to automatically adjust the smoothing and ranking weights for different types of video content.

Summary updated

Abstract

Temporal Sentence Grounding (TSG) aims to localize temporal segments in untrimmed videos that correspond to natural language queries, a critical task for video understanding with applications in video retrieval, summarization, and human-computer interaction. Despite significant progress driven by transformer-based models, existing TSG methods suffer from two key limitations: quantization errors due to temporal downsampling, which misalign segment boundaries, and suboptimal ranking of predictions based on semantic matching rather than localization precision. To address these challenges, we propose the Adaptive Boundary Refinement and Ranking (ABRR) framework, a novel, model-agnostic post-processing approach that enhances the performance of any TSG model without retraining. ABRR integrates three innovative components: (1) Temporal Distribution Smoothing, which mitigates noise in predicted score distributions using Gaussian kernel convolution; (2) Boundary Refinement via Optimal Transport, which leverages the 2-Wasserstein distance and Sinkhorn iterations to align coarse segment boundaries with true temporal moments, grounded in Monge-Kantorovich optimal transport theory; and (3) Quality-Driven Re-Ranking, which reorders predictions using a composite score balancing localization precision and semantic relevance. By correcting quantization errors and prioritizing accurately localized segments, ABRR achieves substantial improvements over state-of-the-art TSG models. Extensive experiments on benchmark datasets, including Charades-STA, ActivityNet Captions, and QVHighlights, demonstrate consistent gains at stringent IoU thresholds (e.g., IoU=0.7), alongside enhanced ranking performance. The framework's computational efficiency and compatibility with existing models make it a versatile and impactful solution for real-world TSG applications.
BibTeX
@misc{fang2026abrr,
  author = {Xiang Fang and Wanlong Fang and Junhao Dong and Daizong Liu},
  title = {{ABRR: Transforming Temporal Sentence Grounding with Wasserstein-Adaptive Refinement and Precision Ranking}},
  year = {2026},
  eprint = {2611.13310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13310},
}
Figure