ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing

Yisu Wang, Yixiang Zhu, Xinjiao Li, Yulong Zhang, Ruilong Wu, Dirk Kutscher

Abstract

ViFusion is a novel communication-aware framework designed to accelerate large-scale video feature indexing. By combining dynamic Tensor Fusion with In-Network Computation, ViFusion addresses the critical bottlenecks of communication overhead and resource underutilization in distributed GPU clusters. The system merges small, asynchronous video queries into consolidated tensors and offloads aggregation tasks to the network fabric. Experimental results demonstrate an 8–22× throughput improvement over state-of-the-art systems and a significant reduction in tail latency while maintaining high retrieval accuracy.

Introduction

The explosion of video-centric digital services has created a massive demand for efficient video retrieval—the ability to find specific moments in vast databases using natural language queries. Modern solutions bridge the "semantic gap" between text and video using Large Language Models (LLMs) and multimodal architectures like CLIP. However, the sheer size of these models necessitates distributed execution across GPU clusters, which introduces two primary bottlenecks:

  1. Communication Overhead: Collective operations like AllReduce (synchronizing data across nodes) become significant bottlenecks, particularly when data must traverse inter-rack links in a datacenter.
  2. Resource Underutilization: Video queries often arrive asynchronously and in small batches. Processing these individually leads to poor GPU utilization as small kernels fail to saturate the hardware's processing capacity.
Overview of the ViFusion system architecture, illustrating the flow from input data through tensor fusion and the LLM backbone to in-networked AllReduce.

ViFusion addresses these challenges by treating the network not just as a data pipe, but as a functional component of the inference pipeline.

Methodology

Tensor Fusion Module (TFM)

The TFM acts as an intelligent buffer that dynamically groups incoming video segments into larger, fused tensors. This approach balances throughput and responsiveness using a dual-threshold triggering mechanism: fusion is initiated when the buffer reaches 75% capacity or a 150ms deadline is met.

To minimize overhead, ViFusion employs zero-copy concatenation via GPUDirect RDMA, moving data directly between GPUs and bypassing the CPU. Furthermore, the framework utilizes multilevel model sharding, partitioning the neural network across GPUs to balance computational load against transfer sizes. An additional optimization involves padding sequences to power-of-two sizes, aligning with GPU memory architectures and libraries like cuBLAS for more efficient kernel execution.

In-Network AllReduce

To mitigate the congestion of inter-rack links, ViFusion implements a hierarchical AllReduce scheme. In standard datacenters, intra-rack bandwidth is abundant, while inter-rack links are often oversubscribed.

The hierarchical communication architecture of ViFusion, showing the interaction between GPU workers, aggregation servers, and the root server.

ViFusion offloads initial data aggregation to "aggregation servers" (middle-host nodes) within a local group. By performing partial reductions locally, the system sends a single consolidated message across the slower inter-rack links, drastically reducing the volume of data traversing the network core and preventing "straggler" nodes from delaying the entire cluster.

Evaluation and Results

The authors evaluated ViFusion using the MSR-VTT (Microsoft Research Video to Text) benchmark and synthetic workloads designed to simulate high-load asynchronous query arrivals following a Poisson distribution.

Performance Gains

  • Throughput: ViFusion achieved an 8–22× throughput improvement compared to hierarchical fusion-based aggregation (HFBA) methods. In micro-benchmarks, it outperformed the standard Gloo library by 37× to 43× for large tensors.
  • Latency: Despite the inherent delays of batching, ViFusion reduced median query latency by 13% and 99th-percentile (tail) latency by 23% due to the drastic reduction in synchronization overhead.
  • Accuracy: On the MSR-VTT dataset, the system maintained a Top-10 mean hit ratio of 81.3%, proving that these efficiency optimizations do not compromise semantic understanding.

Conclusion and Future Work

ViFusion demonstrates that integrating tensor fusion with in-network computation can provide massive scalability for video indexing tasks. By moving the "summation" logic of AI mathematics into the network fabric, the framework effectively bypasses traditional distributed computing bottlenecks.

Future research will focus on scaling ViFusion to clusters with hundreds of nodes to test its resilience against extreme "incast" congestion. Additionally, the authors plan to explore the framework's performance in heterogeneous environments with varying bandwidth and hardware capabilities.

Summary updated

BibTeX
@inproceedings{wang2025vifusion,
  author = {Yisu Wang and Yixiang Zhu and Xinjiao Li and Yulong Zhang and Ruilong Wu and Dirk Kutscher},
  title = {{ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing}},
  booktitle = {ICMR},
  year = {2025},
  doi = {10.1145/3731715.3733461},
}
Figure