Video Relationship Detection Using Mixture of Experts

Ala Shaabana, Zahra Gharaee, Paul W. Fieguth

TL;DR

Researchers from the University of Waterloo have introduced MoE-VRD, a novel framework that applies a "Mixture of Experts" (MoE) architecture to the complex task of detecting relationships in videos. By replacing a single, monolithic neural network with a sparsely-gated committee of smaller models, they achieved state-of-the-art performance on benchmark datasets while maintaining computational efficiency.

What Problem Does This Paper Address?

Visual Relationship Detection (VRD) is the task of identifying triplets in the form of <subject, predicate, object>—for example, "person riding bicycle" or "dog chasing ball." While this is challenging enough in static images, Video VRD (VidVRD) adds a layer of temporal complexity: relationships change over time, motion blur occurs, and objects frequently occlude one another.

The authors identify two primary bottlenecks in current approaches:

  1. The Inference Gap: There is a massive computational hurdle in connecting raw pixels to the nuanced language of actions (predicates).
  2. The Monolithic Struggle: Standard "monolithic" neural networks (single, massive models) often lack the stability and generalization power needed to handle the combinatorial explosion of possible subject-predicate-object combinations. As these models grow to capture more detail, their computational cost scales quadratically, making them difficult to train and deploy.

Methods

The core innovation is the MoE-VRD architecture. Instead of one giant network trying to learn everything, the system uses a Sparsely-Gated Mixture of Experts layer.

  • The Experts: The authors use the existing VidVRD-II model as the base architecture for their experts. Each expert is a relatively small, "nimble" model capable of processing visual features (from CNNs) and relative positional features (spatio-temporal encodings of bounding boxes).
  • The Gating Network: This is the "manager" of the committee. For any given input, the gating network calculates which experts are best suited to handle it.
  • Sparse Gating: Crucially, the system only activates the "Top-K" experts for any single input. This allows the total model capacity (number of parameters) to be very large, while the actual computation performed for a single frame remains low.
  • Iterative Inference: The model doesn't just guess once; it uses an iterative process to refine its understanding of how the subject, predicate, and object relate to one another, leveraging the inter-dependency of these components.

Datasets

The researchers evaluated MoE-VRD on two major benchmarks:

  • ImageNet-VidVRD: The first dataset specifically for video visual relation detection, containing 1,000 manually annotated videos.
  • VidOR (Video Object Relation): A much larger, more recent benchmark containing 10,000 social media videos from YFCC-100M, offering a more diverse and "in-the-wild" challenge.

Key Insights and Ideas

  • The Power of Two: Through ablation studies, the authors found that selecting the top two experts ($K=2$) provided the best performance. Increasing $K$ beyond this actually caused performance to drop.
  • The "Drowning Out" Effect: The authors hypothesize that when too many experts are involved ($K > 2$), the high-performing experts' signals get "drowned out" by the average or inferior outputs of other experts during the aggregation phase.
  • Efficiency via Sparsity: MoE-VRD demonstrates that we can increase a network's "knowledge capacity" (parameters) without a linear increase in the power required to run a forward pass. This is a vital step toward running complex video understanding models on edge devices.
  • Divide and Conquer: The MoE approach naturally handles the "long-tail" problem of rare relationships by allowing different experts to potentially specialize in different types of visual data or predicates.

Key Results

MoE-VRD set a new bar for performance on the ImageNet-VidVRD dataset:

  • Mean Average Precision (mAP): Achieved 33.02%, significantly outperforming the previous state-of-the-art (VidVRD-II) at 29.37%.
  • Relation Tagging: In terms of simply identifying the correct triplet (without worrying about perfect spatial localization), the model reached a 74.12% Precision@1, a notable jump from previous methods.
  • Consistency: The MoE approach showed superior performance across almost every metric (Recall@50, Recall@100, etc.) compared to ten other representative SOTA methods.

Limitations and Future Work

While the results are impressive, the authors note a few areas for improvement:

  • Identical Experts: Currently, all experts share the same architecture. The authors suggest that heterogeneous experts (experts with different internal structures) might capture even more diverse patterns in the data.
  • Simple Gating: The gating network used is a simple single-layer feed-forward network. A more sophisticated, hierarchical gating mechanism could potentially make better "management" decisions about which expert to call upon.
  • Task Division: Future iterations could explicitly train specific experts for sub-problems, such as one expert specializing in "manner verbs" (how an action is done) and another in "result verbs" (the outcome of an action).

Summary updated

BibTeX
@article{shaabana2023video,
  author = {Ala Shaabana and Zahra Gharaee and Paul W. Fieguth},
  title = {{Video Relationship Detection Using Mixture of Experts}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3257280},
}
Figure