FlyVQA: Seeing Video Frame Interpolation Quality through Fly Eyes

Baoliang Chen, Lingyu Zhu, Zhiwei Zhong, Shiqi WangarXiv 2026 · arXiv

TL;DR

FlyVQA is a novel video quality assessment framework that leverages a frozen, biologically-constrained model of the fruit fly's visual system to evaluate frame-interpolated videos. By analyzing discrepancies in motion-sensitive neural pathways—specifically excitatory-inhibitory dynamics and temporal coupling—it achieves superior alignment with human perception compared to standard CNN, RNN, and Transformer baselines.

What Problem Does This Paper Address?

Video Frame Interpolation (VFI) is widely used to increase frame rates, but it often introduces subtle artifacts like ghosting, blur, or "jittery" motion. Traditional metrics like PSNR or SSIM often fail here because a single interpolated frame might look perfect in isolation while destroying the temporal consistency of the sequence.

The "holy grail" of Visual Quality Assessment (VQA) is to replicate human neural computation. However, the human visual system is too complex and incompletely mapped to model directly. This paper tackles the challenge by turning to a more tractable biological system: the fruit fly (Drosophila melanogaster). The fly's visual connectome is well-mapped, particularly its motion-sensitive pathways, providing a unique, anatomically grounded prior for assessing temporal distortions.

Seeing temporal quality through a fly visual circuit.

Methods

The authors introduce FlyVQA, which uses a shared, frozen FlyVis circuit—a recurrent neural network constrained by the actual synaptic connections found in the fly's brain.

The pipeline works as follows:

  1. Compound-Eye Sampling: Reference and distorted video sequences are mapped onto a hexagonal lattice, mimicking the fly's ommatidia.
  2. Neural Encoding: Both sequences pass through the frozen FlyVis circuit. This circuit explicitly models cell types like T4 (ON-pathway) and T5 (OFF-pathway) neurons.
  3. Discrepancy Characterization: Instead of just comparing pixel values, FlyVQA looks at three levels of neural divergence:
  4. Retinotopic Activity: Differences in the final output of the neurons.
  5. Excitatory–Inhibitory (E/I) Drive: Discrepancies in the underlying synaptic inputs that "push" and "pull" a neuron's state.
  6. Lagged Temporal Coupling: How the timing and coordination between excitation and inhibition are disrupted by interpolation artifacts.
  7. Quality Prediction: These discrepancies are fed into a lightweight MLP regression head.
  8. Reference-Guided Aggregation: A temporal pooling mechanism weights segments with stronger reference motion more heavily, as these are where interpolation errors are most perceptually salient.
Biological organization underlying the quality representation.

Datasets

The model was primarily trained and evaluated on:

  • BVI-VFI: A subjective benchmark containing 108 reference and 540 distorted videos across various interpolation methods, frame rates (30 to 120 Hz), and resolutions.
  • CGVQD: Used for cross-dataset evaluation to test the model's generalization to heterogeneous distortions beyond just VFI.

Key Insights and Ideas

  • Biological Priors Beat Generic Capacity: One of the most striking findings is that a frozen fly circuit with random synaptic strengths (but fixed connectivity) can outperform scratch-trained RNNs and GRUs. This suggests that the structure of the fly's motion circuit provides a powerful inductive bias for temporal quality.
  • E/I Dynamics are Essential: The paper demonstrates that distortions often change the internal "drive" of a circuit even when the final output activity looks similar. By monitoring the balance of excitation and inhibition, FlyVQA captures "hidden" signatures of motion inconsistency.
  • Temporal Coordination Matters: Interpolation doesn't just change signal magnitude; it shifts the timing of neural responses. The "Lagged Temporal Coupling" feature specifically targets these phase shifts, which are critical for smooth motion perception.
Figure 3: Connectome-guided quality assessment of frame-interpolated videos. Reference and distorted sequences undergo identical compound-eye sampling and independent processing by a shared, frozen FlyVis circuit. Paired activity, signed synaptic-drive, and lagged excitatory–inhibitory discrepancies form a 1504-dimensional window descriptor. A shared MLP predicts window quality before reference-response weights pool the scores over time.

Key Results

FlyVQA demonstrates state-of-the-art performance for VFI quality assessment:

  • Superior Correlation: On the BVI-VFI dataset, FlyVQA achieved a Spearman rank correlation (SRCC) of 0.733, significantly higher than the best-performing retinal-input RNN baseline (0.678).
  • Generalization: When tested on unseen interpolation methods, FlyVQA maintained a high mean SRCC of 0.613, outperforming all fixed metrics (like LPIPS or DISTS) and learned retinal baselines.
  • Efficiency: The model achieves these results with a very small quality-supervised parameter budget (~18k parameters in the readout head), highlighting the efficiency of using a pre-configured biological backbone.

Limitations and Future Work

  • Domain Specialization: While FlyVQA excels at motion-related artifacts, its performance drops on general datasets like CGVQD (SRCC 0.634). This suggests the frozen motion circuit is highly specialized and may need to be combined with learnable spatial features to handle static distortions like noise or compression.
  • Resolution and Color: The fly's eye has lower spatial resolution and lacks the complex color processing of human vision. Future work could involve integrating more sophisticated retinal sampling models or human-like chromatic pathways.
  • Frozen vs. Fine-tuned: The authors kept the fly circuit frozen to prove the value of the biological prior. Future iterations might explore "biologically-plausible" fine-tuning to further close the gap with human subjective ratings.

Summary updated

Abstract

The ultimate ambition of visual quality assessment is to reproduce the neural computations of human vision and thereby achieve full alignment with human perception. However, the immense complexity of human visual circuitry makes this goal difficult to realize. Recently released fly visual connectomes provide a tractable biological starting point. Despite differences between fly and human vision, the fly's well-characterized motion pathways motivate us to investigate their potential for assessing the perceptual impact of temporal distortions. In this paper, we use a shared, frozen fly visual circuit to perform full-reference quality assessment of frame-interpolated videos, whose artifacts can disrupt temporal consistency. Specifically, compound-eye sampling maps reference and distorted sequences into the circuit, while a connectome-aware readout quantifies discrepancies in cell-type-specific activity and excitatory--inhibitory synaptic dynamics. Reference-guided temporal pooling and a lightweight regression head integrate these differences into a quality score. Experiments across multiple interpolation methods demonstrate better agreement with human ratings than CNN, RNN, and Transformer baselines with comparable parameters, supporting a biologically grounded approach to perceptual quality modeling.
BibTeX
@misc{chen2026flyvqa,
  author = {Baoliang Chen and Lingyu Zhu and Zhiwei Zhong and Shiqi Wang},
  title = {{FlyVQA: Seeing Video Frame Interpolation Quality through Fly Eyes}},
  year = {2026},
  eprint = {2611.24012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24012},
}
Figure