Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding

Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg

TL;DR

Frame2Freq introduces a novel family of frequency-aware parameter-efficient adapters that bridge the gap between static image models and dynamic video understanding by modeling motion in the spectral domain. By applying Fast Fourier Transforms (FFT) along the time axis, it captures crucial mid-frequency motion cues that traditional time-domain adapters miss. This approach achieves state-of-the-art performance on fine-grained action recognition benchmarks while maintaining minimal parameter and computational overhead.

What Problem Does This Paper Address?

Adapting image-pretrained Vision Foundation Models (VFMs)—like CLIP or DINOv2—to video tasks typically relies on Parameter-Efficient Fine-Tuning (PEFT) adapters operating in the time domain (e.g., temporal convolutions or attention fusion).

However, the authors demonstrate that these standard time-domain modules suffer from a strong spectral bias: they primarily focus on static spatial cues (low frequencies) or rapid flickering/noise (high frequencies). This causes them to miss subtle, medium-speed motion dynamics (mid-frequencies) that are absolutely essential for fine-grained video understanding. For instance, distinguishing between nearly symmetric actions like "opening a bottle" versus "closing a bottle," or precisely counting the number of rotations in a gymnastics dive, requires capturing these overlooked mid-frequency temporal patterns.

Methods

The authors introduce Frame2Freq, a family of frequency-domain adapters designed to inject spectral reasoning into frozen VFM backbones. Instead of treating motion purely as sequential frame differences, Frame2Freq explicitly models the frequency structure of the temporal dimension.

The architecture comes in two primary variants:

  1. Frame2Freq-ST (Short-Time Spectral Adapter): This variant applies a Short-Time Fourier Transform (STFT) to video patch embeddings to decompose them into localized temporal windows and frequency spaces. It uses depthwise 3D convolutions to capture short-term transitions and model relationships between neighboring frequency bands before projecting back to the time domain via an inverse STFT (iSTFT).
  2. Frame2Freq-MS (Multi-Scale Spectral Adapter): Designed for complex actions with overlapping temporal frequencies, this variant splits features into two parallel branches. One branch handles short-range temporal continuity via standard convolutions, while the other applies Fourier transforms across multiple window sizes to capture both fine and coarse motion patterns simultaneously.
Figure 2. Overview of Frame2Freq. Left: Frozen VFMs are adapted to video by inserting lightweight Frame2Freq adapters between transformer blocks, enriching spatial embeddings with frequency-aware temporal cues. Right: Unlike the ST-Adapter, our spectral variants use FFT-based branches: Frame2Freq-ST for single scale spectral clues and Frame2Freq-MS for multi-scale motion patterns.

Additionally, the authors propose a formal Frequency Discriminability Analysis framework inspired by Analysis of Variance (ANOVA). This technique mathematically quantifies how much class-separating power is carried by each temporal frequency band, providing a principled way to diagnose model biases.

Datasets

The authors evaluate Frame2Freq across five challenging fine-grained activity recognition datasets:

  • Diving48: A dataset of over 18,000 competitive diving clips where classes are defined by subtle combinations of take-off types, somersault counts, and body positions.
  • Something-Something v2 (SSv2): A large-scale dataset (~220k videos) focused on everyday hand-object interactions, evaluated in both full-dataset and data-scarce few-shot (1-shot and 5-shot) settings.
  • Drive&Act: An in-cabin driver monitoring dataset containing a high volume of nearly symmetric action pairs (e.g., reaching for vs. placing an object).
  • IKEA-ASM: A long-horizon furniture assembly dataset rich in symmetric motions (e.g., tightening vs. loosening screws).
  • HRI-30: A dataset capturing close-range human-robot industrial collaboration tasks.

Key Insights and Ideas

  • The Mid-Frequency Sweet Spot: Fine-grained actions typically reside in the 1–10Hz range (FFT bins 1–5 for a 16-frame video at 30fps). While standard temporal adapters leave this band underutilized, Frame2Freq adaptively reallocates its discriminative capacity directly into these highly informative mid-bands.
  • Disambiguating Symmetric Actions: Nearly symmetric actions look almost identical in raw RGB space because they share the same spatial layouts and object configurations. However, in the spectral domain, their opposite motion phases and directions create distinct orientation-specific energy shifts. Frame2Freq leverages these spectral signatures to easily tell them apart.
Frequency discriminability curves showing how Frame2Freq shifts focus to the informative mid-bands compared to ST-Adapter.

Key Results

  • State-of-the-Art on Diving48: Frame2Freq-MS achieves 92.2% top-1 accuracy, outperforming the main baseline (ST-Adapter) by +1.8% and outperforming fully fine-tuned models (like ORViT) by 4–5% while updating less than 10% of the parameters.
  • Symmetric Action Mastery: On domain-specific datasets (Drive&Act, IKEA-ASM, HRI-30), Frame2Freq surpasses conventional PEFT methods by +9–11% specifically on symmetric action pairs.
  • High Efficiency: Frame2Freq-MS maintains competitive efficiency with only 7.3M trainable parameters and an inference latency of 13.11 ms—making it significantly faster and lighter than heavy multimodal alternatives like VitaCLIP (14.09 ms, 29M params) or M2-CLIP (13.62 ms, 14.8M params).

Limitations and Future Work

The authors note that the performance gains of Frame2Freq are less pronounced on datasets with coarser, less motion-critical action definitions (such as the full SSv2 dataset), where fine-grained frequency modeling provides less added value over standard spatial cues.

For future work, the authors suggest moving beyond standard Fourier analysis to explore alternative time-frequency operators, such as wavelets or learnable multiresolution filters, which could capture even more complex, non-stationary temporal dynamics.

Summary updated

Abstract

Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle). To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer. Code is available at https://github.com/th-nesh/Frame2Freq.
BibTeX
@inproceedings{ponbagavathi2026frame,
  author = {Thinesh Thiyakesan Ponbagavathi and Constantin Seibold and Alina Roitberg},
  title = {{Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=j8v60V4BbK},
}
Figure