TL;DR
Frame2Freq introduces a novel family of frequency-aware parameter-efficient adapters that bridge the gap between static image models and dynamic video understanding by modeling motion in the spectral domain. By applying Fast Fourier Transforms (FFT) along the time axis, it captures crucial mid-frequency motion cues that traditional time-domain adapters miss. This approach achieves state-of-the-art performance on fine-grained action recognition benchmarks while maintaining minimal parameter and computational overhead.
What Problem Does This Paper Address?
Adapting image-pretrained Vision Foundation Models (VFMs)—like CLIP or DINOv2—to video tasks typically relies on Parameter-Efficient Fine-Tuning (PEFT) adapters operating in the time domain (e.g., temporal convolutions or attention fusion).
However, the authors demonstrate that these standard time-domain modules suffer from a strong spectral bias: they primarily focus on static spatial cues (low frequencies) or rapid flickering/noise (high frequencies). This causes them to miss subtle, medium-speed motion dynamics (mid-frequencies) that are absolutely essential for fine-grained video understanding. For instance, distinguishing between nearly symmetric actions like "opening a bottle" versus "closing a bottle," or precisely counting the number of rotations in a gymnastics dive, requires capturing these overlooked mid-frequency temporal patterns.
Methods
The authors introduce Frame2Freq, a family of frequency-domain adapters designed to inject spectral reasoning into frozen VFM backbones. Instead of treating motion purely as sequential frame differences, Frame2Freq explicitly models the frequency structure of the temporal dimension.
The architecture comes in two primary variants:
- Frame2Freq-ST (Short-Time Spectral Adapter): This variant applies a Short-Time Fourier Transform (STFT) to video patch embeddings to decompose them into localized temporal windows and frequency spaces. It uses depthwise 3D convolutions to capture short-term transitions and model relationships between neighboring frequency bands before projecting back to the time domain via an inverse STFT (iSTFT).
- Frame2Freq-MS (Multi-Scale Spectral Adapter): Designed for complex actions with overlapping temporal frequencies, this variant splits features into two parallel branches. One branch handles short-range temporal continuity via standard convolutions, while the other applies Fourier transforms across multiple window sizes to capture both fine and coarse motion patterns simultaneously.
Additionally, the authors propose a formal Frequency Discriminability Analysis framework inspired by Analysis of Variance (ANOVA). This technique mathematically quantifies how much class-separating power is carried by each temporal frequency band, providing a principled way to diagnose model biases.
Datasets
The authors evaluate Frame2Freq across five challenging fine-grained activity recognition datasets:
- Diving48: A dataset of over 18,000 competitive diving clips where classes are defined by subtle combinations of take-off types, somersault counts, and body positions.
- Something-Something v2 (SSv2): A large-scale dataset (~220k videos) focused on everyday hand-object interactions, evaluated in both full-dataset and data-scarce few-shot (1-shot and 5-shot) settings.
- Drive&Act: An in-cabin driver monitoring dataset containing a high volume of nearly symmetric action pairs (e.g., reaching for vs. placing an object).
- IKEA-ASM: A long-horizon furniture assembly dataset rich in symmetric motions (e.g., tightening vs. loosening screws).
- HRI-30: A dataset capturing close-range human-robot industrial collaboration tasks.
Key Insights and Ideas
- The Mid-Frequency Sweet Spot: Fine-grained actions typically reside in the 1–10Hz range (FFT bins 1–5 for a 16-frame video at 30fps). While standard temporal adapters leave this band underutilized, Frame2Freq adaptively reallocates its discriminative capacity directly into these highly informative mid-bands.
- Disambiguating Symmetric Actions: Nearly symmetric actions look almost identical in raw RGB space because they share the same spatial layouts and object configurations. However, in the spectral domain, their opposite motion phases and directions create distinct orientation-specific energy shifts. Frame2Freq leverages these spectral signatures to easily tell them apart.
Key Results
- State-of-the-Art on Diving48: Frame2Freq-MS achieves 92.2% top-1 accuracy, outperforming the main baseline (ST-Adapter) by +1.8% and outperforming fully fine-tuned models (like ORViT) by 4–5% while updating less than 10% of the parameters.
- Symmetric Action Mastery: On domain-specific datasets (Drive&Act, IKEA-ASM, HRI-30), Frame2Freq surpasses conventional PEFT methods by +9–11% specifically on symmetric action pairs.
- High Efficiency: Frame2Freq-MS maintains competitive efficiency with only 7.3M trainable parameters and an inference latency of 13.11 ms—making it significantly faster and lighter than heavy multimodal alternatives like VitaCLIP (14.09 ms, 29M params) or M2-CLIP (13.62 ms, 14.8M params).
Limitations and Future Work
The authors note that the performance gains of Frame2Freq are less pronounced on datasets with coarser, less motion-critical action definitions (such as the full SSv2 dataset), where fine-grained frequency modeling provides less added value over standard spatial cues.
For future work, the authors suggest moving beyond standard Fourier analysis to explore alternative time-frequency operators, such as wavelets or learnable multiresolution filters, which could capture even more complex, non-stationary temporal dynamics.
Summary updated