Self-Adjusting Temporal Search: The 1/5th Success Rule for Frame Selection in Video-Language Models

Video-language models answer questions about long videos by looking at only a handful of frames, typically 8 to 32. Which frames they see largely decides whether they can answer. Showing more frames does not help: accuracy saturates or even drops, while the moment that actually settles the question can be a few seconds inside an hour of footage. Current fixes either sample uniformly or train a reinforcement-learning agent to request more frames. We propose a third option that needs no training. We treat frame selection for a frozen model as a small evolutionary search over time windows. The search step size, meaning how far in time a candidate window is allowed to jump, is set by the 1/5th success rule, a classic feedback rule from evolutionary computation. The rule widens the step when recent jumps often improve the model's answer and narrows it when they rarely do.

The reason to expect this to work comes from evolutionary-algorithm theory. When the best parameter value changes by orders of magnitude over a single run, simple success-based self-adjustment follows it and beats every fixed setting. Localizing a short piece of evidence in a long video looks like exactly that situation: early in the search, candidate windows should jump across the whole video, and late in the search they should shift by fractions of a second. The central question of this proposal is whether that "moving target" structure is real in video. The test is whether a self-adjusting controller beats both fixed step sizes and the much simpler option of drawing step sizes at random from a heavy-tailed distribution, and whether its advantage grows with video length.

Core Claim

For a frozen video-language model and a fixed budget of model forward passes, a (1+λ) evolutionary search over a set of K frames should find better frame sets than uniform sampling, CLIP retrieval, fixed-step search, or power-law random step sizes, with the step size and offspring count controlled by the discrete 1/5th rule. A (1+λ) search keeps one current best frame set and proposes λ modified copies ("offspring") per round. We further predict that the gap over power-law sampling will be near zero on short videos and grow on long ones. That prediction is what makes the claim falsifiable. If the gap does not appear, the honest conclusion is that lazy random step sizes are enough for temporal search, and we will report that.

Relation to Prior Work

The mechanism comes from the self-adjustment literature. Optimal Parameter Choices Through Self-Adjusting: Applying the 1/5-th Rule in Discrete Settings extends the 1/5th rule to discrete problems, adjusting offspring population size from observed success rates, and proves it strictly beats static settings. The (1+λ) Evolutionary Algorithm with Self-Adjusting Mutation Rate applies the same feedback idea to the mutation rate. Runtime analysis for self-adaptive mutation rates shows that encoding the rate in the genome also tracks the best rate. All of this theory is proved on synthetic benchmarks such as OneMax, and the research file notes that its behavior on perception-grounded, noisy objectives is untested. We take the control rule unchanged and test it in that regime.

Two works constrain the design. Lazy parameter tuning and control shows that drawing parameters at each step from a power-law distribution, with no tracking of search state, is often competitive with adaptive control. That result is the main threat to this proposal, so we make it a primary comparison rather than a footnote. Parameter control: strategy or luck? raises the same doubt about whether controllers help beyond chance.

The closest temporal work is VideoBrain. It trains Qwen3-VL-8B with reinforcement learning (GRPO) to decide when to call a CLIP retrieval agent or a dense uniform-sampling agent. It reports +8.1% on LongVideoBench and +9.0% on LVBench with 30–40% fewer frames. It also finds that imprecise sampling, meaning time intervals that are too broad or text search prompts that are too vague, causes 54.3% of its errors. Our approach differs in three ways. It requires no training. It searches directly over frame positions using the model's own answer scores rather than text prompts. And its step size narrows automatically, which targets the "over-broad interval" failure VideoBrain reports.

Other connecting work does not address inference-time temporal control. ACO-MoE-LoRA uses ant-colony optimization to configure adapter ranks during training on static segmentation, and the configuration is frozen at inference. GS-MODAC trains a graph-network PPO controller for evolutionary operators on scheduling problems. Both need offline training, and ACO-MoE-LoRA reports that its fitness signal becomes unstable when validation data is small or noisy. That warns us that noise in the fitness signal is a central risk for our method too.

Three benchmarks motivate the gap. TemporalBench finds accuracy saturates at 8–16 frames. EgoSchema finds more frames sometimes hurt, and reports models at 20–32% against 76.2% for humans. DVBench finds that denser sampling does not improve performance monotonically. Together these say that choosing which K frames to show matters more than raising K.

Research Program

The object being searched is a set S of K frames, with K fixed at 16 to match the TemporalBench saturation point. The set is represented as K timestamps. To make an offspring, we pick a random subset of timestamps and move each by an offset drawn from a Laplace distribution with scale σ seconds. We occasionally also swap one frame for a CLIP-retrieved candidate.

The fitness of a set S is the frozen model's answer margin, $m(S) = \log p(a_1 \mid S, q) - \log p(a_2 \mid S, q)$. Here $q$ is the question, and $a_1$ and $a_2$ are the model's top two multiple-choice options under S. The margin needs one forward pass per candidate and uses no labels.

An offspring counts as a success if its margin beats the parent's by more than a small threshold τ. The threshold makes success robust to near-ties. After each round, the step size is updated with the discrete 1/5th rule:

$$\sigma \leftarrow \sigma F \text{ on success}, \qquad \sigma \leftarrow \sigma F^{-1/4} \text{ otherwise},$$

where $F>1$ is a fixed factor, set to 1.5 as is conventional. This update settles at a success rate of about one in five. The offspring count λ is adjusted with the same rule, following the discrete-setting paper. The search starts from uniform sampling with σ equal to a quarter of the video length and stops at a fixed budget B of forward passes, which we vary over 16, 32 and 64.

We have to take seriously the risk that the search talks itself into a wrong answer, raising the margin of an incorrect option. We address it in two ways. First, we log the margin trajectory together with correctness and measure how often a rising margin coincides with a flip from right to wrong. Second, we run a variant whose fitness is the CLIP similarity between the question text and the frames instead of the model margin, which cannot flatter the model's own beliefs.

The theoretical analysis stays modest. We record the σ trajectory on each question. For questions whose supporting evidence can be localized, we check whether σ shrinks toward the scale of that evidence. This directly tests the "wide early, narrow late" premise instead of assuming it.

Baselines

The naive reference is uniform sampling of K=16 frames with the same frozen backbone, Qwen3-VL-8B (VideoBrain's base) and InternVL from OpenGVLab/InternVL. It costs one forward pass. We also run "uniform at matched compute": B forward passes on disjoint uniform subsets, choosing the answer by majority vote. This separates the benefit of spending more compute from the benefit of searching.

The second baseline is CLIP top-K retrieval, the core of VideoBrain's semantic agent, using the video CLIP backbones in perception_models. It is the strongest cheap, training-free competitor.

The strongest existing method is VideoBrain. Its reported numbers are relative: +8.1% on LongVideoBench and +9.0% on LVBench over uniform sampling, with 30–40% fewer frames. No absolute VideoBrain accuracy on EgoSchema, Video-MME or TemporalBench was found on Lacuna. If its weights are released, we run it on the same backbone and benchmarks. If not, we compare our relative gain over uniform sampling on the same backbone with its reported relative gains on LongVideoBench and LVBench. We count as beating it matching or exceeding +8.1 and +9.0 points there without any training, or matching it while using fewer total model calls.

The key ablations share the same search, budget and fitness and change only the step-size policy. The first fixes σ at each value in a log grid and reports the best fixed value chosen with test-set hindsight, which is a deliberately generous upper bound on fixed settings. The second draws σ afresh each round from a power law over the range from one second to the full video length, following Lazy parameter tuning and control. Self-adjustment counts as meaningful only if it beats the lazy power-law variant with statistical significance on the long-video strata.

Why not just…?

Why not just sample uniformly with a fixed frame budget? TemporalBench shows gains stop around 8–16 frames. EgoSchema's median certificate length, about 100 seconds of evidence spread through a long video, leaves uniformly sampled models at 20–32%. Uniform sampling at matched compute is still included, because if spending B calls on voting matched search, the search would have no value.

Why not just draw step sizes from a power law? This is the real threat. The lazy-control result shows it is often enough, and the GS-MODAC evidence against it comes from combinatorial scheduling, not video. We do not argue it away. The Video-MME duration strata test it directly. Our prediction is parity on short clips and a growing advantage for self-adjustment on 30–60 minute videos, where the best step size must shrink by the most. If parity holds at every duration, the right recommendation is the power-law variant, which is simpler.

Why not just train an agent as VideoBrain does, or use a bigger model? Training costs GRPO compute for each backbone, and VideoBrain's dominant failure is imprecise sampling intervals, which a shrinking step size targets directly. On scale, DVBench reports a dense 27B model beating 122B and 397B mixture-of-experts models, so scale alone does not fix temporal selection. Our method can also run on top of a trained agent's initial frames, and we test that combination where weights exist.

Why not just use CLIP retrieval? Retrieval scores each frame against the question independently. It cannot notice that two frames are only informative together, which matters for ordering and counting. It is included as a baseline and as a search operator, so the search is credited only for gains beyond retrieval.

Proposed Experiments

The evaluation benchmarks are EgoSchema, Video-MME stratified by short, medium and long duration, TemporalBench scored with Multiple Binary Accuracy, and LongVideoBench and LVBench for direct comparison with VideoBrain. Reference points from the file include TemporalBench GPT-4o at 38.5% Multiple Binary Accuracy (human 67.9%), Video-MME GPT-4o at 71.9% and Gemini 1.5 Pro at 75%, and Oryx-32B at 74.9 on Video-MME with subtitles, per Oryx MLLM. These frame the headroom. They are not directly comparable to an 8B frozen model.

The decisive check has four parts. First, accuracy against forward-pass budget for every step-size policy. Second, the gap between self-adjusting and power-law step sizes plotted against Video-MME duration stratum, where we predict it grows. Third, the rate at which a rising margin coincides with a flip from right to wrong, which must stay below the rate of flips from wrong to right. Fourth, on TemporalBench, whether search improves Multiple Binary Accuracy and not just standard multiple-choice accuracy, since the latter can be gamed by single-frame cues. If self-adjustment fails the second test or the search fails the third, the central claim is rejected.