Film and video productions shoot far more than they use. Raw footage can outnumber the final cut by 20:1 to 40:1 (TRECVID 2006 Rushes Exploitation by CAS MCG). Editors then need to find "the take where she laughs before the line" inside hundreds of hours of retakes. The cheap text attached to this footage is speech transcripts, on-screen text and short automatic tags. Keyword search over that text breaks in two ways. The first is that standard term weighting assumes each clip is an independent document, and retakes violate that assumption badly. A line of dialogue repeated in twelve takes of one scene gets a low "rarity" weight because it appears in many clips, even though it pins down exactly one scene. The second is that editors describe what they see ("the funny reaction"), while the transcript records what was said, so many queries share no words with the index.
This proposal borrows a fix from geographic text analysis. Geolocation researchers found that a word's usefulness depends on how concentrated it is across physical places, not across documents (Geolocation Prediction in Social Media Data by Finding Location Indicative Words; Ghent and Cardiff University at the 2012 Placing Task). We treat a production's own camera setups and scenes as its "places." We weight each term by how concentrated its occurrences are across those setups. The same concentration signal then decides which few shots are worth sending to an expensive captioning model, so that the vocabulary gap is closed only where cheap text leaves it open.
Core Claim
Inside raw footage, a term's weight should come from the entropy of its occurrence counts across automatically discovered setups. Its document frequency across clips is the wrong quantity. Entropy here means how evenly a distribution is spread; low entropy means concentrated. Concretely, we cluster shots into $K$ setups using cheap visual features plus timecode. For a term $t$ we let $p(c \mid t)$ be the fraction of its occurrences that fall in setup $c$. Its weight is then
$$w(t) = \log K + \sum_{c=1}^{K} p(c \mid t)\,\log p(c \mid t),$$
which is the information, in nats, that seeing $t$ gives about which setup a shot comes from. The weight is $\log K$ when all of $t$'s occurrences fall in one setup and $0$ when they are spread uniformly. We place $w(t)$ in the rarity slot of an otherwise standard BM25 scorer. BM25 is the usual saturating term-frequency ranking function.
The closest simple alternative is to collapse each group of retakes into one document and then compute ordinary IDF over the groups. This is the inverse-city-frequency idea of Han et al. applied to setups. The two differ in a specific, testable place. Inverse setup frequency only asks how many setups contain $t$. Entropy uses the counts. A word said 200 times in one setup and once each in 30 others gets a low inverse-setup-frequency weight but a high entropy weight. Repeated dialogue plus stray mentions elsewhere is exactly the pattern retakes produce.
The second claim concerns budget. Captioning every shot with a vision-language model is the approach of LAVE, and it does bridge the vocabulary gap, but it scales poorly. Under a fixed captioning budget, it is better to caption one representative per group of retakes, and to prioritize setups whose cheap text contains few low-entropy terms. Neither uniform sampling nor frame-novelty selection allocates the budget this way.
Research Program
The mechanism comes from three pieces of prior work. Han et al. selected location-indicative words with information gain ratio and inverse city frequency, raising accuracy within 161 km on North American tweets to 45%. That was a 10.6-point absolute gain, with a 5x prediction speedup. Van Laere et al. showed that ranking tags by geographic concentration ("geospread") beat $\chi^2$ selection across 500 to 10,000 world clusters. CERTH at MediaEval Placing Task 2013 used the spatial entropy of topics to build a hard exclusion list. All three operate on the Earth's surface and use the weighting as a filter or classifier feature. We move it to the internal geography of a single production and use it as a continuous retrieval weight.
Large-Scale Location Recognition and the Geometric Burstiness Problem motivates the move to video. There, repeated structures such as windows and fences matched across unrelated places and caused false matches. Down-weighting features that matched across many places lifted recall at 90% precision on Pittsburgh from 7.2% to 51.1%. Kitchen counters, slates and repeated room tone in footage are the same kind of repeated, uninformative structure.
The first build step is setup discovery. We detect shots with color-histogram boundaries, then group them into takes and setups. Grouping uses normalized cuts on a shot-similarity graph as in the CAS MCG rushes system, which reported 84.0% precision and 77.0% recall on repeated-take detection. We add timecode adjacency, and we compute setups at two or three granularities, following Van Laere's multi-scale partition. The second step is the cheap text layer: speech transcripts, on-screen text, and a small open-vocabulary tagger run on one keyframe per shot. The third step is the entropy-weighted index. The fourth is budgeted captioning. Given a budget of $B$ captioning calls, we choose take representatives by a greedy rule. The rule favors setups whose summed $w(t)$ over cheap terms is lowest. These are the places where the index knows least about what is on screen. The resulting captions enter the index as an extra field, similar to the pseudo-query field used for passages in MINDER.
The analysis side asks when entropy weighting helps and when it hurts. We expect it to hurt on terms whose relevance does not follow setup structure, such as a prop that travels between scenes. We will report per-query-type breakdowns to show this.
There is no public benchmark for natural-language search over raw footage. We will build a test set with two parts. The first part is editor-style queries written against the 48 TRECVID 2006 rushes test videos, which have real retake structure. The second part is a larger synthetic "rushes" collection built from YouCook2 and CrossTask. We pool several videos of the same recipe or task so that they act as retakes of one scene, and we reuse the datasets' step descriptions as queries. Following the WikiQA measurement that 20.3% of correct answers share no content words with the question (WikiQA), we will split every query set into queries that overlap lexically with the cheap text and queries that share zero content words with it. We report the two splits separately.
Baselines
The naive reference point is BM25 with ordinary IDF computed over individual shots, applied to the cheap text layer. Every other system must beat it on both query splits.
The most important competitor is take-collapsed BM25. We merge detected retakes into one document before computing IDF, which amounts to inverse setup frequency. It is the strongest threat because it captures most of the "retakes distort IDF" intuition at zero extra cost. It will use exactly the same setup clusters, the same text layer, and the same BM25 parameters, with $k_1$ and $b$ tuned on a shared development split under an equal grid. Entropy weighting counts as beating it only with a statistically significant gain in Recall@10 and mean reciprocal rank on the lexical-overlap split, using a paired bootstrap over queries.
The ablation that isolates the key ingredient swaps the clusters used in $w(t)$. We compare setups discovered from content, fixed-length time windows, and random partitions of the same size. If fixed time windows do as well, the "place" structure is not what matters.
The strongest dense competitor embeds one keyframe per shot, or short clips, with a video-language model such as InternVideo2 and retrieves by cosine similarity. We also run a hybrid that interpolates dense and sparse scores with weights tuned on the development split, in the spirit of the lexical-plus-neural fusion in WikiQA's CNN-Cnt (0.6520 MAP) and in WUY at SemEval-2020 Task 7. For reference, CAST reports 36.75% zero-shot InternVideo2 accuracy on YouCook2 consistent retrieval. That is a different task from ours. For shot retrieval in raw footage we found no reported result on Lacuna, so all baselines will be run by us.
The quality ceiling captions every shot, as in LAVE. For budgeted captioning, the competitors are uniform sampling, one caption per detected shot, and frame selection by the Shannon entropy of CLIP features as in TV-RAG. TV-RAG reported 76.5 on Video-MME with LLaVA-Video 72B, but for question answering rather than indexing. Every selector gets the same budget $B$ and the same captioning model. Our selector wins if it reaches a given zero-overlap recall with fewer captioning calls.
Why not just…?
Why not just collapse retakes and use ordinary BM25? This is the real threat, and the proposal is designed to test it rather than argue around it. Entropy and inverse setup frequency differ only for terms that are heavily concentrated in one setup but also appear sparsely in others. We will report how often such terms occur in the query-relevant vocabulary, alongside the head-to-head scores. If take-collapsed BM25 matches entropy weighting, we will report that the weighting contribution reduces to inverse setup frequency. The budget-allocation result would then carry the work.
Why not just use dense video-language embeddings for everything? Dense embeddings are cheaper than captioning and partly cover the vocabulary gap. However, CAST shows that zero-shot dense retrieval confuses near-identical procedural states and different people doing the same action. That confusion is the retake problem in another form. Dense embeddings also ignore dialogue unless transcripts are added. The dense and hybrid baselines above settle whether the sparse layer adds anything. If the hybrid gains nothing over pure dense retrieval on either query split, the sparse component is not worth keeping.
Why not just caption every shot with a large model, as LAVE does? On quality it probably wins, which is why it serves as the ceiling. The objection is cost at 20:1 to 40:1 footage ratios. Even adding audio captions costs about 1.77x the inference time of vision-only processing (AudioVLM2Vec). We report the full recall-versus-GPU-hours curve, so the reader can see what fraction of the ceiling a 5% or 10% budget recovers.
Why not just rewrite queries with a trained rewriter? Rewriting with reranker feedback, as in RaFe, improved open-domain QA accuracy by 2–3%. It is orthogonal to our method rather than a replacement: it cannot invent visual words that the index never contains. We will test a RaFe-style rewriter on top of each index to check whether gains add up or overlap.
Proposed Experiments
The first decisive check is entropy weighting against take-collapsed BM25 and shot-level BM25 on the lexical-overlap split of both test sets, together with the cluster-swap ablation. The second is the zero-overlap split under captioning budgets of 1%, 5%, 10% and 100% of shots. Here we compare our selector against uniform sampling, one caption per shot, and TV-RAG-style feature entropy, measuring Recall@10 and mean reciprocal rank against captioning GPU-hours. The third is the dense-plus-sparse hybrid, used to test whether the sparse index remains useful once dense retrieval is present. As a sanity check, we will measure setup-discovery quality on the TRECVID rushes repetition annotations against the 84.0% precision and 77.0% recall reported by CAS MCG. This tells us whether weaknesses in the weighting are really weaknesses in the clustering.
Three outcomes support the proposal. Entropy weighting should beat take-collapsed BM25 on overlap queries. Budgeted selection should reach, at 10% of the captioning cost, most of the zero-overlap recall that full captioning achieves, and do so more efficiently than feature-entropy selection. The hybrid should beat dense-only retrieval. If only the second holds, the paper becomes a study of caption budgeting in which spatial-entropy weighting is used only to allocate captions.
References
- TRECVID 2006 Rushes Exploitation by CAS MCG
- Geolocation Prediction in Social Media Data by Finding Location Indicative Words
- Ghent and Cardiff University at the 2012 Placing Task
- LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
- CERTH at MediaEval Placing Task 2013
- Large-Scale Location Recognition and the Geometric Burstiness Problem
- Multiview Identifiers Enhanced Generative Retrieval
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- WUY at SemEval-2020 Task 7: Combining BERT and Naive Bayes-SVM for Humor Assessment in Edited News Headlines
- CAST: Modeling Visual State Transitions for Consistent Video Retrieval
- TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding
- OmniCVR: A Benchmark for Omni-Composed Video Retrieval with Vision, Audio, and Text
- RaFe: Ranking Feedback Improves Query Rewriting for RAG