TL;DR
Vision-Language Models (VLMs) often struggle with the precise geometry required to understand cinematic composition. CompositionEvidence addresses this by extracting explicit geometric features (like depth and visual mass) using specialized "expert" tools and injecting them into frozen VLMs through calibrated text or soft tokens, significantly boosting reasoning accuracy without full model retraining.
What Problem Does This Paper Address?
While modern VLMs are excellent at identifying what is in a scene (semantics), they are notoriously inconsistent at understanding where things are in a way that matters for cinematography (geometry). Tasks like determining if a shot is "left-heavy" or follows the "rule of thirds" require fine-grained spatial reasoning that standard vision encoders—optimized for general image-text alignment—often discard.
The authors identify a "naive injection" trap: simply adding tool-generated geometric data to a prompt can actually confuse a VLM, causing it to ignore its own visual features and perform worse than the baseline. The challenge is to provide this "geometric evidence" in a way that the model can actually reconcile with the image.
Methods
The researchers developed a pipeline called CompositionEvidence that operates in three stages:
- Extraction: Three frozen "expert" models—GroundingDINO (object detection), Depth Anything V2 (monocular depth), and LSD (line segments)—process the image.
- Formalization: A rule-based module synthesizes these outputs into a 24-dimensional vector representing nine key cinematic features, such as subject centroid, mirror similarity, and relative depth layers.
- Injection: The evidence is delivered to a frozen VLM via two distinct interfaces:
- Channel A (Textual): A training-free method that appends gated, neutral descriptions of the geometry after the question. Gating ensures only "reliable" measurements (based on confidence thresholds) are shown.
- Channel B (Soft Tokens): A minimal-training approach using a small 7.37M-parameter MLP projector to map the 24D vector into four "soft tokens" that are injected directly into the VLM's embedding space.
Datasets
- ShotBench: The primary evaluation benchmark, containing 3,608 expert-labeled QA pairs across eight cinematic dimensions. The study focuses on the 479-item Composition subset.
- ShotQA: A larger dataset used to fit the confidence gates for Channel A and train the small projector for Channel B.
- CineTechBench: Used for cross-benchmark evaluation to see how well the geometric features transfer to different cinematic taxonomies.
Key Insights and Ideas
- The Order Matters: A major finding is that placing evidence before the question (naive injection) hurts performance. Placing it after the question allows the VLM to form a task-specific prior first, which it then refines with the provided evidence.
- Gating Prevents Hallucination: Tools aren't perfect. By using the Youden J statistic to set thresholds, the authors only show evidence when the tools are likely correct. This "confidence gating" makes the system robust; even when the evidence is intentionally corrupted, the calibrated interface falls back to the model's visual prior rather than crashing.
- Integration vs. Measurement: The paper distinguishes between the tool being wrong (measurement error) and the model failing to use a correct tool output (integration error). High-end models like Claude-Opus actually showed regressions with textual evidence, suggesting they may "over-trust" their own internal (but flawed) spatial reasoning.
- The Vocabulary Gap: Every model tested failed completely on the "Short side" composition label. The features showed the geometry was recoverable, but the models lacked the specific cinematic language grounding to link the geometry to that specific term.
Key Results
Using Qwen2.5-VL-7B as the primary test subject:
- Baseline (Zero-shot): 49.5% accuracy.
- Naive Injection: 46.3% (Performance dropped!).
- Channel A (Gated Text): 56.2% (+6.7 points).
- Channel B (Soft Tokens): 61.0% (+11.5 points), nearly matching the performance of models that were fully fine-tuned on cinematic data.
The researchers also tested the training-free Channel A across nine different VLMs, including commercial APIs like GPT-4o and Gemini. As seen in Figure 4, while most models improved, the impact varied wildly, highlighting that the effectiveness of tool-augmentation depends heavily on the "receiving" model's internal architecture.
Limitations and Future Work
- Subjectivity: Cinematic composition is inherently subjective. The authors note that exact-match accuracy on a single label might not capture the full nuance of a shot.
- Feature Scope: The current nine features omit color, motion, and character gaze, all of which are vital to cinematic storytelling.
- Language Grounding: Future work is needed to help models bridge the gap between "raw geometry" and "cinematic terminology," particularly for rare labels like "Short side."
- API Constraints: Channel B (soft tokens) requires access to model embeddings, making it currently incompatible with closed-source commercial APIs.
Summary updated