The most abundant source of training data for video understanding is the open web—specifically platforms like YouTube. However, this data is notoriously "noisy." In a typical instructional video, a narrator might describe a recipe's history while the camera shows a cat walking across the kitchen, or they might mention an ingredient seconds before it actually appears on screen.
This multimodal misalignment—where the visual, auditory, and textual streams do not semantically or temporally match—is the primary bottleneck in scaling video-language models. Recent research has shifted from simply collecting more data to developing sophisticated mechanisms for identifying, filtering, and restructuring these noisy signals.
The Density of Truth
If you have millions of video-text pairs, how do you know which ones to trust? Elad Amrani, Rami Ben-Ari, and colleagues proposed a solution based on density estimation in Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning.
The intuition is that "correct" associations are consistent across a dataset, while "noise" is idiosyncratic. If many videos of "people hunting" share similar visual features and similar textual descriptions, they will form a dense cluster in a shared embedding space. A noisy pair—where the text says "hunting" but the video shows a menu screen—will be a spatial outlier. By calculating the local density around each pair, models can automatically down-weight the loss from outliers, allowing them to learn from massive datasets like HowTo100M without manual cleaning.
Synthetic Alignment and Text-Only Proxies
When real-world data is too messy, one alternative is to create "perfect" data. Thomas Hayes and the creators of MUGEN built a synthetic playground where video, audio, and text are generated simultaneously from a game engine. This ensures 100% synchronization, providing a "clean" baseline to help models learn the fundamental physics of multimodal interaction before they tackle the chaos of the web.
Taking this a step further, Hehe Fan and Yi Yang introduced TOPA, which grants models video understanding without using real videos at all during the alignment phase. By training on "textual videos"—synthetic descriptions of temporal sequences—and leveraging the shared embedding space of models like CLIP, they bypass the noise of web-scraped subtitles entirely.
Anchoring with Knowledge Graphs
A major source of noise in instructional videos is that spoken commentary often includes conversational filler or irrelevant anecdotes. Honglu Zhou and Mubbasir Kapadia address this in Paprika by using Procedural Knowledge Graphs (PKG).
Instead of letting the model guess what matters from raw Automated Speech Recognition (ASR), they anchor the learning process to structured data from sources like wikiHow. By knowing the logical steps of a task beforehand (e.g., "boil water" must come before "add pasta"), the model can filter out irrelevant narration and focus on the visual cues that actually represent progress. Similarly, Hyeongcheol Park uses VAT-KG to provide a "concept-centric" anchor, reducing the hallucinations that occur when a model tries to reconcile misaligned audio and video.
Granular and Temporal Refinement
Standard retrieval models often try to align an entire video clip with a single sentence. However, Jielin Qiu and Bo Li argue in SCCS that this "global" alignment is a source of noise itself. They use Optimal Transport to align smaller, thematic segments of video and text. This allows the model to ignore irrelevant segments (like a 10-second intro) and focus on the specific frames that match the text.
In specialized domains like Minecraft, Haobin Jiang and Hao Luo found that even with large datasets, models struggle because narrators don't describe their actions precisely. Their model, CLIP4MC, overcomes this by introducing a "task completion" objective. By using the size of an entity on screen as a proxy for distance and progress, they create a more reliable training signal than the noisy YouTube transcripts alone could provide.
Why It Works: Quality Over Quantity
The recurring theme across these papers—from M2-RAAP to TeachText—is that the field is moving away from "brute force" scaling. Xingning Dong demonstrated that using LLMs to clean captions and selecting only high-quality keyframes can achieve better results with 90% less data. By treating noise not as an inevitability to be averaged out, but as a measurable variable to be filtered or restructured, these researchers are building models that are both more efficient and more factually grounded.