Abstract
Recent advances in large video-language models have displayed promising outcomes in video comprehension. Current approaches straightforwardly convert video into language tokens and employ large language models for multi-modal tasks. However, this method often leads to the generation of irrelevant content, commonly known as “hallucination”, as the length of the text increases and the impact of the video diminishes. To address this problem, we propose Vista-llama, a novel framework that maintains the consistent distance between all visual tokens and any language tokens, irrespective of the generated text length. Vista-llama omits relative position encoding when determining attention weights between visual and text tokens, retaining the position encoding for text and text tokens. This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens. The proposed attention mechanism significantly reduces the chance of producing irrelevant text related to the video content. Furthermore, we present a sequential visual projector that projects the current video frame into tokens of language space with the assistance of the previous frame. Our approach significantly outperforms various previous methods on four challenging open-ended video question answering benchmarks, reaching an accuracy of 60.7 on zero-shot NExT-QA and 60.5 on zero-shot MSRVTT-QA.
Introduction: The Hallucination Challenge
Large Language Models (LLMs) like LLaMA are increasingly being adapted for video understanding by projecting video frames into "visual tokens" that the LLM processes alongside text. However, these models suffer from a significant flaw: hallucination.
As an LLM generates a response, the "distance" between newly generated words and the original visual tokens increases. Standard LLMs utilize relative position encodings (such as Rotary Positional Embeddings, or RoPE), which cause the influence of visual tokens to decay as the sequence grows longer. Consequently, the model often "forgets" the video context and begins generating text based solely on its own previous words, leading to inaccurate descriptions of the video content.
Methodology
Vista-LLaMA introduces two primary architectural innovations to mitigate these issues: EDVT-Attention and a Sequential Visual Projector.
EDVT-Attention (Equal Distance to Visual Tokens)
To prevent the fading impact of visual information, Vista-LLaMA employs Equal Distance to Visual Tokens (EDVT) attention. While standard attention mechanisms use RoPE to help the model understand token order, Vista-LLaMA selectively omits RoPE for visual-to-text interactions. This ensures that every visual token remains at a constant, "equal distance" from every generated word, regardless of the output length. By making the video context "timeless" relative to the text generation, the model maintains a persistent anchor to the visual evidence.
Sequential Visual Projector
Beyond attention, Vista-LLaMA improves temporal reasoning through its visual projector. Instead of treating video frames as independent images, the model uses a "SeqQ-Former." This component uses the projected features of the previous frame as a query to process the current frame. This allows the model to encode temporal relationships directly into the visual tokens before they reach the LLM, resulting in a more compact and information-dense representation of the video sequence.
Experimental Results
Vista-LLaMA was evaluated on several standard benchmarks, including NExT-QA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA. The model set new state-of-the-art (SoTA) records for zero-shot video question answering:
- NExT-QA: Achieved 60.7% accuracy, significantly outperforming Video-ChatGPT (54.6%) and MovieChat (49.9%).
- MSRVTT-QA: Reached 60.5% accuracy, representing a nearly 10% improvement over previous leading models.
Qualitative analysis confirms that Vista-LLaMA is far less prone to common hallucinations. For example, while baseline models might incorrectly describe a person's posture based on linguistic priors, Vista-LLaMA correctly identifies the action because the visual tokens remain highly weighted in the attention mechanism throughout the generation process.
Conclusion
By addressing the fundamental issue of decaying visual influence in long-form generation, Vista-LLaMA provides a robust framework for video-language understanding. The combination of EDVT-Attention and sequential temporal projection allows the model to maintain high fidelity to visual evidence, significantly reducing hallucinations and setting a new standard for performance in zero-shot video QA tasks.
Summary updated