Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens

Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, Yi Yang

Abstract

Recent advances in large video-language models have displayed promising outcomes in video comprehension. Current approaches straightforwardly convert video into language tokens and employ large language models for multi-modal tasks. However, this method often leads to the generation of irrelevant content, commonly known as "hallucination", as the length of the text increases and the impact of the video diminishes. To address this problem, we propose Vista-LLaMA, a novel framework that maintains the consistent distance between all visual tokens and any language tokens, irrespective of the generated text length. Vista-LLaMA omits relative position encoding when determining attention weights between visual and text tokens, retaining the position encoding for text and text tokens. This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens. The proposed attention mechanism significantly reduces the chance of producing irrelevant text related to the video content. Furthermore, we present a sequential visual projector that projects the current video frame into tokens of language space with the assistance of the previous frame. Our approach significantly outperforms various previous methods (e.g., Video-ChatGPT, MovieChat) on four challenging open-ended video question answering benchmarks, setting a new state-of-the-art performance.

Introduction: The Challenge of Video Hallucination

Large Language Models (LLMs) adapted for video (Video-LMs) often suffer from a specific type of "forgetfulness" during long-form generation. Most modern LLMs utilize Rotary Positional Embeddings (RoPE) to encode the relative distance between tokens. While effective for natural language where local context is paramount, this mechanism causes the influence of visual tokens to decay as the generated text grows longer.

Figure 1: Comparison of Vanilla Attention (LLaMA) vs. EDVT Attention (Vista-LLaMA) in video QA. Vanilla attention loses focus on visual tokens over time, leading to hallucinations.

As the "distance" between the current word and the original video tokens increases, the model begins to prioritize its own recently generated text over the actual video content. This leads to hallucinations—where the model describes events or objects not present in the video because it has effectively "lost sight" of the visual evidence.

Methodology

Equal Distance to Visual Tokens (EDVT)

To combat visual decay, Vista-LLaMA introduces the Equal Distance to Visual Tokens (EDVT) attention mechanism. Instead of treating visual tokens as a sequential part of the text stream subject to positional decay, EDVT treats them as a persistent global anchor.

By omitting relative position encoding (RoPE) when calculating attention weights specifically between visual tokens and text tokens, every generated word remains mathematically "equidistant" from the video content. This ensures that the visual impact remains constant regardless of the response length, forcing the model to stay grounded in the visual evidence.

Figure 2: Detailed architecture of Vista-LLaMA, showing the EDVT attention mechanism where RoPE is omitted for visual-text interactions.

Sequential Visual Projector

Standard models often project video frames into tokens independently, which can neglect the temporal flow of the video. Vista-LLaMA employs a Sequential Visual Projector where the projection of the current frame is informed by the tokens of the previous frame ($x_t = f_Q(o_i, x_{t-1})$). This allows the model to capture vital temporal relationships—such as sequential actions—without requiring additional parameters in the frozen LLM backbone.

Experimental Results

Vista-LLaMA was evaluated on four major benchmarks (NExT-QA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA) and a new curated dataset, CineClipQA, which focuses on complex plot-driven elements like intention and perception.

Performance Highlights

  • State-of-the-Art Accuracy: The model achieved 60.7% accuracy on NExT-QA and 60.5% on MSRVTT-QA, significantly outperforming previous leaders like Video-ChatGPT and MovieChat.
  • Causal Reasoning: On the NExT-QA benchmark, the EDVT mechanism specifically boosted causal question accuracy from approximately 65% to 72.3%, demonstrating that superior visual grounding directly enhances reasoning capabilities.
  • Hallucination Reduction: Qualitative comparisons show that while baseline models often hallucinate common actions (e.g., assuming a person is "sitting"), Vista-LLaMA correctly identifies specific visual states (e.g., "standing") due to its persistent attention to visual tokens.

Conclusion

Vista-LLaMA demonstrates that hallucination in Video-LMs is largely a "distance problem" caused by standard positional embeddings. By implementing EDVT and a sequential projector, the model maintains a reliable connection to visual evidence throughout long-form generation. Future work may involve integrating EDVT during the initial large-scale pre-training phase to further enhance model robustness across diverse LLM architectures.

Summary updated

Abstract

Recent advances in large video-language models have displayed promising outcomes in video comprehension. Current approaches straightforwardly convert video into language tokens and employ large language models for multi-modal tasks. However, this method often leads to the generation of irrelevant content, commonly known as "hallucination", as the length of the text increases and the impact of the video diminishes. To address this problem, we propose Vista-LLaMA, a novel framework that maintains the consistent distance between all visual tokens and any language tokens, irrespective of the generated text length. Vista-LLaMA omits relative position encoding when determining attention weights between visual and text tokens, retaining the position encoding for text and text tokens. This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens. The proposed attention mechanism significantly reduces the chance of producing irrelevant text related to the video content. Furthermore, we present a sequential visual projector that projects the current video frame into tokens of language space with the assistance of the previous frame. This approach not only captures the temporal relationship within the video, but also allows less visual tokens to encompass the entire video. Our approach significantly outperforms various previous methods (e.g., Video-ChatGPT, MovieChat) on four challenging open-ended video question answering benchmarks. We reach an accuracy of 60.7 on the zero-shot NExT-QA and 60.5 on the zero-shot MSRVTT-QA, setting a new state-of-the-art performance. This project is available at https://jinxxian.github.io/Vista-LLaMA.
BibTeX
@inproceedings{ma2024vista,
  author = {Fan Ma and Xiaojie Jin and Heng Wang and Yuchen Xian and Jiashi Feng and Yi Yang},
  title = {{Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=v3PzaQzzw8},
}
Figure