Enhancing Long Video Understanding via Hierarchical Event-Based Memory

Dingxin Cheng, Mingda Li, Jingyu Liu, Yongxin Guo, Bin Jiang, Qingbin Liu, Xi Chen, Bo Zhao

Abstract

Understanding long-form video content requires capturing both fine-grained local details and broad temporal context. Existing video-language models often rely on coarse compression, which blurs distinct events and loses critical information. We present HEM-LLM, a Hierarchical Event-based Memory-enhanced Large Language Model. HEM-LLM adaptively segments videos into semantic events and employs a dual-memory system: local memory for intra-event details and global memory for inter-event dependencies. Our approach demonstrates significant improvements across nine benchmarks, notably achieving a 19.4% accuracy boost on the MovieChat-1K dataset.

Introduction

The transition from short clips to long-form video—such as movies, tutorials, and security footage—exposes the limitations of standard video-language models. To fit within the limited context window of a Large Language Model (LLM), current models typically compress the entire visual sequence into a small, fixed set of tokens. This "flat" approach results in information redundancy and the blending of unrelated actions, making precise temporal reasoning nearly impossible. For instance, in a 10-minute cooking tutorial, specific steps like "chopping onions" and "boiling water" may become indistinguishable in the compressed representation.

Methodology

The HEM-LLM architecture addresses these issues through a hierarchical approach that mimics human-like attention. The framework consists of three primary stages:

  1. Adaptive Sequence Segmentation: Instead of uniform sampling or fixed-length chunks, the model calculates cosine similarity between adjacent frame tokens. This allows it to identify natural boundaries where characters, backgrounds, or actions change significantly, partitioning the video into $K$ discrete semantic events.
  2. Intra-event Local Memory: Each detected event is processed by a frozen visual encoder and a trainable Q-Former. This specialized architecture extracts high-fidelity visual features specific to that scene, storing them in a local memory bank.
  3. Inter-event Global Memory: To maintain long-term context, the model utilizes a global memory bank. Information from previous events is compressed and injected into the processing of the current event, establishing dependencies across the entire video duration.
Overview of the HEM-LLM architecture (left) and the adaptive sequence segmentation process (right).

Finally, these refined visual tokens are projected into the embedding space of a Vicuna-7B LLM, which generates the final text response based on the user's prompt.

Experimental Results

HEM-LLM was evaluated across nine diverse benchmarks, including Video Question Answering (VQA), video captioning, and long video activity classification. The model consistently outperformed existing state-of-the-art (SOTA) methods.

Key performance highlights include:

  • MovieChat-1K: Achieved a 19.4% improvement in accuracy over the previous best model, MovieChat+.
  • Activity Classification: On the Breakfast and COIN datasets, the model improved Top-1 accuracy by 4% and 1.8%, respectively.
  • Efficiency: HEM-LLM demonstrated superior performance while sampling significantly fewer frames (e.g., 100 frames vs. 2048 frames in competing models), proving that intelligent segmentation is more effective than raw frame density.

Qualitative analysis further illustrates the model's ability to accurately segment complex activities into logical steps and generate descriptive captions for each event.

Qualitative example of adaptive segmentation and event-based captioning in a cooking video.

Conclusion

The success of HEM-LLM highlights the importance of shifting from frame-based to event-based processing for long-form video AI. By separating local context from global context, the model reduces information clutter and preserves semantic clarity. Future work will focus on optimizing memory storage and further reducing the computational overhead of the hierarchical pipeline to enable real-time processing of even longer video sequences.

Summary updated

BibTeX
@inproceedings{cheng2025enhancing,
  author = {Dingxin Cheng and Mingda Li and Jingyu Liu and Yongxin Guo and Bin Jiang and Qingbin Liu and Xi Chen and Bo Zhao},
  title = {{Enhancing Long Video Understanding via Hierarchical Event-Based Memory}},
  booktitle = {ICME},
  year = {2025},
  doi = {10.1109/icme59968.2025.11210102},
}
Figure