Introduction
Dense Video Captioning (DVC) is a complex task requiring models to identify when events occur in long, untrimmed videos (localization) and what happens within those segments (description). Recent advancements have leveraged Retrieval-Augmented Generation (RAG) to utilize external memory banks of captions. However, these memories are typically "flat"—unorganized lists of text that introduce significant redundancy and noise into the search space.
HiCM² (Hierarchical Compact Memory Modeling) addresses these limitations by mimicking human cognitive processes. Unlike flat AI models, human memory is hierarchical; we tend to recall abstract concepts (e.g., "cooking pasta") before refining them into specific actions (e.g., "pouring cream"). HiCM² bridges this cognitive gap by organizing prior knowledge into a structured, hierarchical memory bank.
Methodology
The HiCM² framework introduces two core innovations: hierarchical memory construction and top-down retrieval.
Hierarchical Compact Memory Construction
To move beyond noisy, flat databases, HiCM² builds a compact "summary tree" of training captions.
- Bottom-Up Clustering: The model employs the FINCH algorithm to iteratively cluster captions based on semantic similarity.
- LLM-Based Summarization: To ensure the memory is both compact and clean, a Large Language Model (Llama3 70B) summarizes each cluster. This process distills thousands of redundant entries into a few thousand representative "prototypes," effectively denoising the data.
- Multi-Level Granularity: The resulting hierarchy spans from high-level abstract concepts at the top to detailed, specific actions at the bottom.
Top-Down Hierarchical Retrieval
During inference, the model processes new video input using a recursive retrieval strategy:
- Abstract Contextualization: Visual cues from the video first query the high-level memory to establish broad context.
- Recursive Refinement: The model "tracks" relevant high-level clusters down to their more detailed "children" in lower levels.
- Feature Integration: These retrieved semantic features are aggregated with temporal visual features and fed into a Transformer-based decoder (based on the Vid2Seq architecture) to generate final timestamps and captions.
Evaluation and Results
The model was evaluated on two standard DVC benchmarks: YouCook2 (procedural cooking videos) and ViTT (instructional videos).
HiCM² achieved state-of-the-art (SOTA) results across all primary metrics. On YouCook2, it reached a CIDEr score of 71.84, a substantial improvement over the previous SOTA (Vid2Seq) at 66.29. Beyond caption quality, the hierarchical approach significantly improved event localization, achieving an F1 score of 32.51. This suggests that structured semantic memory helps the model better understand the temporal boundaries of events.
Furthermore, the framework demonstrated high efficiency. Through LLM summarization, the memory size was reduced from approximately 10,000 components to just 2,000 without sacrificing performance, proving that structured, compact data is superior to raw, redundant datasets.
Discussion and Conclusion
The success of HiCM² highlights the value of semantic denoising via LLMs. By using LLMs as filters rather than generators, the model leverages a cleaner set of prior knowledge. The transition from a "list" to a "map" of memory allows the model to ignore irrelevant details early in the retrieval process, leading to more accurate and stable predictions.
While the current implementation relies on domain-specific training data, future work could involve integrating this hierarchical structure into larger multi-modal foundation models to enhance general-purpose video understanding. HiCM² represents a significant step toward more cognitively-inspired and efficient video AI.
Summary updated