Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

Abhishek Dalvi, Vasant Honavar

TL;DR

HDFLIM (HyperDimensional computing with Frozen Language and Image Models) achieves cross-modal alignment between vision and language models without updating a single parameter. By projecting frozen embeddings into a 50,000-dimensional space and using symbolic operations like "binding" and "bundling," it constructs an associative memory in a single pass over the data, offering a computationally efficient alternative to traditional gradient-based fine-tuning.

What Problem Does This Paper Address?

Aligning large unimodal foundation models (like a vision encoder and an LLM) typically requires "glue" in the form of adapters or end-to-end fine-tuning. This process is computationally expensive, requires massive datasets, and risks catastrophic forgetting, where the models lose their original unimodal strengths.

Furthermore, recent research suggests that independently trained vision and language models naturally converge toward a shared "latent semantic structure" of the world. The authors ask: if these models already speak a similar internal language, can we align them using lightweight symbolic operations instead of heavy-duty backpropagation?

Methods

The authors introduce HDFLIM, a framework that treats multimodal alignment as a high-dimensional memory retrieval task rather than an optimization problem.

  1. Hyperdimensional Projection: Features from frozen models (DINOv3 for vision, Qwen3-4B for text) are projected into a 50,000-dimensional bipolar space (vectors of -1s and 1s) using Locality Sensitive Hashing (LSH). This preserves the semantic similarity of the original embeddings.
  2. Symbolic Operations:
    • Binding ($\otimes$): A dimension-wise multiplication that associates a visual vector with a textual vector to create a joint representation.
  3. Bundling ($\oplus$): A majority-rule operation that aggregates multiple joint representations into a single "prototype" vector.
  4. Single-Pass Learning: Unlike deep learning which requires multiple epochs, HDFLIM builds a Prototype Memory in a single pass. It iterates through image-caption pairs, binds the image features with the token features, and "bundles" them into a massive matrix indexed by token position and vocabulary.
  5. Inference via Retrieval: To generate a caption, the model computes the joint representation of the image and the current partial text. It then performs a similarity search (using Hamming distance) against the stored prototypes in memory to find the most likely next token.
  6. Logit Mixing & CLIP Guidance: To ensure grammatical fluency and visual grounding, the raw hyperdimensional scores are mixed with the LLM’s own internal logits and refined using CLIP-based similarity scores during sampling.

Datasets

The models were trained and evaluated on several standard benchmarks:

  • MS-COCO (Karpathy split): ~82k images used for the primary learning phase.
  • PixelProse: A large-scale dataset of 13 million image-caption pairs used to test the scalability of the single-pass learning approach.
  • NoCaps: Used to evaluate zero-shot generalization to out-of-domain objects.
  • Flickr30k: Used for cross-domain evaluation.

Key Insights and Ideas

  • Alignment as Mapping, Not Training: The core "aha moment" is that alignment doesn't require changing weights. If the models are "smart" enough, you just need a structured way to map their existing representational spaces to one another.
  • The Power of Near-Orthogonality: In 50k-dimensional space, randomly chosen vectors are almost certainly orthogonal. This property allows HD computing to store vast amounts of information in a single vector without interference, which HDFLIM exploits to build its associative memory.
  • Robustness to Model Swapping: Because the alignment is symbolic, the authors found they could swap the base LLM for an "Instruct" version at inference time with only minimal performance degradation, proving the learned prototypes are robust to distributional shifts.
  • Efficiency: Because there is no backpropagation, the "training" (learning phase) is limited only by disk I/O and the speed of a single forward pass through the frozen models.

Key Results

  • Competitive Performance: HDFLIM achieves SPICE and CIDEr scores comparable to some end-to-end trained models and significantly outperforms training-free baselines like ZeroCap and ConZIC.
  • Semantic Grounding: Captions produced by HDFLIM were found to be more semantically grounded (higher CLIP-S scores) than zero-shot methods.
  • Speed: In terms of token generation, HDFLIM is significantly faster than iterative optimization-based methods (like ZeroCap) because it relies on simple bitwise XOR operations and lookups rather than computing gradients at test time.
  • Scalability: The authors successfully scaled the method to 13 million image-caption pairs, a feat that would be prohibitively expensive for many researchers using standard end-to-end fine-tuning.

Limitations and Future Work

  • Memory Footprint: The prototype memory is massive. For a large vocabulary and long sequences, the matrix can reach 1.2 TB. The authors mitigated this using np.memmap (on-disk storage) and bit-packing, but it remains a hardware challenge.
  • Unidirectional Mapping: Currently, the framework only maps Vision $\rightarrow$ Language. The authors suggest that Language $\rightarrow$ Vision (image generation) is a logical next step.
  • Batching: The current implementation does not support batched inference, which limits its throughput in production environments.
  • Linguistic Fluency: While the HD memory captures semantics well, it can sometimes struggle with perfect grammar, necessitating the "Logit Mixing" with the frozen LLM to maintain sentence structure.

Summary updated

Abstract

Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.
BibTeX
@misc{dalvi2026hyperdimensional,
  author = {Abhishek Dalvi and Vasant Honavar},
  title = {{Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning}},
  year = {2026},
  eprint = {2602.23588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.23588},
}
Figure