HIL: Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval

Jaeyoung Kim, Dohyeon Lee, Seung-won HwangNAACL 2024 · DOI

Introduction

Dense retrieval models, such as ColBERT, have revolutionized information retrieval (IR) by utilizing "late interaction" mechanisms. While these models excel in supervised settings like MS MARCO, they often struggle with "zero-shot" generalization to unseen domains, sometimes underperforming compared to traditional keyword-based methods like BM25. Researchers from Seoul National University have addressed this gap by introducing Hybrid Isotropy Learning (HIL), an architecture that balances isotropic and anisotropic vector representations to enhance zero-shot performance.

The Challenge of Vector Space Isotropy

The effectiveness of dense retrieval is closely tied to the geometric properties of the embedding space.

  • Isotropy refers to a state where word embeddings are uniformly distributed in all directions, which is typically beneficial for calculating cosine similarity.
  • Anisotropy occurs when embeddings are clustered within a narrow cone. While often viewed as a limitation, recent research suggests that anisotropy can help models generalize better to novel data.

Standard models often lean heavily toward one extreme. Furthermore, traditional metrics for measuring these properties often fail to account for the specific interaction between queries and documents in IR tasks.

Hybrid Isotropy Learning (HIL)

To overcome these limitations, the authors propose two key innovations: a new metric called InterIso and the HIL architecture.

InterIso: Interactive Isotropy

InterIso is a metric designed specifically for IR. Unlike general isotropy measures, it specifically calculates the cosine similarity between query and passage embeddings. A value near 0 indicates an isotropic space with sparse interaction, while a value near 1 indicates an anisotropic space with dense interaction.

The HIL Architecture

The HIL framework integrates both isotropic and anisotropic representations into a single model.

  • The L-layer (Isotropic): This layer utilizes a projected vocabulary embedding to maintain a "spread out" representation, preserving lexical information.
  • The H-layer (Anisotropic): This layer leverages the last hidden state of the BERT encoder, which naturally tends toward anisotropy.
Comparison of standard ColBERT and the proposed ColBERT-HIL architecture.

The model is trained using a specialized loss function based on the InterIso metric. This loss maximizes the "isotropy difference" ($\Delta$InterIso) between the L and H layers, forcing them to capture distinct and complementary features.

Layer-wise Analysis and Insights

The authors observed that BERT naturally develops isotropic embeddings in lower layers and anisotropic ones in higher layers. HIL formalizes this gradient. Analysis of the InterIso metric across layers shows that HIL successfully separates these properties, whereas standard ColBERT remains relatively neutral.

InterIso values across layers for various model configurations, showing how HIL diverges into isotropic and anisotropic components.

Key insights from the study include:

  • The "Goldilocks" Balance: Zero-shot success requires a harmonized approach where isotropic layers help with precise scoring and anisotropic layers handle the variance of unseen data.
  • Complementary Retrieval: Venn diagram analysis reveals that the isotropic and anisotropic modules identify different sets of relevant documents. By combining them, HIL achieves a more comprehensive retrieval.
  • Lexical Preservation: By maintaining an isotropic layer grounded in vocabulary, HIL avoids the "lexical breakage" that can occur in purely anisotropic models.

Experimental Results

The HIL architecture was evaluated on the MS MARCO training set and the BEIR benchmark for zero-shot performance.

  • BEIR Performance: ColBERT-HIL outperformed the standard ColBERT baseline by 2.7% and surpassed BM25 by 2.22% in nDCG@10.
  • Generalizability: The HIL framework was also applied to the CITADEL model, resulting in a 2.29% improvement on BEIR, demonstrating that the hybrid isotropy principle is broadly applicable across different dense retrieval architectures.

Conclusion

HIL represents a significant step forward in making dense retrieval models more robust for real-world, zero-shot applications. By strategically balancing isotropic and anisotropic representations, it narrows the performance gap with traditional lexical search methods. Future work may explore more sophisticated non-linear fusion strategies for combining these representations and optimizing the computational overhead associated with hybrid processing.

Summary updated

BibTeX
@inproceedings{kim2024hil,
  author = {Jaeyoung Kim and Dohyeon Lee and Seung-won Hwang},
  title = {{HIL: Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.437},
}
Figure