A Needle in a Haystack: Intent-driven Reusable Artifacts Recommendation with LLMs

Zhi Jin, Fellow of Ieee and Ccf, Dongming Jin, Zhi Jin, Xiaohong Chen, Zheng Fang, LinYu Li, Yuanpeng He, Jia Li, Yirang Zhang, Yingtao FangarXiv 2025 · DOI · arXiv

Introduction

Modern software development relies heavily on reusable artifacts, such as NPM packages for JavaScript, pre-trained models on HuggingFace, and system groups in Linux. However, as these ecosystems explode in size, finding the exact tool for a specific task has become a significant bottleneck. Traditional search methods like BM25 or TF-IDF rely on keyword matching, which often fails to bridge the "abstraction gap" between a developer’s high-level intent and an artifact’s low-level technical description.

The abstraction gap between high-level developer intent and low-level artifact descriptions across different ecosystems.

The TreeRec Framework

To solve this, we introduce TreeRec, a framework that combines the efficiency of hierarchical indexing with the reasoning power of Large Language Models (LLMs). TreeRec organizes software artifacts into hierarchical "feature trees," allowing for search complexity to move from linear ($O(n)$) to logarithmic ($O(\log n)$).

Methodology

The framework operates in three main phases:

  1. Tree Indexing (Bottom-Up): Artifact descriptions are converted into semantic vectors and clustered using Gaussian Mixture Models (GMM). An LLM (such as GPT-4) then summarizes these clusters into high-level "features," repeating the process recursively to form a tree.
  2. Tree-Guided Search (Top-Down): When a query is entered, TreeRec traverses the tree from the root, pruning irrelevant branches and only following nodes semantically similar to the user's intent.
  3. Refined Re-rank: A small set of candidate artifacts at the leaf level is passed to an LLM for a final, precise ranking based on the specific nuances of the request.

Experimental Setup and Datasets

We evaluated TreeRec using a new multi-ecosystem benchmark called IntentRecBench, which includes 968 JavaScript samples, 306 HuggingFace models, and 142 Linux system groups. To ensure high quality, the dataset underwent rigorous quality control, including independent reviews by humans and GPT-5, achieving over 90% acceptance for semantic consistency.

Key Results

TreeRec demonstrates substantial improvements in both accuracy and search speed across all tested ecosystems and metrics.

Comprehensive performance comparison (P@k and DCG@k) of TreeRec across JavaScript, HuggingFace, and Linux ecosystems.

Accuracy and Efficiency Gains

As illustrated in the performance metrics, TreeRec significantly outperforms traditional information retrieval (IR) methods and raw LLM-based scoring. Key highlights include:

  • Precision Boost: TreeRec improved GPT-4's Precision@1 (P@1) by 18% in JavaScript and 40% in HuggingFace.
  • Massive Speedup: In the JavaScript ecosystem, the average recommendation time was slashed from 643 seconds to just 4 seconds.
  • Model Agnostic: The hierarchical structure consistently improved performance regardless of the underlying LLM, showing strong results with DeepSeek-R1 and various Qwen models.

Conclusion and Future Work

TreeRec effectively addresses the "needle in a haystack" problem by using LLMs to bridge the abstraction gap through structured feature trees. This approach not only improves recommendation accuracy but also makes LLM-based search practical for large-scale production environments. Future research will explore methods for efficiently updating these feature trees in dynamic ecosystems and integrating real-time developer feedback.

Summary updated

BibTeX
@misc{ccf2025needle,
  author = {Zhi Jin, Fellow of IEEE and CCF and Dongming Jin and Zhi Jin and Xiaohong Chen and Zheng Fang and LinYu Li and Yuanpeng He and Jia Li and Yirang Zhang and Yingtao Fang},
  title = {{A Needle in a Haystack: Intent-driven Reusable Artifacts Recommendation with LLMs}},
  year = {2025},
  eprint = {2511.18343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.18343},
}
Figure