SSRB: Direct Natural Language Querying to Massive Heterogeneous Semi-Structured Data

Xin Zhang, Mingxin Li, Yanzhao Zhang, Dingkun Long, Yongqi Li, Yinghui Li, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang, Philip S. YuNeurIPS 2025 · OpenReview

TL;DR

This paper introduces SSRB, a novel benchmark for querying massive, heterogeneous semi-structured data using natural language. It addresses the growing complexity of these queries and the diversity of data schemas by proposing a unified neural retrieval approach and a comprehensive dataset generated with LLMs. The work highlights current neural retrievers' limitations in precisely understanding complex constraints and suggests avenues for future research.

What Problem Does This Paper Address?

Accessing information within semi-structured data (like JSON or XML documents) using natural language (NL) queries is becoming increasingly important, especially with the rise of LLM agents and RAG systems. However, this task faces two major hurdles:

  1. Data Diversity: Semi-structured data comes in a vast array of domains and schema variations. Creating custom solutions for each is prohibitively expensive.
  2. Query Complexity: NL queries are evolving beyond simple keyword searches. They now often combine exact field matching (e.g., "price > 100") with fuzzy semantic requirements (e.g., "high-quality product") and can involve multiple fields and implicit reasoning.

Traditional methods like formal database querying (e.g., NL2SQL) or simple keyword search fall short. This paper aims to bridge this gap by exploring neural retrievers as a unified solution for handling these complex NL queries over diverse semi-structured data.

Methods

The core of this work is the development of a new benchmark, SSRB (Semi-Structured Retrieval Benchmark), and the evaluation of neural retrievers on it.

  • Benchmark Construction: SSRB was built using a three-stage LLM-driven process:
  • Schema Generation: LLMs were prompted to create 99 diverse schemas across 6 domains, ensuring variety in structure and attributes.
  • Data Triplet Generation: For each schema, LLMs generated millions of <query, positive, negative> triplets. This involved configuring queries with varying numbers of exact and semantic conditions to ensure diversity and complexity.
  • Relevance Annotation: LLMs were used to label the relevance of retrieved documents for test queries. Human evaluation confirmed that LLM annotations achieved high agreement with human judgments, offering a cost-effective and efficient annotation strategy.
  • Querying Task: The task is framed as a text retrieval problem where neural retrievers are used to index semi-structured data and understand NL queries. The goal is to rank data objects based on their semantic relevance to the query.
  • Evaluation Settings: Two settings were considered:
  • In-schema retrieval: Data objects share the same schema.
  • Cross-schema retrieval: Data objects come from different schemas within the same domain.
  • Model Evaluation: The authors evaluated various neural retrievers, including encoder-based models (like InstructOR, BGE) and LLM-based models (like E5-mistral), alongside the traditional BM25 lexical retriever.

Datasets

The paper introduces the Semi-Structured Retrieval Benchmark (SSRB).

  • SSRB: This benchmark comprises:
  • 6 domains
    • 99 different data schemas
    • 14 million semi-structured data objects
    • 8,485 test queries with varying difficulty levels, combining exact and fuzzy matching conditions.
  • The dataset is publicly available.

Key Insights and Ideas

  • LLMs as Data Generators: The paper demonstrates the effectiveness of using LLMs for generating large-scale, diverse, and complex datasets for specialized retrieval tasks, significantly reducing the cost and effort of manual data creation.
  • LLMs for Annotation: LLM-based relevance judgments are shown to be a reliable and efficient alternative to human annotation for creating benchmark datasets.
  • Neural Retrievers' Promise: Neural retrievers, especially LLM-based ones, show significant potential for handling complex NL queries over semi-structured data, outperforming traditional methods like BM25.
  • Current Limitations: Despite progress, current state-of-the-art neural retrievers still struggle with precisely understanding and enforcing complex matching constraints in semi-structured data.
  • In-domain Fine-tuning is Key: Fine-tuning dense retrievers on task-specific data (like that generated for SSRB) can lead to substantial performance improvements, highlighting the need for specialized models.

Key Results

  • BM25 Struggles: The traditional BM25 lexical retriever performed poorly on this task, underscoring the need for semantic understanding.
  • Encoder-based Models Improve: BERT-style encoder models showed better performance than BM25.
  • LLM-based Models Lead: LLM-based retrievers achieved the best performance among generalist models, demonstrating their superior semantic understanding and reasoning capabilities.
  • Performance Gap: Even the best models achieved recall scores under 40% in in-schema retrieval, indicating significant room for improvement.
  • Fine-tuning Boosts Performance: Models fine-tuned on the SSRB training data achieved comparable or superior performance to larger, pre-trained models, validating the dataset's effectiveness.
  • Cross-schema Impact: Mixing schemas within a domain introduced some performance degradation but models could still distinguish semantic characteristics, suggesting the viability of unified neural indices.
  • Query Complexity Matters: Performance generally improved with more filtering conditions, especially semantic ones, showing that models can leverage richer query information.

Limitations and Future Work

  • LLM Bias: The reliance on LLMs for data generation and annotation introduces potential biases inherent in LLMs, which could affect the benchmark's fairness and inclusivity.
  • Lack of Formal Methods Comparison: The paper focuses on neural retrieval and does not compare its approach with traditional database query methods (like NL2SQL).
  • Suboptimal Recall: Current neural dense retrievers still have suboptimal recall performance for precise matching constraints, indicating a need for further research.
  • Future Directions:
    • Developing more task-specific retrievers that better handle the complexities of semi-structured data.
  • Investigating how model size scaling impacts performance on semi-structured data retrieval.
  • Exploring comparisons with pure database query approaches.
  • Further research into LLM-based data synthesis and automatic evaluation for specialized retrieval tasks.

Summary updated

Abstract

Searching over semi-structured data with natural language (NL) queries has attracted sustained attention, enabling broader audiences to access information easily. As more applications, such as LLM agents and RAG systems, emerge to search and interact with semi-structured data, two major challenges have become evident: (1) the increasing diversity of domains and schema variations, making domain-customized solutions prohibitively costly; (2) the growing complexity of NL queries, which combine both exact field matching conditions and fuzzy semantic requirements, often involving multiple fields and implicit reasoning. These challenges make formal language querying or keyword-based search insufficient. In this work, we explore neural retrievers as a unified non-formal querying solution by directly index semi-structured collections and understand NL queries. We employ LLM-based automatic evaluation and build a large-scale semi-structured retrieval benchmark (SSRB) using LLM generation and filtering, containing 14M semi-structured objects from 99 different schemas across 6 domains, along with 8,485 test queries that combine both exact and fuzzy matching conditions. Our systematic evaluation of popular retrievers shows that current state-of-the-art models could achieve acceptable performance, yet they still lack precise understanding of matching constraints. While by in-domain training of dense retrievers, the performance can be significantly improved. We believe that our SSRB could serve as a valuable resource for future research in this area, and we hope to inspire further exploration of semi-structured retrieval with complex queries.
BibTeX
@inproceedings{zhang2025ssrb,
  author = {Xin Zhang and Mingxin Li and Yanzhao Zhang and Dingkun Long and Yongqi Li and Yinghui Li and Pengjun Xie and Meishan Zhang and Wenjie Li and Min Zhang and Philip S. Yu},
  title = {{SSRB: Direct Natural Language Querying to Massive Heterogeneous Semi-Structured Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=LuMGoG6lBA},
}
Figure