TL;DR
As autonomous "Deep Research" AI agents emerge to replace traditional search engines, evaluating their ability to synthesize complex, specialized knowledge is critical. This paper introduces a standardized 6-pillar evaluation framework and tests OpenAI’s and Google’s Deep Research tools on the challenging task of writing academic literature surveys for low-resource African NLP. The findings reveal that while these tools can rapidly structure long-form reports, they struggle significantly with domain-specific coverage, source tracking, and over-reliance on secondary summaries.
What Problem Does This Paper Address?
Large Language Models (LLMs) have progressed from simple chat interfaces to autonomous agents capable of multi-step web browsing, information extraction, and long-form report generation—a capability widely marketed as "Deep Research." While these tools promise to draft comprehensive literature reviews in minutes instead of days, they operate largely as black boxes. They risk hallucinating references, citing low-credibility sources, or completely missing niche but foundational research.
This problem is particularly acute in low-resource domains, such as Natural Language Processing (NLP) for African languages, where critical papers rarely rank at the top of standard search engine optimization (SEO). The authors address the lack of rigorous, standardized frameworks to evaluate how well these autonomous agents actually perform when tasked with highly specialized academic synthesis.
Methods
The authors propose a structured evaluation sheet organized around six core "pillars":
- Domain-Specific Coverage & Depth: Ability to identify and consolidate the most impactful research in a targeted field.
- Hallucination Assessment: Minimization of factual errors or fabricated claims.
- Correctness of Sources: Ability to distinguish authoritative academic papers from low-credibility web content.
- Information/Link Validity: Accessibility and accuracy of the cited references.
- Information Latestness: Prioritization of recent trends and emerging resources over outdated, highly-cited historical data.
- Comparison vs. Traditional Search: Quantifying the added value of the agent over standard keyword-based Google searches.
Reviewers grade these pillars using a 6-point Likert scale (from 0 = Strongly Disagree to 5 = Strongly Agree).
To test this framework, the authors used a comprehensive prompt template to instruct both OpenAI’s Deep Research and Google’s Deep Research tools to generate academic survey papers on the NLP landscapes of three countries: Ethiopia, Nigeria, and Kenya. The typical agentic workflow being evaluated involves an iterative loop of searching, reading, and reasoning, as illustrated in Figure 1.
Datasets
Rather than introducing a dataset for model training, this paper establishes an evaluation methodology using three recent, human-written regional NLP survey papers as the ground-truth baseline:
- The Ethiopian language survey (Tonja et al., 2023)
- The Nigerian language survey (Inuwa-Dutse, 2025)
- The Kenyan language survey (Amol et al., 2024)
Three expert reviewers directly compared the AI-generated reports against these human-compiled baselines to determine their accuracy and coverage.
Key Insights and Ideas
- The Domain Blindspot: Both OpenAI and Google's tools scored below average in identifying the most valuable research works. Because papers on low-resource languages are often buried deep in search results, the autonomous agents failed to surface them, highlighting a major risk of AI reinforcing existing data biases.
- Surface-Level Aggregation: Instead of extracting insights from original, primary research papers, the Deep Research tools heavily favored pre-existing survey papers and secondary summaries.
- Social Media Contamination: Both tools frequently integrated low-credibility sources, such as Facebook and Reddit links, into academic reports alongside verified peer-reviewed catalogs.
- Google's Aggressive Summarization: Google’s tool tended to heavily summarize information to the point of completely omitting the citations it discovered during its search phase, severely damaging its source verification scores.
Key Results
The human evaluation, visualized via radar charts across the six pillars (A through F), showed a stark contrast between the two primary tools:
- OpenAI Deep Research demonstrated superior and more consistent performance, generally scoring between 3 and 4 out of 5 across all three target countries.
- Google Deep Research exhibited a severe regional disparity. While it performed decently on the Kenyan and Nigerian use cases, its performance plummeted to scores of 1 and 2 when handling the Ethiopian NLP landscape, showing a significant failure in regional retrieval depth.
While outright text hallucination rates were not explicitly high, both tools struggled significantly with link validity and identifying the true original sources of the information they presented.
Limitations and Future Work
The authors acknowledge that this study is a preliminary proof-of-concept restricted to a small sample size of generated reports within a single domain (African NLP). Furthermore, because commercial Deep Research tools are updated rapidly by tech providers, performance benchmarks represent a moving target.
Future work aims to scale this evaluation framework by involving a larger pool of human experts and expanding the testing to other emerging proprietary and open-source Deep Research tools across diverse domains like financial market analysis and sports science.
Summary updated