Attribute or Abstain: Evaluating Evidence-Grounded Retrieval on the Low-Resource African Business Web

Bakary Gibba, Muhammad Abba Mandara, Karim Ismail Abd-Ellah MohamedarXiv 2026 · arXiv

TL;DR

Researchers introduced AfriBizClaims, a new benchmark designed to evaluate how well AI systems research companies in the "low-resource" African business web. The study reveals a surprising "less is more" result: simple retrieval-augmented generation (RAG) systems significantly outperformed complex, multi-step agent pipelines, which often hallucinated real information for fictitious companies.

What Problem Does This Paper Address?

In high-resource environments like Wikipedia, Retrieval-Augmented Generation (RAG) works remarkably well. However, the African commercial web is a different beast: data is sparse, websites are often outdated or inaccessible, and information is spread across multiple languages (English, French, Portuguese).

In a business context, a "confident wrong answer"—like an AI hallucinating a company's headquarters or sector—is far more dangerous than no answer at all. Current benchmarks don't sufficiently test a system's ability to abstain when evidence is missing. This paper tackles the challenge of "entity fidelity": ensuring the AI doesn't just find a fact, but the correct fact for the specific entity requested, even when the web is "thin."

Methods

The authors evaluated three distinct architectures, all powered by the same underlying language model (DeepSeek-V4.1-Flash):

  1. BM25: A classic keyword-based retrieval baseline searching over a pooled corpus of relevant pages.
  2. Web RAG: A standard pipeline that sends queries to a web search API, fetches the top results, and generates an answer.
  3. Konexium V1 (Agent Pipeline): A complex, multi-step agent designed to plan searches, crawl sites, and apply internal "verification gates" to ensure data quality.

To evaluate these, they used a rigorous "AI Judge" framework (using GPT-6.1-sol) to check if claims were actually supported by the cited source. They also cross-referenced these labels with a second AI judge (Claude-Sonnet-5-5) and human annotators to ensure the evaluation itself was robust.

Datasets

The paper introduces AfriBizClaims, a benchmark consisting of:

  • 150 queries covering six African markets: Ghana, Kenya, Nigeria, South Africa (English), Senegal (French), and Mozambique (Portuguese).
  • 84 discovery queries (e.g., "Find 3 logistics companies in Kenya").
  • 66 verification queries about specific companies.
  • Crucially, 24 queries about fictitious companies. A perfect system must decline to answer these, as no evidence exists.

Key Insights and Ideas

  • The Complexity Trap: More steps between a query and the evidence did not lead to better answers. The multi-step agent often "lost the plot," turning specific company queries into generic sector searches and then incorrectly attributing those results to the original company.
  • Entity Fidelity vs. Citation Support: A system can provide a perfectly cited sentence that is technically "supported" by a webpage, but if that webpage is about the wrong company, the answer is a failure. The authors argue that "entity fidelity" must be measured alongside citation accuracy.
  • The Illusion of Caution: The agent pipeline appeared to abstain frequently, but it often did so because it failed to initiate research, not because it correctly identified a lack of evidence. True abstention requires knowing why you aren't answering.
  • Baselines are Hard to Beat: Simple RAG systems that stay close to the retrieved text were highly reliable, achieving 98% attribution precision.

Key Results

  • Attribution Precision: BM25 and Web RAG both achieved 98% precision. In contrast, the complex agent pipeline struggled at 68% per-query precision.
  • Fictitious Companies: The simple baselines correctly declined every field for every fictitious company. The agent pipeline, however, hallucinated real-world companies as answers for 5 out of the 24 fictitious names.
  • Verification Failure: On verification queries (asking about a specific named firm), the agent pipeline never answered a question correctly. It consistently substituted other firms for the one requested.
  • Stability and Cost: The agent pipeline was the least stable (sharing only 1/3 of results between identical runs) and the most expensive to operate due to its high number of model calls.

Limitations and Future Work

The authors acknowledge that AfriBizClaims is relatively small (150 queries) and represents a "snapshot" of a constantly changing web. The "AI Judge" approach, while validated against humans, may still harbor systematic biases.

Future work suggests that agentic workflows need better "identity anchors" to prevent them from drifting away from the user's specific request during multi-step reasoning. The researchers have released the queries and labels to encourage the development of systems that know exactly when to speak up—and when to stay silent.

Summary updated

Abstract

Retrieval-augmented systems that research companies are judged by how much they find, yet in markets where public evidence is sparse, a confident answer without support costs more than no answer. We introduce AfriBizClaims, a benchmark of 150 company-research queries across six African markets in English, French and Portuguese, including queries about fictitious companies that a correct system must decline. Each claim a system makes is judged against the page it cites by an AI judge, checked against a second AI judge from another model family and against two human annotators, and abstentions are scored against answerability labels. We compare BM25 over a pooled corpus, standard web retrieval-augmented generation, and a multi-step agent pipeline (Konexium V1), all using the same language model. The two simple baselines are well attributed: 98% of their judged claims are supported by the page they cite, and they decline every field of every fictitious company. Counting claims about a company other than the one asked about as errors, the agent pipeline's per-query precision is 68%, and it never answers a question about a named company correctly: it substitutes other firms, including for 5 of the 24 fictitious names, and it is the least stable between runs and the most expensive. In the pipeline we evaluated, more steps between the query and the evidence did not buy more grounded answers. We will release the queries, system outputs, labels and short evidence excerpts.
BibTeX
@misc{gibba2026attribute,
  author = {Bakary Gibba and Muhammad Abba Mandara and Karim Ismail Abd-Ellah Mohamed},
  title = {{Attribute or Abstain: Evaluating Evidence-Grounded Retrieval on the Low-Resource African Business Web}},
  year = {2026},
  eprint = {2611.19900},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19900},
}
Figure