A Benchmark and Layered Evaluation Protocol for Evidence-Grounded Shared Belief Updating in Connected Robotics

Ranulfo Bezerra, Kimitoshi YamazakiarXiv 2026 · arXiv

In the world of connected robotics, a single robot rarely acts alone. Instead, a fleet of mobile robots, IoT sensors, and edge-AI modules must fuse their observations into a "shared belief" about the environment. But how do we know if a system is actually reasoning correctly, or just getting lucky with its final action?

A new paper from researchers at Tohoku University introduces the Connected Robotics Evidence Graph (CREG) Benchmark, a framework designed to peek inside the "black box" of robotic belief updating. By moving beyond simple success/failure metrics, CREG provides a layered look at how AI models handle the messy, contradictory, and delayed evidence typical of real-world deployments.

TL;DR

CREG is a new benchmark that evaluates the entire "evidence-to-belief" trajectory in connected robotics, rather than just the final action. It reveals that many AI models fail to "ground" observations to the correct physical entities and that while semantic relations are crucial for accuracy, complex graph formatting offers no inherent advantage over simple structured lists.

What Problem Does This Paper Address?

Most existing AI benchmarks for robotics use "endpoint evaluation"—they check if the robot reached the goal or picked the right object. However, in a connected system where data comes from multiple sources with varying reliability, a robot might take the "right" action for the "wrong" reason. For example, it might ignore a critical sensor contradiction or attribute a failure to the wrong machine.

The authors argue that we need to evaluate the belief trajectory: the step-by-step process of how a system’s internal world model changes as new evidence arrives. This is critical for safety-critical applications where we need to trust that the system is handling uncertainty and contradictions logically.

Fig. 1. Overview of CREG. Connected agents produce attributed semantic evidence organized in a shared graph; belief trajectory, contradiction records, next action, and explanation are exposed and scored separately, so an end-task success cannot conceal an earlier-layer error.

Methods

The researchers developed a suite of 350 scenarios generated from a "normative specification." This means they created a mathematical "ground truth" for what a perfect Bayesian reasoner should believe at every single step of a data stream.

Key methodological components include:

  • Layered Evaluation: Instead of one score, the benchmark tracks entity grounding, evidence attribution, belief revision, contradiction handling, and action selection.
  • Representation Ablations: They tested how different data formats—Canonical JSON, structured lists, and full graphs—impacted performance while keeping the underlying information identical.
  • Model Panel: They evaluated several open-weight Large Language Models (LLMs), including Qwen2.5 (1.5B and 7B), Mistral-7B, and Gemma-3-12B, alongside six transparent fusion baselines (like majority voting and recency-weighting).

Datasets

The primary contribution is the CREG Benchmark, which consists of:

  • 350 Scenarios: Covering seven reasoning regimes such as "justified revision" (changing your mind when evidence is strong) and "stale evidence" (ignoring old data that has been superseded).
  • 1,150 Prefix Pairs: Scored reference points that allow for step-by-step trajectory analysis.
  • Comparison to Prior Work: CREG is compared against general belief-revision benchmarks like BeliefTrack and BayesBench, but it is unique in its focus on physical robotic entities and time-stamped, attributed evidence.

Key Insights and Ideas

  • The Grounding Deficit: A major finding was that "weaker" models (like Mistral-7B or Qwen-1.5B) don't necessarily fail at logic; they fail at grounding. They often produce a belief update that isn't bound to the required physical entity (e.g., they talk about "the sensor" generally rather than "sensor-a").
  • Content Over Container: While semantic relations (knowing that "Sensor A measures Tank B") are vital for accuracy, the format doesn't seem to matter. A simple structured list performed just as well as, or better than, a complex graph serialization.
  • Non-Monotonic Failures: Belief accuracy isn't a straight line. Models often experience a "mid-trajectory collapse" when a contradiction is introduced, only to recover by the final step. Endpoint-only evaluation completely misses these dangerous transient failures.
The representation study compares how different data formats impact reasoning.

Key Results

  • Grounding Gaps: Qwen-7B achieved a perfect 1.0 grounding rate, whereas Mistral-7B lagged at 0.694. When the authors looked only at cases where models did successfully ground the entity, the performance gap between weak and strong models narrowed significantly.
  • Relation Impact: Deleting semantic relations (the "measures" link) caused a 15.3 point drop in strict belief-state accuracy in relevant scenarios.
  • Trajectory vs. Endpoint: The study showed that a model could have high final-step accuracy while having a very low "trajectory score" due to errors in the middle of the evidence sequence.
  • Neural vs. Baseline: The best neural models (Qwen-7B and Gemma-12B) outperformed all six transparent fusion baselines, but still struggled with complex contradiction resolution compared to the normative reference.

Limitations and Future Work

  • Synthetic Nature: The scenarios are currently synthetic and involve binary states (normal vs. anomalous). Future work should move toward more complex, multi-entity state spaces.
  • Expert Consensus: The "expert audit" used to validate the scenarios relied on a single expert; a multi-expert study would strengthen the "human-plausibility" claims.
  • Model Scale: The study focused on models up to 12B parameters. Testing larger models or those with specialized tool-use (like RAG) could reveal different scaling behaviors in belief updating.
  • Real-world Replay: While the toolkit includes a pathway for replaying physical logs, the paper primarily evaluates the reasoning engine rather than the raw perception pipeline.

Summary updated

Abstract

Connected physical systems fuse observations from robots, IoT sensors, and edge-AI modules into a shared account of the world, yet most evaluations score only the final answer or action and cannot tell whether a method attributed the evidence, updated its belief, handled contradictions, and justified its decision. We argue that shared reasoning in Connected Robotics should be evaluated as an evidence-to-belief trajectory, and introduce the Connected Robotics Evidence Graph (CREG) Benchmark to make that trajectory measurable. CREG comprises 350 deterministically generated scenarios across seven regimes (justified revision, belief preservation, relation association, contradiction onset and resolution, stale/delayed evidence, and source reliability), each exposing a normative posterior reference at every evidence prefix, validated by an independent log-odds checker and screened by a blinded expert audit. Nine representation conditions separate relation content from serialization, ordering, and metadata. Evaluating four open-weight instruction models and six transparent fusion baselines yields three findings. First, entity grounding accounts for a substantial part of the weak-model deficit. Second, deleting semantic relations costs 15.3 points of strict belief-state accuracy on relation-dependent scenarios (Holm-corrected p<0.001), but an information-matched structured list outperforms the full graph and shuffling the graph does not hurt: relation content matters where required, while graph serialization confers no consistent advantage. Third, belief accuracy is strongly non-monotonic along the evidence sequence, with mid-trajectory failures that endpoint-only evaluation cannot detect. All scenarios, references, prompts, metrics, and analysis code are released for offline, hardware-free reproduction.
BibTeX
@misc{bezerra2026benchmark,
  author = {Ranulfo Bezerra and Kimitoshi Yamazaki},
  title = {{A Benchmark and Layered Evaluation Protocol for Evidence-Grounded Shared Belief Updating in Connected Robotics}},
  year = {2026},
  eprint = {2611.11842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11842},
}
Figure