In the world of connected robotics, a single robot rarely acts alone. Instead, a fleet of mobile robots, IoT sensors, and edge-AI modules must fuse their observations into a "shared belief" about the environment. But how do we know if a system is actually reasoning correctly, or just getting lucky with its final action?
A new paper from researchers at Tohoku University introduces the Connected Robotics Evidence Graph (CREG) Benchmark, a framework designed to peek inside the "black box" of robotic belief updating. By moving beyond simple success/failure metrics, CREG provides a layered look at how AI models handle the messy, contradictory, and delayed evidence typical of real-world deployments.
TL;DR
CREG is a new benchmark that evaluates the entire "evidence-to-belief" trajectory in connected robotics, rather than just the final action. It reveals that many AI models fail to "ground" observations to the correct physical entities and that while semantic relations are crucial for accuracy, complex graph formatting offers no inherent advantage over simple structured lists.
What Problem Does This Paper Address?
Most existing AI benchmarks for robotics use "endpoint evaluation"—they check if the robot reached the goal or picked the right object. However, in a connected system where data comes from multiple sources with varying reliability, a robot might take the "right" action for the "wrong" reason. For example, it might ignore a critical sensor contradiction or attribute a failure to the wrong machine.
The authors argue that we need to evaluate the belief trajectory: the step-by-step process of how a system’s internal world model changes as new evidence arrives. This is critical for safety-critical applications where we need to trust that the system is handling uncertainty and contradictions logically.
Methods
The researchers developed a suite of 350 scenarios generated from a "normative specification." This means they created a mathematical "ground truth" for what a perfect Bayesian reasoner should believe at every single step of a data stream.
Key methodological components include:
- Layered Evaluation: Instead of one score, the benchmark tracks entity grounding, evidence attribution, belief revision, contradiction handling, and action selection.
- Representation Ablations: They tested how different data formats—Canonical JSON, structured lists, and full graphs—impacted performance while keeping the underlying information identical.
- Model Panel: They evaluated several open-weight Large Language Models (LLMs), including Qwen2.5 (1.5B and 7B), Mistral-7B, and Gemma-3-12B, alongside six transparent fusion baselines (like majority voting and recency-weighting).
Datasets
The primary contribution is the CREG Benchmark, which consists of:
- 350 Scenarios: Covering seven reasoning regimes such as "justified revision" (changing your mind when evidence is strong) and "stale evidence" (ignoring old data that has been superseded).
- 1,150 Prefix Pairs: Scored reference points that allow for step-by-step trajectory analysis.
- Comparison to Prior Work: CREG is compared against general belief-revision benchmarks like BeliefTrack and BayesBench, but it is unique in its focus on physical robotic entities and time-stamped, attributed evidence.
Key Insights and Ideas
- The Grounding Deficit: A major finding was that "weaker" models (like Mistral-7B or Qwen-1.5B) don't necessarily fail at logic; they fail at grounding. They often produce a belief update that isn't bound to the required physical entity (e.g., they talk about "the sensor" generally rather than "sensor-a").
- Content Over Container: While semantic relations (knowing that "Sensor A measures Tank B") are vital for accuracy, the format doesn't seem to matter. A simple structured list performed just as well as, or better than, a complex graph serialization.
- Non-Monotonic Failures: Belief accuracy isn't a straight line. Models often experience a "mid-trajectory collapse" when a contradiction is introduced, only to recover by the final step. Endpoint-only evaluation completely misses these dangerous transient failures.
Key Results
- Grounding Gaps: Qwen-7B achieved a perfect 1.0 grounding rate, whereas Mistral-7B lagged at 0.694. When the authors looked only at cases where models did successfully ground the entity, the performance gap between weak and strong models narrowed significantly.
- Relation Impact: Deleting semantic relations (the "measures" link) caused a 15.3 point drop in strict belief-state accuracy in relevant scenarios.
- Trajectory vs. Endpoint: The study showed that a model could have high final-step accuracy while having a very low "trajectory score" due to errors in the middle of the evidence sequence.
- Neural vs. Baseline: The best neural models (Qwen-7B and Gemma-12B) outperformed all six transparent fusion baselines, but still struggled with complex contradiction resolution compared to the normative reference.
Limitations and Future Work
- Synthetic Nature: The scenarios are currently synthetic and involve binary states (normal vs. anomalous). Future work should move toward more complex, multi-entity state spaces.
- Expert Consensus: The "expert audit" used to validate the scenarios relied on a single expert; a multi-expert study would strengthen the "human-plausibility" claims.
- Model Scale: The study focused on models up to 12B parameters. Testing larger models or those with specialized tool-use (like RAG) could reveal different scaling behaviors in belief updating.
- Real-world Replay: While the toolkit includes a pathway for replaying physical logs, the paper primarily evaluates the reasoning engine rather than the raw perception pipeline.
Summary updated