Learning Interaction-Aware Path Corrections from Unlabeled Trajectories: A LiDAR-Based Sim-to-Real GNN Framework for Construction Robots

Yilong Chen, Yong K. ChoarXiv 2026 · arXiv

TL;DR

This paper introduces a LiDAR-only, sim-to-real framework that enables construction robots to navigate socially around workers by learning path corrections from unlabeled human demonstrations. By using a Graph Neural Network (GNN) to model agent interactions and a self-supervised "constant-velocity residual" as a learning target, the system provides a planner-agnostic way to improve safety and smoothness without requiring manually labeled data.

What Problem Does This Paper Address?

Construction sites are the "wild west" of robotics. Unlike structured factories, they are cluttered, constantly evolving, and shared by workers performing complex, task-driven movements. For a robot to be a good "coworker," it needs social navigation: the ability to anticipate human motion and avoid disruptive or unsafe behavior.

The challenge is three-fold:

  1. Data Scarcity: Labeled examples of "socially correct" navigation are rare and expensive to produce, especially for specific construction scenarios.
  2. Generalization: Hand-crafted rules (like Social Force Models) often fail to transfer across different site layouts or robot platforms.
  3. Perception Gaps: Real-world LiDAR data is noisy. Converting raw point clouds into the stable, tracked object identities needed for sophisticated interaction modeling is a significant hurdle.

Methods

The authors propose a modular framework that separates perception, interaction modeling, and path execution.

Figure 1: Overview of the proposed interaction-aware path correction framework. During training, residual targets are automatically generated from unlabeled human demonstrations using constant-velocity extrapolation, and a GNN learns relationships among the agents after scenario-to-graph translation. During deployment, Dynablox detections from LiDAR point clouds are associated and tracked using EKFs to construct dynamic graphs online. The trained GNN then predicts spatial corrections to refine planner- generated local waypoints in real time.

The "Secret Sauce": Constant-Velocity Residuals

Instead of requiring humans to label "good" paths, the researchers used Self-Supervised Learning (SSL). They looked at how human operators steered robots in simulation. By comparing the actual human-steered path to a simple "constant-velocity" (CV) extrapolation of the robot's previous state, they calculated a spatial residual—essentially the "correction" a human made to avoid an obstacle or worker.

Figure 4: Constant-velocity residual.

Interaction Modeling with GNNs

The framework translates the scene into a graph where the robot, workers, and the navigation goal are nodes, and their spatial relationships are edges. A Graph Neural Network (GNN) processes this graph to predict the necessary path offset. Because GNNs handle a variable number of nodes, the system scales naturally as more workers enter the scene.

Real-World Deployment

To bridge the sim-to-real gap, the authors developed a LiDAR-only pipeline. It uses Dynablox for dynamic object detection and an Extended Kalman Filter (EKF) to track those objects over time, providing the stable velocity estimates the GNN requires. Crucially, the GNN outputs a correction to an existing local planner's path, making it "planner-agnostic" and easy to integrate into existing systems.

Datasets

  • Simulated HRI Patterns: 74 trials (36 used for training) conducted in Unity 3D, featuring scenarios like path crossing, merging, and overtaking.
  • Bricklaying Trial: A custom real-world testbed where a quadruped robot navigated a cross-shaped corridor while workers transported bricks.
  • Dynablox Dataset: A public, real-world LiDAR benchmark featuring diverse environments like ramps, stairs, and corridors to test cross-domain generalization.
Figure 7: Dynablox dataset: (a) corridor, (b) hg, (c) ramp, and (d) stairs.

Key Insights and Ideas

  • Planner-Agnosticism is a Feature: By predicting spatial residuals (offsets) rather than direct control commands (like velocity), the model can be "plugged in" to different robots and local planners without retraining the whole stack.
  • Goal Context Matters: The ablation studies showed that including the "navigation goal" as a node in the graph is vital. It helps the GNN distinguish between a robot turning because it's avoiding a worker versus a robot turning because it's reaching its destination.
  • Symmetry Augmentation: Flipping training data (reflection augmentation) significantly improved the model's ability to handle interaction directions it hadn't seen frequently in the small demonstration dataset.
  • The 1.0s Sweet Spot: Testing different prediction intervals for the CV residual revealed that 1.0 second provides the best balance. Shorter intervals get lost in sensor noise; longer ones accumulate too much error from the simple constant-velocity assumption.

Key Results

The GNN-based framework was compared against a standard Social Force Model (SFM) and a non-graph Multi-Layer Perceptron (MLP) baseline.

  • Bricklaying Trial: The GNN reduced path irregularity by 39.2% compared to the original path, resulting in much smoother trajectories. It also increased the minimum Time-to-Collision (TTC), indicating safer, more anticipatory behavior.
  • Dynablox Dataset: The framework increased the minimum TTC by 38.3% over the original trajectories.
  • Efficiency vs. Safety: While the GNN slightly increased path length (by ~6-7%) to navigate around people, it produced shorter and smoother paths than the SFM and MLP baselines while maintaining superior safety margins.
Figure 6: Bricklaying Experiment: (a) testbed layout and (b) quadruped robot with LiDAR sensor.

Limitations and Future Work

While successful, the authors note that the constant-velocity assumption is a simplification; it doesn't perfectly capture sudden accelerations or complex maneuvers. Furthermore, the evaluation relied on objective metrics (distance, TTC). Future work aims to include subjective human-centered metrics, such as how comfortable or "natural" the workers feel when the robot is nearby. The researchers also plan to test the framework across a wider variety of robot platforms and more chaotic construction tasks.

Summary updated

Abstract

Construction robots operating near workers require navigation that can anticipate human motion while remaining robust to dynamic and unstructured site conditions. However, existing social navigation methods depend on hand-crafted interaction rules, labeled demonstrations, or robot-specific control policies, limiting their scalability and transferability in construction environments. We propose a real-time, LiDAR-only framework that learns interaction-aware path corrections from unlabeled demonstrations. Constant-velocity extrapolation generates residual learning targets, while a GNN models relationships among the robot, pedestrians, and goal context. During deployment, an extended Kalman filter tracks LiDAR detections to construct dynamic graphs, and the GNN offsets planner-generated waypoints through a planner-agnostic interface. Within the bricklaying trial, the GNN reduced path irregularity by 39.2\% and increased minimum TTC. Across Dynablox sequences, it increased minimum TTC by 38.3\% while producing shorter, smoother paths than the baselines. Ablation studies supported a 1.0-s prediction interval and confirmed the benefits of target context, reflection augmentation, and feature normalization.
BibTeX
@misc{chen2026learning,
  author = {Yilong Chen and Yong K. Cho},
  title = {{Learning Interaction-Aware Path Corrections from Unlabeled Trajectories: A LiDAR-Based Sim-to-Real GNN Framework for Construction Robots}},
  year = {2026},
  eprint = {2610.34060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34060},
}
Figure