MVSTF: Multi-View Spatio-Temporal Fusion for LiDAR-Based Moving Object Segmentation

Jiaoju Zhou, Chao Ye, Xuebo Yang, Weiyang LinarXiv 2026 · arXiv

TL;DR

MVSTF (Multi-View Spatio-Temporal Fusion) is a novel LiDAR-based moving object segmentation (MOS) framework that fuses features from raw point clouds, Range View (RV), and Bird’s-Eye View (BEV) representations. By explicitly modeling motion through feature differences in the BEV domain and utilizing a multi-branch architecture with auxiliary semantic supervision, it achieves state-of-the-art performance on the SemanticKITTI-MOS and KITTI-Road benchmarks.

What Problem Does This Paper Address?

For autonomous robots, distinguishing between moving objects (like a driving car) and static ones (like a parked car or a building) is essential for safe navigation and mapping. This task, known as Moving Object Segmentation (MOS), is notoriously difficult due to the inherent sparsity and disorder of LiDAR point clouds.

Existing methods typically project 3D data into a single 2D representation to use standard computer vision techniques. However, each projection has a "blind spot": Range Views (RV) suffer from spherical distortion and scale changes, while Bird’s-Eye Views (BEV) struggle with occlusions and uneven point density. Furthermore, many models lack explicit temporal modeling, making it hard to capture subtle motion cues for distant or small objects.

Methods

The authors propose MVSTF, a unified framework that processes sequential point clouds through three parallel branches to exploit the complementary strengths of different views.

1. Multi-Branch Architecture

As shown in the system overview (Figure 3), the network consists of:

  • Point-based Branch: Uses a shared MLP to extract fine-grained geometric features directly from raw points.
  • BEV Branch: Processes a concatenation of the current BEV image and Feature Differences from historical frames.
  • RV Branch: Processes the current range image to capture spherical geometric context.
Fig. 3. Overview of MVSTF. Multi-view feature encoding: We convert the point cloud into BEV and RV images, perform ego-motion compensation on the historical frames, and calculate BEV feature differences. Multi-Branch Networks: The Fully Convolutional Network (FCN) consists of Dual Down-sampling Blocks (DDB), Spatial Attention Blocks (SAB), Multi-Kernel Convolutional Neural Network blocks (MKCNN), Dual Feature Fusion Blocks (DFFB), and Point Fusion (PF) modules to enable end-to-end network training. In the training phase, the auxiliary semantic head promotes parameter convergence and remarkably improves network performance, and we remove the auxiliary semantic head in the inference phase. The BEV feature difference module takes the current and historical BEV feature maps as input, computes their differences, and outputs the concatenation of the current-frame features and the feature differences, which is then fed into the BEV branch.

2. Key Innovations

  • Explicit Motion Modeling: Instead of simple geometric residuals (like depth subtraction), MVSTF computes differences in the feature space after an MLP embedding. This "Feature Difference" strategy more effectively cancels out static background noise while highlighting motion.
  • Multi-Kernel CNN (MKCNN): To handle objects of varying sizes and distances, the authors designed the MKCNN block (Figure 4). It uses directionally decomposed convolutions (e.g., 1×7 and 7×1) and multiple kernel sizes to expand the receptive field without massive computational overhead.
  • Point Fusion (PF) Module: This module maps 2D features from the RV and BEV branches back to the original 3D points using bilinear interpolation, enabling a fully end-to-end pipeline that avoids the need for heuristic post-processing like K-Nearest Neighbors (KNN).
  • Auxiliary Semantic Head: During training, the model is tasked with performing both MOS and semantic segmentation (identifying object classes). This "Two Head" structure injects structural priors—knowing an object is a "pedestrian" helps the model decide if it is likely to move—speeding up convergence and improving robustness.
Fig. 4. The Multi-Kernel Convolutional Neural Network block (MKCNN).

Datasets

The model was evaluated on two primary benchmarks:

  • SemanticKITTI-MOS: The standard benchmark for LiDAR-based moving object segmentation.
  • KITTI-Road: Used to demonstrate the model's generalization capabilities across different environments.

Key Insights and Ideas

  • View Complementarity: The authors found that BEV is superior for capturing planar motion trajectories, while RV is better at preserving near-field object geometry. Fusing them (Figure 2) significantly reduces boundary blurring and ghosting artifacts.
  • Feature vs. Geometry: Subtracting learned features is more robust than subtracting raw coordinates or depths. Feature-level residuals are less sensitive to the noise and sparsity inherent in raw LiDAR scans.
  • The Power of Semantics: Even though the final goal is just to detect motion, learning semantic classes (car, pole, sidewalk) provides the network with "common sense" about the world's structure, which directly benefits motion detection.
Fig. 2. The Multi-view representations: range view and bird’s-eye view.

Key Results

  • State-of-the-Art Performance: MVSTF achieved 84.5% IoU on the SemanticKITTI-MOS validation set, outperforming recent high-performance models like MambaMOS (82.3%) and LiDAR-IMU-GNSS (79.0%).
  • Long-Range Accuracy: In scenarios beyond 50 meters, where point density is extremely low, MVSTF maintained a significant performance lead over single-view methods like LMNet and MotionSeg3D.
  • Real-World Impact: When integrated into a SLAM system (SuMa++), MVSTF reduced translational errors by approximately 35%, proving its utility for improving autonomous vehicle localization.

Limitations and Future Work

The primary trade-off in MVSTF is computational complexity. The "Feature Difference" mode, while highly accurate, has an average inference time of 238.92ms, which is slower than the 10Hz (100ms) scan rate of standard LiDAR sensors. The authors provide a faster "Height Residual" mode (94.99ms) for real-time use, but it comes with a performance drop (~5% IoU).

Future work will focus on:

  1. Multi-sensor fusion: Incorporating camera data for better texture and semantic understanding.
  2. Model Compression: Developing lightweight versions of the network for deployment on edge computing platforms.
  3. Environmental Robustness: Improving performance in adverse weather conditions like rain or snow.

Summary updated

Abstract

Autonomous mobile robots confront significant chal lenges in recognizing the motion properties of objects in dy namic environments. However, existing methods are hindered by limitations such as boundary ambiguity, occlusion, viewpoint constraints, and training complexity. In this paper, we propose an innovative end-to-end Multi-View Spatio-Temporal Fusion network, named MVSTF, which utilizes sequential point clouds to generate multi-view images and feature differences, and designs a multi-branch network to extract spatio-temporal and dimen sional features in parallel, and realizes high-precision moving object segmentation. First, sequential point clouds are utilized to generate multi-view images and feature differences. Then, an end-to-end multi-branch network including Dual Down-sampling Blocks (DDB), Spatial Attention Blocks (SAB), Multi-Kernel Convolutional Neural Network blocks (MKCNN), Dual Feature Fusion Blocks (DFFB), and the Point Fusion (PF) module, is designed to extract motion features from feature differences and geometric and semantic features from multi-view images. Finally, an auxiliary semantic head is introduced to further improve the algorithm's robustness and the model's interpretability. This work is validated on the SemanticKITTI and KITTI-Road bench marks, achieving state-of-the-art results on the SemanticKITTI MOS validation set and the KITTI-Road validation set, and competitive results on the hidden SemanticKITTI-MOS test set. The code is available at https://github.com/ZhouJJ111/MVSTF.
BibTeX
@misc{zhou2026mvstf,
  author = {Jiaoju Zhou and Chao Ye and Xuebo Yang and Weiyang Lin},
  title = {{MVSTF: Multi-View Spatio-Temporal Fusion for LiDAR-Based Moving Object Segmentation}},
  year = {2026},
  eprint = {2611.20872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20872},
}
Figure