MFSeg: Efficient Multi-Frame 3D Semantic Segmentation

Chengjie Huang, Krzysztof Czarnecki

Abstract

MFSeg is a novel framework that brings the accuracy benefits of multi-frame LiDAR data to 3D semantic segmentation without the massive computational "tax" of processing millions of points. By aggregating features using a mathematically grounded "homomorphism" approach and employing a selective point decoder, it achieves state-of-the-art results—especially for small, vulnerable road users—while maintaining the efficiency needed for real-world autonomous driving.

Introduction

In the field of autonomous driving, LiDAR sensors provide a critical 3D view of the environment. However, a single frame is often sparse and noisy. To improve the identification of objects like pedestrians or cyclists, researchers often concatenate multiple sequential frames to create a denser point cloud.

This approach introduces a significant computational bottleneck. A single frame from a high-end LiDAR contains approximately 177k points; stacking six frames results in over a million points. Modern state-of-the-art models, such as Point Transformer v3 (PTv3), struggle to process this volume in real-time. Furthermore, traditional U-Net architectures waste resources by upsampling and predicting labels for points in past frames, even though only the current frame's labels are required. MFSeg aims to bridge this gap, delivering the accuracy of multi-frame data with the speed of single-frame processing.

The MFSeg Architecture

The MFSeg framework utilizes a four-stage pipeline designed for maximum efficiency:

  1. Local Feature Extractor (LFE): Instead of feeding raw points into a heavy backbone, MFSeg voxelizes the cloud and uses a lightweight Voxel Feature Extractor (VFE) to capture essential local geometric details while reducing point density.
  2. Feature Aggregator: This module treats feature fusion as an algebraic operation (a "commutative monoid"). A symmetrized function ensures that the frame processing order does not affect the result, mimicking raw point concatenation within a compressed feature space.
  3. PTv3 Encoder: A powerful backbone that processes the aggregated features.
  4. Lightweight Point Decoder: Unlike standard U-Nets, MFSeg uses a simple MLP-based decoder that queries encoder features using K-Nearest Neighbors (KNN) only for points in the current frame, skipping redundant past-frame decoding.
The MFSeg pipeline overview, featuring the Local Feature Extractor (LFE), Feature Aggregator, and selective Point Decoder.

Key Innovations

Algebraic Regularization

By framing feature fusion as a mathematical homomorphism, the network learns to compress temporal data without losing the spatial relationships that make multi-frame data valuable. To ensure this fusion accurately represents raw point concatenation, a 3-way Cosine Similarity Loss is used during training to align features from original sequences, permuted sequences, and raw concatenated inputs.

Selective Decoding

A major efficiency gain stems from the realization that segmenting the current frame does not require decoding the entire historical sequence. By restricting decoding to the current frame, MFSeg significantly reduces the latency typically associated with multi-frame models.

Experimental Results

MFSeg was evaluated on the nuScenes (32-beam LiDAR) and Waymo Open Dataset (64-beam LiDAR) benchmarks.

Accuracy and Efficiency

On nuScenes, MFSeg achieved 81.3% mIoU, matching heavy 10-frame concatenation models while remaining significantly faster. On Waymo, it reached 72.1% mIoU, outperforming the single-frame PTv3 baseline. As shown in the figures below, MFSeg’s latency scales much more gracefully than standard PTv3 as the number of frames increases. For instance, with 10 frames on nuScenes, MFSeg is nearly 2x faster than the baseline (76.5ms vs 141.4ms).

Inference time vs. number of frames on nuScenes (left) and Waymo (right), demonstrating MFSeg's superior efficiency compared to PTv3 baselines.

Vulnerable Road User (VRU) Performance

Multi-frame context is particularly beneficial for small objects like bicycles and motorcycles, which often have very few points in a single frame. MFSeg showed a significant performance jump for these classes, with gains of +5.6% for bicycles and +4.4% for motorcycles compared to single-frame methods.

Conclusion

MFSeg demonstrates that multi-frame 3D segmentation can be both accurate and efficient. While the current model excels, future work may address the "KNN bottleneck" in the decoder by moving toward voxel-based association, further improving scalability for ultra-high-resolution sensors. Additionally, exploring global context within the feature aggregator could provide further performance gains.

Summary updated

BibTeX
@inproceedings{huang2025mfseg,
  author = {Chengjie Huang and Krzysztof Czarnecki},
  title = {{MFSeg: Efficient Multi-Frame 3D Semantic Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11127629},
}
Figure