PatchMixer: Rethinking network design to boost generalization for 3D point cloud understanding

Davide Boscaini, Fabio Poiesi

Introduction

Deep learning for 3D point clouds has seen a surge in architectural complexity, with researchers employing sophisticated graph convolutions and attention mechanisms to maximize performance on standard benchmarks. However, these models often struggle with "domain shift"—the performance gap encountered when a model trained on synthetic data (like CAD models) is applied to real-world sensor data (like Kinect or RealSense scans). PatchMixer addresses this by prioritizing generalization through a streamlined, MLP-based design that emphasizes local geometric understanding and semantic information mixing.

The PatchMixer Architecture

PatchMixer adapts the MLP-Mixer concept for 3D data, moving away from complex convolutions in favor of simple Multi-Layer Perceptrons (MLPs). The pipeline consists of three primary stages designed to create robust representations:

  1. Patch-based Processing: The input point cloud is decomposed into local patches. This approach provides inherent robustness against partiality and occlusions, which are common in real-world sensor data.
  2. Attentive Token Mixer: This module facilitates information exchange across patches. It utilizes a learnable attention mechanism to identify and aggregate semantically significant patches into a global shape descriptor.
  3. Channel Mixer: Operating within each patch, this module processes features to capture fine-grained local geometric details.
The PatchMixer architecture extracts local patches, embeds them, and then mixes information across patches and within features.

By relying primarily on MLPs, PatchMixer reduces inductive biases—preconceived notions about data structure—allowing the network to learn more flexible and transferable geometric representations that are less prone to overfitting on specific training distributions.

Evaluation and Benchmarks

The model's robustness was tested across several transfer learning (TL) scenarios, where models are trained on one dataset and tested on another:

  • GraspNetPC: Cross-sensor evaluation between Kinect2 and Intel RealSense data.
  • PointDA: Cross-dataset evaluation involving ModelNet (synthetic), ShapeNet (synthetic), and ScanNet (real-world indoor scenes).
  • PointSegDA: Cross-dataset part segmentation on human pose datasets like Adobe Fuse and FAUST.

Key Findings

PatchMixer demonstrates that architectural simplicity can lead to superior robustness. In transfer learning tasks on GraspNetPC, PatchMixer achieved an average accuracy of 80.0%, significantly outperforming more complex state-of-the-art models like PointMLP (73.5%) and PointTransformer (68.9%).

Furthermore, the model is highly efficient. It achieves these results with approximately 4 million parameters—roughly 4x fewer than its closest competitors. This efficiency makes it particularly suitable for edge devices or real-time applications where computational resources are limited.

PatchMixer achieves the highest accuracy in transfer learning scenarios while maintaining a low parameter count.

Qualitative analysis via t-SNE visualizations confirms that PatchMixer produces more distinct and tighter feature clusters for object categories compared to other models. This indicates a more robust feature extraction process, which is a hallmark of a model that has learned generalizable geometric principles rather than just memorizing a dataset.

Conclusion

PatchMixer represents a shift away from over-engineered 3D architectures toward designs that favor generalization and efficiency. While it maintains competitive in-domain performance, its true strength lies in its ability to handle real-world domain shifts with a fraction of the computational overhead. Future work will explore scaling the architecture for larger datasets and investigating more advanced patch selection strategies beyond Farthest Point Sampling.

Summary updated

BibTeX
@article{boscaini2023patchmixer,
  author = {Davide Boscaini and Fabio Poiesi},
  title = {{PatchMixer: Rethinking network design to boost generalization for 3D point cloud understanding}},
  journal = {Image and Vision Computing},
  year = {2023},
  doi = {10.1016/j.imavis.2023.104768},
}
Figure