Introduction
Representing 3D shapes efficiently is a fundamental challenge in computer vision. Traditionally, researchers have relied on two primary methods: Voxel Grids and Implicit Functions. Voxel grids are intuitive and compatible with standard convolutional neural networks, but their memory requirements scale cubically ($O(N^3)$), making high-resolution modeling prohibitively expensive. Conversely, implicit functions (like Occupancy Networks) offer memory efficiency by representing shapes as continuous boundaries. However, they require dense, irregular sampling during training and rely on computationally heavy algorithms like "marching cubes" for final reconstruction.
SeqXY2SeqZ introduces a novel middle ground by treating 3D shapes as 2D functions. Instead of evaluating a full 3D volume, the model predicts 1D "occupancy segments" along a tube for any given 2D coordinate. This approach achieves state-of-the-art accuracy while significantly reducing memory and time requirements.
Methods: Voxel Tubelization and Seq2Seq Modeling
The core innovation of this work is Voxel Tubelization. The authors conceptualize a 3D voxel grid as a collection of parallel "tubes" aligned with a specific axis (e.g., the Z-axis). Each tube is uniquely identified by its 2D coordinate $(x, y)$. Within each tube, the shape is represented as a sequence of occupied segments, defined by their start ($s$) and end ($e$) points.
To model these tubes, the authors propose a Sequence-to-Sequence (Seq2Seq) architecture with an attention mechanism:
- Encoder: This component processes a 2D coordinate $(x, y)$ alongside a global shape feature (derived from an image or latent code). It utilizes learnable Location Embeddings to transform discrete coordinates into high-dimensional feature vectors.
- Decoder: A Recurrent Neural Network (RNN), specifically a Gated Recurrent Unit (GRU), sequentially predicts the 1D boundaries of every occupied segment within the target tube. It also outputs a global occupancy flag ($b$) to efficiently skip empty regions.
- Attention Mechanism: This allows the decoder to focus on specific spatial features within the encoded 2D domain, ensuring precise placement of the 1D segments.
By predicting only the boundaries, the model performs a learnable form of Run-Length Encoding (RLE), effectively reducing the complexity of the reconstruction process from $O(R^3)$ to $O(R^2)$.
Key Insights
- Dimensionality Reduction: The primary breakthrough is the realization that 3D structure can be decomposed into a series of 1D problems mapped over a 2D domain, bypassing the need for exhaustive 3D volume sampling.
- Shareable Spatial Features: The research demonstrates that the same location embedding matrix can be used across the X, Y, and Z axes. This suggests the network learns generalized spatial representations that are not strictly tied to a single dimension.
- Threshold-Free Reconstruction: Unlike implicit models that require a manually tuned probability threshold to define shape boundaries, SeqXY2SeqZ directly predicts coordinate values, simplifying the inference pipeline.
Experimental Results
The model was rigorously evaluated using the ShapeNetCore dataset across tasks including auto-encoding and single-image 3D reconstruction.
Quantitative Performance
SeqXY2SeqZ consistently outperformed established baselines such as Occupancy Networks (OccNet) and the Implicit Decoder (IM-AE). In auto-encoding tasks for the "Chair" category, SeqXY2SeqZ achieved an IoU (Intersection over Union) of 74.32, a substantial improvement over IM-AE's 65.65. For single-image reconstruction across 13 categories, it reached a mean IoU of 63.6, surpassing both supervised methods like Pixel2Mesh and unsupervised approaches like DIB-R.
Qualitative Results
Visual comparisons demonstrate the model's ability to capture fine structural details across diverse object categories, including complex geometries like rifles and chairs.
Efficiency Benchmarks
The model's efficiency is one of its most compelling features. For a $64^3$ resolution reconstruction, SeqXY2SeqZ required only 286MB of RAM, whereas competing models like DISN required over 11GB. Furthermore, the model proved to be approximately 6x faster than OccNet during CPU-based inference (8.79s vs 55.80s).
Conclusion and Future Work
SeqXY2SeqZ provides a highly efficient and accurate alternative to traditional 3D shape representation methods. While the current implementation focuses on tubelization along a single axis, the results show the model is robust regardless of the chosen orientation. Future research may explore multi-axis ensembles to capture even finer details in thin-walled structures or extend this sequential coordinate prediction to dynamic 3D environments and point cloud generation.
Summary updated