SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments from 2D Coordinates

Zhizhong Han, Guanhui Qiao, Yu-Shen Liu, Matthias Zwicker

Introduction

Representing 3D shapes efficiently is a fundamental challenge in computer vision. Traditionally, researchers have relied on two primary methods: Voxel Grids and Implicit Functions. Voxel grids are intuitive and compatible with standard convolutional neural networks, but their memory requirements scale cubically ($O(N^3)$), making high-resolution modeling prohibitively expensive. Conversely, implicit functions (like Occupancy Networks) offer memory efficiency by representing shapes as continuous boundaries. However, they require dense, irregular sampling during training and rely on computationally heavy algorithms like "marching cubes" for final reconstruction.

SeqXY2SeqZ introduces a novel middle ground by treating 3D shapes as 2D functions. Instead of evaluating a full 3D volume, the model predicts 1D "occupancy segments" along a tube for any given 2D coordinate. This approach achieves state-of-the-art accuracy while significantly reducing memory and time requirements.

Methods: Voxel Tubelization and Seq2Seq Modeling

The core innovation of this work is Voxel Tubelization. The authors conceptualize a 3D voxel grid as a collection of parallel "tubes" aligned with a specific axis (e.g., the Z-axis). Each tube is uniquely identified by its 2D coordinate $(x, y)$. Within each tube, the shape is represented as a sequence of occupied segments, defined by their start ($s$) and end ($e$) points.

Figure 1: Overview of the SeqXY2SeqZ framework, illustrating how 3D voxels are decomposed into 1D segments predicted by an RNN.

To model these tubes, the authors propose a Sequence-to-Sequence (Seq2Seq) architecture with an attention mechanism:

  1. Encoder: This component processes a 2D coordinate $(x, y)$ alongside a global shape feature (derived from an image or latent code). It utilizes learnable Location Embeddings to transform discrete coordinates into high-dimensional feature vectors.
  2. Decoder: A Recurrent Neural Network (RNN), specifically a Gated Recurrent Unit (GRU), sequentially predicts the 1D boundaries of every occupied segment within the target tube. It also outputs a global occupancy flag ($b$) to efficiently skip empty regions.
  3. Attention Mechanism: This allows the decoder to focus on specific spatial features within the encoded 2D domain, ensuring precise placement of the 1D segments.

By predicting only the boundaries, the model performs a learnable form of Run-Length Encoding (RLE), effectively reducing the complexity of the reconstruction process from $O(R^3)$ to $O(R^2)$.

Key Insights

  • Dimensionality Reduction: The primary breakthrough is the realization that 3D structure can be decomposed into a series of 1D problems mapped over a 2D domain, bypassing the need for exhaustive 3D volume sampling.
  • Shareable Spatial Features: The research demonstrates that the same location embedding matrix can be used across the X, Y, and Z axes. This suggests the network learns generalized spatial representations that are not strictly tied to a single dimension.
  • Threshold-Free Reconstruction: Unlike implicit models that require a manually tuned probability threshold to define shape boundaries, SeqXY2SeqZ directly predicts coordinate values, simplifying the inference pipeline.

Experimental Results

The model was rigorously evaluated using the ShapeNetCore dataset across tasks including auto-encoding and single-image 3D reconstruction.

Quantitative Performance

SeqXY2SeqZ consistently outperformed established baselines such as Occupancy Networks (OccNet) and the Implicit Decoder (IM-AE). In auto-encoding tasks for the "Chair" category, SeqXY2SeqZ achieved an IoU (Intersection over Union) of 74.32, a substantial improvement over IM-AE's 65.65. For single-image reconstruction across 13 categories, it reached a mean IoU of 63.6, surpassing both supervised methods like Pixel2Mesh and unsupervised approaches like DIB-R.

Qualitative Results

Visual comparisons demonstrate the model's ability to capture fine structural details across diverse object categories, including complex geometries like rifles and chairs.

Figure 3: Qualitative results comparing SeqXY2SeqZ reconstructions (Ours) against Ground Truth (GT) for various ShapeNet categories.

Efficiency Benchmarks

The model's efficiency is one of its most compelling features. For a $64^3$ resolution reconstruction, SeqXY2SeqZ required only 286MB of RAM, whereas competing models like DISN required over 11GB. Furthermore, the model proved to be approximately 6x faster than OccNet during CPU-based inference (8.79s vs 55.80s).

Conclusion and Future Work

SeqXY2SeqZ provides a highly efficient and accurate alternative to traditional 3D shape representation methods. While the current implementation focuses on tubelization along a single axis, the results show the model is robust regardless of the chosen orientation. Future research may explore multi-axis ensembles to capture even finer details in thin-walled structures or extend this sequential coordinate prediction to dynamic 3D environments and point cloud generation.

Summary updated

Abstract

Structure learning for 3D shapes is vital for 3D computer vision. State-of-the-art methods show promising results by representing shapes using implicit functions in 3D that are learned using discriminative neural networks. However, learning implicit functions requires dense and irregular sampling in 3D space, which also makes the sampling methods affect the accuracy of shape reconstruction during test. To avoid dense and irregular sampling in 3D, we propose to represent shapes using 2D functions, where the output of the function at each 2D location is a sequence of line segments inside the shape. Our approach leverages the power of functional representations, but without the disadvantage of 3D sampling. Specifically, we use a voxel tubelization to represent a voxel grid as a set of tubes along any one of the X, Y, or Z axes. Each tube can be indexed by its 2D coordinates on the plane spanned by the other two axes. We further simplify each tube into a sequence of occupancy segments. Each occupancy segment consists of successive voxels occupied by the shape, which leads to a simple representation of its 1D start and end location. Given the 2D coordinates of the tube and a shape feature as condition, this representation enables us to learn 3D shape structures by sequentially predicting the start and end locations of each occupancy segment in the tube. We implement this approach using a Seq2Seq model with attention, called SeqXY2SeqZ, which learns the mapping from a sequence of 2D coordinates along two arbitrary axes to a sequence of 1D locations along the third axis. SeqXY2SeqZ not only benefits from the regularity of voxel grids in training and testing, but also achieves high memory efficiency. Our experiments show that SeqXY2SeqZ outperforms the state-of-the-art methods under the widely used benchmarks.
BibTeX
@inproceedings{han2020seqxy,
  author = {Zhizhong Han and Guanhui Qiao and Yu-Shen Liu and Matthias Zwicker},
  title = {{SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments from 2D Coordinates}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58586-0\_36},
}
Figure