Learning 3D Representations from Sequential View Trajectories

33 concepts19 papers

Imagine looking at a complex sculpture. If someone hands you a shuffled deck of photos taken from random angles, you can eventually piece together what the object looks like, but you lose the immediate, intuitive sense of how its shapes fluidly transform as you walk around it.

Traditional multi-view 3D neural networks do exactly this: they take a set of 2D projections, process them independently through a convolutional network, and then apply an order-independent pooling operation (such as max-pooling or average pooling) to collapse them into a single shape descriptor. As noted by Xuan et al., this conventional approach treats viewpoints as an unordered set and ignores the spatial correlations between successive frames. Max-pooling effectively treats a 3D object like an unordered "bag of images," discarding the continuous geometric information available along a camera path.

To capture the true topology of a 3D object, a model must understand how features evolve across a continuous trajectory. By treating multi-view images as an ordered sequence rather than an isolated collection, recent architectures leverage the spatial continuity of an object's surface to build more robust, discriminative 3D representations.

Recurrent Aggregation and Spatial Continuity

If we feed images to a network in the exact order they were captured along a viewing path, the model can track geometric transitions step-by-step. Recurrent neural networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs) are uniquely suited for this task because their hidden states update incrementally.

As a recurrent cell iterates through a sequence of viewpoints, it preserves a running memory of the object's geometry. Sepas-Moghaddam et al. demonstrated that this incremental update allows the network to capture dependencies between different perspectives that are entirely lost in parallel processing. This method preserves nuanced geometric data—such as how a surface curves or how an edge disappears—that order-independent pooling methods typically discard.

Unordered Pooling:  [View A] \
                    [View B] -> [Max/Avg Pool] -> Global Descriptor (Order Lost)
                    [View C] /

Sequential Flow:    [View A] -> [Recurrent Cell] -> [View B] -> [Recurrent Cell] -> [View C] -> Topology-Aware Descriptor

This sequential integration is particularly powerful for capturing long-term structural context, such as global object symmetry. Xu et al. utilized a Bidirectional LSTM (Bi-LSTM) to exploit these long-term view dependencies, making the network highly robust against variations in the camera's starting position. Wrapping this sequential pipeline in a Siamese network with contrastive loss, Dai et al. confirmed that maintaining the spatial continuity of the object's surface significantly outperforms static pooling techniques.

Furthermore, the structure of the sequence itself can be tailored to match the geometry. Jiang organized 2D views into a hierarchical structure of "rotation loops," using recurrent layers to process sequential views within a loop to capture the intrinsic topology of the 3D object more effectively than treating views as an unordered set.

Spatiotemporal Flow via 3D Convolutions

Another way to conceptualize a sequence of views is to treat it like a video. As an observer moves around an object, the spatial features change smoothly across the frames. Instead of using recurrent loops to update a hidden state over time, we can stack the ordered views into a single volume and use 3D convolutions to process the spatial and viewpoint dimensions simultaneously.

Guo et al. applied this concept by treating a sequence of 2D views as frames in a video, using a 3D convolutional neural network to capture the spatial and temporal relationships between adjacent perspectives. By analyzing the flow and appearance changes across a specific viewing trajectory, the model learns the global structure of an object without discarding its logical order.

This approach is formalized in the MV-C3D architecture by Xuan et al., which uses 3D convolutional kernels to process height, width, and viewpoint dimensions at the same time. This allows the network to capture exactly how specific object features evolve across a camera trajectory.

A major practical benefit of this approach is its efficiency with partial data: because the 3D kernels rely on local spatial correlations between adjacent frames, the model excels even when given only a short, contiguous segment of viewpoints rather than a full 360-degree orbit. This sequential integration concept also extends to volumetric data; for instance, Abdul Kareem et al. demonstrated that the specific ordering of directional attention operations (such as processing depth before horizontal and vertical axes) is vital for progressively building a coherent structural understanding of how objects extend through space.

Predictive Self-Supervision and Global Memories

How do we force a network to develop a true understanding of 3D geometry from a sequence of views without relying on expensive, manually labeled datasets? The sequential nature of the data offers a built-in solution: self-supervised predictive tasks. By forcing a network to predict missing parts of a sequence, it must internalize the underlying 3D structure.

Several unsupervised frameworks leverage this insight:

  • Unseen View Estimation: Xu et al. introduced a generative task where the model must predict what an object looks like from a completely new angle based on its current recurrent state. This forces the model to integrate spatial transitions into a unified 3D descriptor that remains robust even when views are missing or occluded.
  • View Inter-Prediction: Han et al. developed VIP-GAN, which tasks a network with predicting "the view in the middle" of a sequence. By learning global shape memories to support local view predictions, the recurrent architecture implicitly captures the spatial flow between adjacent views.
  • Half-to-Half Prediction: In the point cloud domain, Han et al. split point clouds into halves from multiple angles, using an RNN to integrate features from a sequence of partial viewpoints to predict the hidden back half from the visible front half.

The geometric continuity of the trajectory itself can resolve fundamental ambiguities. Kuang et al. represented 3D models using "loop views"—circular sequences of images captured along a closed 360-degree path. Because the sequence has no abrupt breaks, the model can easily map the relationships between adjacent viewpoints, providing a highly informative structure for unsupervised similarity matching.

Scaling to Synthesis and Multimodal Representations

When a model successfully masters the continuous flow of geometry, it can do more than just classify shapes—it can synthesize novel views and align 3D structures with language.

In free view synthesis, Riegler et al. utilized a convolutional gated recurrent unit (GRU) to sequentially process warped features from an arbitrary number of source views. This recurrent setup allows the model to handle a variable number of inputs and progressively improve its blending and inpainting strategies as more source information is integrated. For dynamic objects, Ni et al. used a Convolutional LSTM to aggregate features across multiple video frames, constructing a high-fidelity 3D canonical representation of talking heads that preserves fine-grained spatial details better than simple temporal averaging.

Finally, this sequential paradigm has been integrated into multimodal foundation models. The JM3D framework, developed by Wang et al. and extended by Ji et al., represents 3D objects using continuous sequences of multi-view 2D renders augmented with explicit angle and depth encodings. By sampling views in a continuous flow rather than random snapshots, the model preserves semantic continuity and prevents the confusion caused by abrupt perspective shifts. This allows the network to capture spatial depth and fine-grained details, enabling seamless alignment with hierarchical text and large language models.

Key Foundational Work

Siamese CNN-BiLSTM Architecture for 3D Shape Representation Learning (2018) — contributed 3 concept elements to this direction.
Emphasizing 3D Properties in Recurrent Multi-View Aggregation for 3D Shape Retrieval (2018) — contributed 2 concept elements to this direction.
Weakly Supervised 3D Reconstruction with Adversarial Constraint (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_view_aggregation_f6350be5,
  title = {sequential view aggregation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-view-aggregation/rdr_f6350be5593447e88489d721c157c4c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_view_aggregation_f6350be5_v1,
  title = {sequential view aggregation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-view-aggregation/rdr_f6350be5593447e88489d721c157c4c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake