Abstract
Despite significant recent progress, the best available computer vision algorithms still lag far behind human capabilities, even for recognizing individual discrete objects under various poses, illuminations, and backgrounds. Here we present a new approach to using object pose information to improve deep network learning. While existing large-scale datasets, e.g. ImageNet, do not have pose information, we leverage the newly published turntable dataset, iLab-20M, which has ~22M images of 704 object instances shot under different lightings, camera viewpoints and turntable rotations, to do more controlled object recognition experiments. We introduce a new convolutional neural network architecture, what/where CNN (2W-CNN), built on a linear-chain feedforward CNN (e.g., AlexNet), augmented by hierarchical layers regularized by object poses. Pose information is only used as feedback signal during training, in addition to category information; during test, the feedforward network only predicts category. To validate the approach, we train both 2W-CNN and AlexNet using a fraction of the dataset, and 2W-CNN achieves 6% performance improvement in category prediction. We show mathematically that 2W-CNN has inherent advantages over AlexNet under the stochastic gradient descent (SGD) optimization procedure. Further more, we fine-tune object recognition on ImageNet by using the pretrained 2W-CNN and AlexNet features on iLab-20M, results show that significant improvements have been achieved, compared with training AlexNet from scratch. Moreover, fine-tuning 2W-CNN features performs even better than fine-tuning the pretrained AlexNet features. These results show pretrained features on iLab-20M generalizes well to natural image datasets, and 2W-CNN learns even better features for object recognition than AlexNet.
Introduction
Standard Convolutional Neural Networks (CNNs) are typically designed to achieve "pose-invariance" by discarding spatial "where" information (such as orientation and lighting) through successive layers of convolution and pooling. The goal is to focus exclusively on the "what"—the object category. However, this approach is counter-intuitive to biological vision systems, where spatial awareness and object identity are often processed in parallel. Discarding pose information too early may prevent a network from learning the underlying structural relationships that define an object's category across different viewpoints.
This paper addresses the challenge of utilizing pose information to learn superior feature representations without requiring that pose information during actual deployment (test time). We propose the What/Where CNN (2W-CNN), which treats object pose as an auxiliary training signal to regularize the learning process.
The 2W-CNN Architecture
The 2W-CNN augments a standard backbone, such as AlexNet, with auxiliary branches dedicated to predicting object pose. We explore two primary variants of this architecture:
- 2W-CNN-I (Injection): Pose information is linked only to the final fully connected layer (fc7), providing a late-stage regularization signal.
- 2W-CNN-MI (Multi-layer Injection): This more advanced version provides direct backpropagation pathways from pose labels to all convolutional layers, ensuring that even low-level features are influenced by the spatial configuration of the object.
During training, the network minimizes a joint loss function: $L = L(object) + \lambda L(pose)$. This forces intermediate layers to learn features that are simultaneously useful for identification and orientation. Crucially, once training is complete, the auxiliary pose branches are pruned. The resulting "what" network is identical in size and computational cost to a standard AlexNet but possesses superior internal feature representations.
Experimental Setup and Datasets
To evaluate the 2W-CNN, we utilized the iLab-20M dataset, a massive "turntable" collection containing ~22 million images of 704 object instances (e.g., cars, boats, planes). These images were captured under controlled variations of 11 camera azimuths, 8 turntable rotations, 5 lighting conditions, and 3 focus levels. This controlled environment allows for precise pose-based regularization.
To test the generalizability of the learned features, we also conducted transfer learning experiments on ImageNet (ILSVRC-2010). We fine-tuned the iLab-pretrained models on small subsets of ImageNet to determine if pose-aware features translate to natural, uncontrolled environments.
Internal Representation and Feature Decoupling
A key insight of this work is that 2W-CNN neurons become better at "specializing" compared to standard architectures. In a standard network, neurons often exhibit high uncertainty, attempting to represent both identity and pose simultaneously in a tangled fashion.
As shown in the correlation analysis, 2W-CNN neurons tend to decouple these signals, becoming either "identity-defining" or "pose-defining." This leads to a cleaner internal representation. Furthermore, qualitative visualization suggests that 2W-CNN learns more distinctive structural filters (edges and shapes) and fewer "dead" filters than AlexNet, which often relies more heavily on color blobs.
Key Results
- Categorization Performance: On the iLab-20M dataset, the 2W-CNN-MI achieved a 6% absolute improvement in category prediction over the AlexNet baseline.
- Transfer Learning: When fine-tuning on ImageNet with extremely limited data (5 images per class), 2W-CNN-MI features outperformed AlexNet by a staggering margin (9.27% vs 1.47% accuracy). This advantage persisted even as more training data was added.
- Optimization Advantage: We provide mathematical evidence that the 2W-CNN architecture creates a loss landscape that allows the network to escape mediocre local minima or saddle points that would typically trap a standard linear-chain network.
Conclusion
By treating pose as a temporary "where" signal during training, the 2W-CNN learns significantly more robust features for object categorization. While the current method requires explicit pose labels—which are more accessible in lab settings than in-the-wild datasets—the resulting performance gains in both controlled and natural settings demonstrate the power of pose-regularized learning. Future work will explore incorporating other factors, such as lighting source and background context, into this multi-layer injection framework.
Summary updated