Latent Space Motion Analysis for Collaborative Intelligence

Mateen Ulhaq, Ivan V. Bajic

TL;DR

This paper investigates how motion in video inputs is represented in the "latent space" (intermediate feature tensors) of deep neural networks. The authors demonstrate that motion is largely preserved, albeit scaled, through common DNN operations, suggesting that traditional video processing techniques can be adapted for this latent space. This has implications for efficient collaborative intelligence systems that process video at the edge.

What Problem Does This Paper Address?

Collaborative Intelligence (CI) systems split deep neural networks (DNNs) between edge devices and the cloud. The edge device processes the initial part of the network, sending intermediate "feature tensors" to the cloud for further analysis. When the input is a video, these feature tensors form a sequence. A key challenge is efficiently compressing and processing these feature tensors. Since video inherently involves motion, understanding how this motion is encoded in the feature tensors is crucial for developing effective compression and processing methods. This paper aims to answer: "What is the relationship between the motion in the input video and the motion represented in the intermediate feature tensors of a DNN?"

Methods

The authors analyze the effect of common DNN operations on optical flow, which is a standard way to represent motion in videos. They theoretically examine how convolutions, pointwise nonlinearities (like ReLU or sigmoid), and pooling operations affect optical flow. Their analysis, using a simplified 1D motion model, shows that these operations tend to preserve the input motion, approximately, with a scaling factor. Specifically, they derive that pooling operations (which downsample the feature maps) lead to a reduction in motion magnitude by a factor related to the number of pooling layers. They then propose a model (equation 12) that predicts latent-space motion based on input-space motion and the network's architecture, particularly its pooling layers.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they use synthetic input-space motion transformations (translation, rotation, scaling, shearing) applied to images of a dog. They also analyze layers from pre-trained ResNet-34 and DenseNet-121 models.

Key Insights and Ideas

  • Motion Preservation in Latent Space: The core finding is that motion from input video frames is not lost but rather approximately preserved in the feature tensors of intermediate DNN layers.
  • Predictable Scaling: The degree of motion preservation is predictable and primarily influenced by downsampling operations (pooling). Each pooling operation scales down the motion magnitude.
  • Bridging the Gap: This work suggests that established techniques from conventional video processing (motion estimation, compensation) can serve as a foundation for developing similar methods in the latent space of DNNs.
  • Foundation for Latent-Space Video Processing: The proposed model provides a starting point for developing latent-space motion compensation and compression techniques, which are vital for efficient CI systems.

Key Results

The paper presents both theoretical analysis and experimental validation.

  • Theoretical: They derive that convolutions and pointwise nonlinearities preserve motion, while pooling operations scale it down.
  • Experimental:
    • Figure 2 visually demonstrates the similarity between input-space motion and motion in feature tensor channels.
  • Figure 4 shows that their proposed model (equation 12) can reasonably predict transformed feature tensors based on the original tensor and known motion transformations.
  • Figure 5 and 6 quantify the accuracy of their model using Normalized Root Mean Square Error (NRMSE). NRMSE values are shown to be comparable to those found in conventional motion-compensated frame prediction in video coding (around 0.009–0.044). For instance, NRMSE is generally below 0.04 for reasonable ranges of transformation parameters.

Limitations and Future Work

  • Approximation: The analysis relies on approximations, particularly for the local maximum operation in pooling.
  • Simplified Model: The theoretical analysis uses a simplified "1D+t" model for optical flow, although they state it captures the main ideas.
  • Future Work: The authors suggest that their findings provide a good starting point for developing latent-space motion compensation and compression techniques for applications like tracking and action recognition within CI systems. They also imply that further research could refine these latent-space motion processing methods.

Summary updated

Abstract

When the input to a deep neural network (DNN) is a video signal, a sequence of feature tensors is produced at the intermediate layers of the model. If neighboring frames of the input video are related through motion, a natural question is, "what is the relationship between the corresponding feature tensors?" By analyzing the effect of common DNN operations on optical flow, we show that the motion present in each channel of a feature tensor is approximately equal to the scaled version of the input motion. The analysis is validated through experiments utilizing common motion models.
BibTeX
@inproceedings{ulhaq2021latent,
  author = {Mateen Ulhaq and Ivan V. Bajic},
  title = {{Latent Space Motion Analysis for Collaborative Intelligence}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=lcDSQq6QNV},
}

Loading related content...

Figure