Abstract
Fine-grained action segmentation in long untrimmed videos is an important task for many applications such as surveillance, robotics, and human-computer interaction. To understand subtle and precise actions within a long time period, second-order information (e.g. feature covariance) or higher is reported to be effective in the literature. However, extracting such high-order information is considerably non-trivial. In particular, the dimensionality increases exponentially with the information order, and hence gaining more representation power also increases the computational cost and the risk of overfitting. In this paper, we propose an approach to representing high-order information for temporal action segmentation via a simple yet effective bilinear form. Specifically, our contributions are: (1) From the multilinear perspective, we derive a bilinear form of low complexity, assuming that the three-way tensor has low-rank frontal slices. (2) Rather than learning the tensor entries from data, we sample the entries from different underlying distributions, and prove that the underlying distribution influences the information order. (3) We employed our bilinear form as an intermediate layer in state-of-the-art deep neural networks, enabling to represent high-order information in complex deep models effectively and efficiently. Our experimental results demonstrate that the proposed bilinear form outperforms the previous state-of-the-art methods on the challenging temporal action segmentation task.
Introduction
Fine-grained action segmentation involves identifying the precise start and end times of specific actions within long, untrimmed video sequences. This task is critical for applications ranging from automated surveillance to assistive robotics. While modern deep learning architectures, particularly Temporal Convolutional Networks (TCNs), have significantly advanced the field, they often struggle to capture the subtle nuances required to distinguish between highly similar actions.
To address this, researchers have looked toward high-order feature interactions. While "first-order" features provide a basic representation, "high-order" information—such as feature covariance—captures how different features co-occur and interact over time. This added expressiveness is key to understanding complex human activities.
The Challenge of High-Order Information
The primary barrier to utilizing high-order information is the "curse of dimensionality." Standard bilinear pooling methods typically involve a three-way tensor. As the number of features increases, the size of this tensor grows cubically. This leads to several significant issues:
- Computational Cost: The memory and processing power required for dense 3D tensors are prohibitive for deep networks.
- Overfitting: The massive number of parameters in a full bilinear layer makes the model highly susceptible to overfitting, especially on smaller datasets.
- Integration Difficulty: Due to these constraints, bilinear layers are often relegated to the final stages of a network, limiting their ability to influence intermediate feature representations.
Proposed Method: Frontal Low-rank Random Tensors
The authors introduce a novel bilinear form designed to capture high-order interactions without the associated computational burden. Their approach rests on two key innovations:
1. Frontal Low-rank Approximation
Instead of a dense 3D tensor, the authors assume that each "frontal slice" of the tensor is a low-rank matrix. This mathematical shortcut allows for the decomposition of a large operation into a series of smaller matrix multiplications followed by a vector outer product. This reduction is significant: the runtime complexity drops from $O(Dd + d)$ to $O(D\sqrt{d} + d)$, where $D$ and $d$ represent input and output dimensions, respectively.
2. Random Projections as Kernels
A major departure from traditional methods is the use of random tensors rather than learned ones. Instead of training millions of parameters via backpropagation, the entries are sampled from specific distributions:
- RPBinary: Uses the Rademacher distribution (random values of -1 or 1).
- RPGaussian: Uses Gaussian distributions.
The authors provide a theoretical proof that these random projections approximate specific kernels within a Reproducing Kernel Hilbert Space (RKHS). Essentially, choosing a distribution is equivalent to choosing the "order" of information the model captures. For example, Gaussian random projections allow the model to approximate an "infinite-order" Gaussian kernel. This use of randomness also acts as a powerful regularizer, mitigating the risk of overfitting.
Architecture Integration
The efficiency of this bilinear module allows it to be used as an intermediate layer within complex architectures. The authors demonstrate this by integrating it into the Multi-Stage Temporal Convolutional Network (MS-TCN).
As shown in the figure above, the bilinear pooling layer is inserted after the standard convolutional blocks. The authors found that the performance is sensitive to the temporal window of high-order interaction, identifying a convolution kernel size of 25 as optimal for capturing relevant temporal dynamics.
Experiments and Results
The proposed method was rigorously evaluated on three benchmark datasets: 50Salads, GTEA, and Breakfast. These datasets cover a wide range of activities, from salad preparation to complex breakfast cooking tasks.
Performance and Efficiency
The RPGaussian variant consistently outperformed existing state-of-the-art models across various metrics, including Frame-wise Accuracy, Edit Score, and F1 scores.
- Accuracy: On the GTEA dataset, the model achieved an [email protected] score of 74.6, surpassing the baseline MS-TCN (70.4) and other bilinear methods like Compact Bilinear Pooling (70.2).
- Speed: The method is remarkably efficient. On the 50Salads dataset, the module recorded a runtime of approximately 69ms per batch, a nearly 8x improvement over Temporal Bilinear Pooling (TBP), which clocked in at 553ms.
Conclusion
By leveraging frontal low-rank approximations and random projections, this work provides a scalable and effective way to incorporate high-order feature interactions into deep temporal models. The approach not only achieves state-of-the-art results in action segmentation but does so with a fraction of the computational cost and improved robustness against overfitting. Future work may explore learning the optimal distributions for sampling via deep generative models, further refining this "random" yet principled approach to feature representation.
Summary updated