Abstract
This paper aims at recognizing partially observed human actions in videos. Action videos acquired in uncontrolled environments often contain corrupt frames, which make actions partially observed. Furthermore, these frames can last for arbitrary lengths of time and appear irregularly. They are inconsistent with training data and degrade the performance of pre-trained action recognition systems. We present an approach to address this issue. For each training and testing actions, we divide it into segments and explore the mutual dependency between temporal segments. This property states that the similarity of two actions at one segment often implies their similarity at another. We augment each segment with extra alternatives retrieved from training data. The augmentation algorithm is designed in a way where a few alternatives are good enough to replace the original segment where corrupt frames occur. Our approach is developed upon hidden conditional random fields and leverages the flexibility of hidden variables for uncertainty handling. It turns out that our approach integrates corrupt segment detection and alternative selection into the process of prediction, and can recognize partially observed actions more accurately. It is evaluated on both fully observed actions and partially observed ones with either synthetic or real corrupt frames. The experimental results manifest its general applicability and superior performance, especially when corrupt frames are present in the action videos.
Introduction
Action recognition is a cornerstone of computer vision, yet most state-of-the-art models struggle when faced with "broken" video data. In real-world scenarios—ranging from surveillance to home robotics—videos are rarely pristine. They are frequently plagued by sensor glitches, physical occlusions (e.g., a person walking behind furniture), or failures in skeleton-tracking algorithms. These issues result in Partially Observed Action Recognition (POAR), where the temporal flow of information is disrupted by irregular outliers.
The Challenge of Partially Observed Actions
Standard action recognition systems typically assume full observation. When outliers occur, they introduce two critical failures: information loss (missing key poses) and temporal discontinuity. Simply removing corrupted frames breaks the sequence, making it difficult for traditional models like Hidden Markov Models (HMMs) or standard Conditional Random Fields (CRFs) to maintain an accurate state of the action.
Methodology
Alternative Augmentation via Mutual Dependency
The core innovation of this research is the concept of "Alternative Augmentation." This method leverages the mutual dependency between temporal segments: if two actions appear similar at one point in time, they are likely to be similar at others. For every segment of a video, the system searches the training database for high-quality "alternatives."
If a segment in a query video is corrupted, the model identifies a non-corrupted segment in the same video, finds a training video with a matching segment, and "borrows" that training video's corresponding segment as a potential replacement. This allows the model to perform data imputation without needing to "hallucinate" frames from scratch.
Extended Hidden Conditional Random Fields (HCRFs)
The authors integrate this augmentation into an extended HCRF framework. Unlike standard HCRFs that use hidden variables solely for latent poses, this model utilizes composite hidden variables. These variables simultaneously decide which pose is being performed and which observation to trust—the original segment or one of the retrieved alternatives.
A built-in "bias" trick ($\epsilon$) ensures the model favors the original observation unless an alternative provides a significantly better fit for the predicted action. This effectively automates outlier detection and data cleaning within the inference process itself.
Experimental Results
The framework was evaluated on two primary datasets:
- CITI-DailyActivities3D: A dataset of 15 daily activities with natural sensor noise and self-occlusions. While standard HCRFs saw accuracy drop from 80.3% to 60.3% on noisy data, the proposed method maintained a robust 74.1%.
- UT-Interaction: A benchmark for human interaction. Under synthetic noise conditions with a 50% outlier ratio, the proposed method outperformed competitors like RNNs by approximately 30%.
Conclusion
By treating data cleaning and action recognition as a joint inference problem, this approach offers a robust solution for real-world video analysis. The system is not only accurate but also efficient, running at near-real-time speeds. Future work aims to refine this granularity from temporal segments to specific spatial regions or trajectories, allowing the model to "mix and match" specific body parts from different observations to handle complex occlusions.
Summary updated