Introduction
Multi-view action clustering (MVC) is a critical task in computer vision, aimed at grouping human actions—such as walking, jumping, or waving—by leveraging data from multiple camera perspectives. Traditional MVC algorithms operate under the assumption that all camera views are available simultaneously from the outset. However, in practical applications like evolving security networks or multi-sensor robotics, new cameras are often integrated sequentially over time.
This "incremental view" scenario presents significant challenges for existing methods:
- Data Divergence: Significant perspective shifts between old and new views make alignment difficult.
- Unpredictability: The characteristics of future views are unknown, preventing effective pre-training.
- Storage Constraints: Storing all historical raw video data for re-clustering is computationally prohibitive and often impractical.
To address these issues, researchers have proposed Continual Action Clustering (CAC), a framework designed to learn and retain action categories from historical views without the need for raw data storage.
The Continual Action Clustering (CAC) Framework
The CAC method treats the arrival of each new camera view as a distinct task within a continual learning pipeline. Unlike static methods, CAC focuses on maintaining a persistent knowledge base that evolves as new data arrives.
Core Components
- Category Memory Library ($B$): Instead of retaining raw data, CAC stores latent cluster centroids—the "essence" of an action—learned from historical views. This library serves as a repository of "never-ending knowledge," allowing the system to recall previous categories when processing new views.
- Consensus Partition Matrix ($H^*$): The system maintains a global "map" of data point assignments. When a new view arrives, the system updates this matrix by aligning the new view's local clusters with the existing global consensus.
- Three-Step Alternate Optimization: Efficiency is maintained through an optimization strategy that iteratively updates category assignments ($E$), the memory library ($B$), and alignment (permutation) matrices ($W_t$). This ensures the model remains accurate as it "lives and learns" from new perspectives.
Experimental Evaluation
The CAC model was rigorously evaluated on six realistic multi-view action datasets: IXMAS, MMI, MSR, WVU, UCLA, and MV-TJU. These datasets encompass a wide range of actions, lighting conditions, and camera configurations. For instance, the WVU dataset includes 11 actions captured by 8 cameras; to increase difficulty, non-adjacent cameras were selected for testing.
Key Results
The CAC method was compared against 15 state-of-the-art baselines, including traditional, deep, and incremental MVC approaches.
- Superior Accuracy: CAC consistently outperformed competitors in Accuracy (ACC), Normalized Mutual Information (NMI), and Purity. Notably, on the IXMAS dataset, it improved NMI by 7.41% to 16.48% compared to deep learning baselines like GCFAgg and AONGR.
- Robustness to View Order: Performance remained stable regardless of the sequence in which camera views arrived, demonstrating the framework's reliability in unpredictable environments.
- Scalability and Retention: The time complexity is linear ($O(n)$) relative to the number of samples. Furthermore, CAC exhibited minimal "forgetting" of historical information compared to other incremental methods like SCGL or CMVC, thanks to its dedicated memory library.
Discussion and Future Work
By decoupling cluster assignments from centroids and utilizing permutation matrices for alignment, CAC effectively "translates" new perspectives into an existing knowledge base. Its late-fusion approach based on kernel learning provides a lightweight alternative to resource-heavy deep learning models.
Despite its strengths, several avenues for future research remain:
- Dynamic Category Discovery: Currently, the model assumes a fixed number of action categories ($k$). Future iterations should aim to discover new action types autonomously.
- Incomplete Views: Addressing scenarios where certain actions are only captured by a subset of cameras (sample misalignment) is a priority.
- Incremental Samples: Combining incremental views with incremental data samples would move the framework closer to a truly "live" autonomous system.
Summary updated