Cat Royale Comprehensive Dataset: A Dataset for Pose Detection, Segmentation, Animal Action Recognition and Scene Understanding

Zane K. J. Hartley, Joseph Ho, Alan Chamberlain, Michael P. Pound, Steve BenfordarXiv 2026 · arXiv

TL;DR

The Cat Royale Comprehensive Dataset (CRCD) is a massive, multi-modal resource designed to push the boundaries of feline behavior analysis. By linking 20,000 segmentation masks, 5,000 pose-annotated images, and over 800 expert-captioned video clips, this dataset provides the ground truth necessary to move from simple object detection to nuanced animal welfare monitoring.

What Problem Does This Paper Address?

While computer vision has made leaps in understanding human actions and environments, the animal kingdom remains a "frontier." Animals present unique challenges: they have highly deformable bodies, shifting coat patterns that blend into backgrounds, and complex social interactions that lead to frequent occlusions.

Existing animal datasets are often fragmented—focusing on just one task like pose or segmentation—and rarely provide the multi-modal links (e.g., connecting a specific skeletal pose to a behavioral label and a natural language description) needed for high-level scene understanding. The authors address this gap to enable automated, non-invasive 24/7 monitoring, which could revolutionize animal welfare in homes, shelters, and zoos.

Methods

The data was born from an unlikely source: an interactive artwork called Cat Royale, where three cats lived in a bespoke environment and played with a robot arm. Researchers captured over 500 hours of 1080p video from nine different angles.

To turn this mountain of footage into a dataset, the team employed several clever strategies:

  • Human-in-the-Loop Annotation: They used SOTA models like DeepLabCut (DLC) for pose and SAM2 for segmentation to generate "pseudo-labels," which were then meticulously corrected by human experts.
  • Prompt-Based Segmentation: To handle tricky occlusions between cats, they used DLC-detected keypoints as positive prompts for SAM2, while using keypoints from other cats as negative prompts to sharpen boundaries.
  • Expert Ethograms: Behavioral annotations weren't just "cat playing." A clinical animal behaviorist used a validated 41-action ethogram (The Feline Emotions Guide) to label clips, ensuring the data reflects actual emotional states like "Joy/Play" or "Interest."
The dataset covers four key tasks: (a) Segmentation, (b) Pose Estimation, (c) Action Recognition, and (d) Video Captioning.

Datasets

The CRCD is divided into four specialized subsets, summarized in the table below:

Overview and Composition of the CRCD.

For the pose estimation task, the authors defined a specific set of 18 keypoints tailored to feline physiology:

The 18 keypoints used for CRCD-pos.

Key Insights and Ideas

  • Segmentation is "Solved," Pose is Not: While models like YOLOv8-seg achieved over 85% AP, pose estimation mAP scores were significantly lower (around 53-64%). This highlights that while we can find a cat, precisely tracking its contorted limbs during grooming or play remains a major hurdle.
  • The "Subtle Cue" Challenge: Many animal behaviors rely on micro-movements—pupil dilation or slight tail twitches—that are difficult for current Temporal Action Localisation (TAL) models to catch.
  • Art as a Research Engine: The paper highlights how artistic projects can act as "unusual research methods," investing heavy labor into creating rich, documented environments that traditional labs might struggle to replicate.
  • Multi-modal Synergy: The authors hypothesize that the real power of CRCD lies in the intersection of labels. For example, using segmentation masks to provide better context for pose estimation in crowded scenes.

Key Results

The authors established baselines using several popular models:

  • Instance Segmentation: YOLOv8s-seg led the pack with an APmask of 0.890.
  • Pose Estimation: Keypoint R-CNN achieved a mAP of 0.64, outperforming YOLOv8-pose (0.53), though YOLO had better general localization (AP50).
  • Temporal Action Localisation (TAL): This proved to be the most difficult task. The best model, TriDet, only achieved a [email protected] of 17.23%, suggesting that current SOTA models for human action detection struggle to translate to the animal domain.
  • Video-Text Retrieval: FrozenInTime showed promise with a Rank@1 of 35.0%, significantly outperforming Clip4Clip.

Limitations and Future Work

The authors are transparent about the dataset's constraints:

  • Environmental Bias: Because the Cat Royale environment was designed to be "utopian," there are very few examples of negative emotions like fear or anger.
  • Technical Constraints: Even with 1080p video, subtle indicators like pupil dilation were often indiscernible, forcing annotators to rely on more overt cues.
  • Subjectivity: Despite using experts, labeling animal "emotion" involves a degree of inference that makes an "absolute ground truth" elusive.

Future work will likely focus on leveraging the multi-modal nature of the dataset—using the high-quality segmentation masks to improve the accuracy of pose and action recognition in complex, multi-subject scenes.

Summary updated

Abstract

Animal focused computer vision datasets are of a growing interest in the fields of both Computer Vision and Animal behaviour. The ability to make sense of complex scenes via a variety of Computer Vision tasks has substantial implications for behavioural understanding and welfare. The perfection of these tasks will help lead to better health and emotional outcomes for wild and domestic animals. In this work we present the Cat Royale Comprehensive Dataset (CRCD), a large, comprehensive, multi-modal dataset for feline understanding, based on a huge database of over 500 hours of video captured during the Cat Royale artwork and research project. Our dataset contains annotations for a range of computer vision problems including 20,000 images with instance segmentation masks, 5000 images with multi-subject pose coordinates, over 800 video clips with action classes, and over 800 clips with detailed text description. This is the first such dataset to link this wide range of annotation types together for multi-modal use. We release this entire dataset, and our custom annotation tools will also be made available, in order to allow for ongoing expansion. We also provide a series of benchmarks for the tasks of Instance Segmentation, Pose detection, Temporal Action Localisation, and finally video-text retrieval. The totality of this multi-modal dataset is the data needed to solve both these individual computer vision tasks as well as enabling more complex multi-modal tasks making use of a combination of annotations.
BibTeX
@misc{hartley2026cat,
  author = {Zane K. J. Hartley and Joseph Ho and Alan Chamberlain and Michael P. Pound and Steve Benford},
  title = {{Cat Royale Comprehensive Dataset: A Dataset for Pose Detection, Segmentation, Animal Action Recognition and Scene Understanding}},
  year = {2026},
  eprint = {2611.15601},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15601},
}
Figure