Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification

Bennet Breier, Arno OnkenICLR 2020

Introduction

In neuroscientific research, zebrafish are vital model organisms. Their translucent bodies and genetic similarity to humans allow researchers to link neural activity directly to behavior. A key challenge in this field is accurately classifying "swim bouts"—specifically, distinguishing between spontaneous swimming and the precise movements used to capture prey.

Previously, the state-of-the-art approach relied on Support Vector Machines (SVMs) using hand-picked features (like maximum tail curvature) defined by human experts. While effective, this "feature engineering" is labor-intensive and might miss subtle patterns. Convolutional Neural Networks (CNNs) offer a powerful alternative, but they are often treated as "black boxes." Without knowing why a model makes a decision, researchers risk relying on "Clever Hans" predictions—where a model achieves high accuracy by latching onto unintended spurious correlations (like a background artifact) rather than the actual biological behavior.

This paper applies a Two-Stream CNN to classify zebrafish swimming behaviors, outperforming traditional hand-engineered models. By using AI explainability techniques, the researchers discovered the model was initially "cheating" by looking at experimental artifacts. This discovery allowed them to refine the model and reveal that the network identifies prey-capture behavior by the unique steadiness of the fish's trunk.

Methods

The authors adapted a Two-Stream CNN architecture to process both the appearance and the movement of the zebrafish:

  1. Spatial Stream: Analyzes a single grayscale frame to capture the fish's appearance.
  2. Temporal Stream: Analyzes Optical Flow (the pattern of apparent motion between frames) across 170 channels to capture the dynamics of the swim.
Two-Stream CNN architecture showing the spatial and temporal streams.

To peek inside the black box, they used Deep Taylor Decomposition (DTD) via the iNNvestigate toolbox. DTD generates heatmaps that highlight which pixels in the video were most "relevant" to the model's classification. This allowed the researchers to compare the machine's logic with known biological kinematics.

Datasets and Preprocessing

The study utilized a dataset originally gathered by Semmelhack et al. (2014), consisting of high-speed video (300 fps) of larval zebrafish. The dataset included 1,214 video snippets labeled as either "spontaneous" (56.1%) or "prey" (43.9%) bouts. To prevent overfitting, the authors used subsampling, flipping, and cropping to expand the dataset to 174,816 augmented samples.

Crucially, during preprocessing, the fish's head was cropped out to ensure the model didn't "cheat" by looking at eye movements, focusing the analysis entirely on the tail and trunk.

Preprocessing steps showing detected contours, the cropped frame, and the resulting optical flow frame.

Key Insights and Results

  • Catching the "Clever Hans" Effect: Initial heatmaps showed the model was focusing on the top-left corner of the frames. It turned out the agarose (the gel used to hold the fish's head) moved slightly when experimenters tapped the petri dish to induce spontaneous bouts. The model learned to identify "spontaneous" bouts by the gel's vibration rather than the fish!
  • The "Steadiness" Feature: After whiting out the corners to remove the artifact, the model shifted its focus to the trunk (the base of the tail). While human experts traditionally look at the tail tip and its curvature, the CNN found that the steadiness of the trunk was a better predictor.
  • Biological Alignment: The CNN's focus on the trunk aligns with biological reality: during prey capture, zebrafish must stabilize their heads and trunks to maintain a steady line of sight on their target, whereas spontaneous bouts involve more "yaw" and oscillation.
  • Motion over Form: The temporal stream (motion) was significantly more accurate than the spatial stream (appearance), confirming that the way a fish moves is more diagnostic than what it looks like at any single moment.
  • Superior Accuracy: The final refined CNN achieved a classification accuracy of 96.32%, beating the previous state-of-the-art SVM (90.2%) by 6.12 percentage points.
  • Explainability as a Debugger: The study proves that explainability tools aren't just for "trust"—they are essential debugging tools that can identify flaws in experimental setups and lead to more robust scientific discoveries.

Limitations and Future Work

The authors note that calculating optical flow is computationally expensive and storage-heavy. Future work could explore whether the model can achieve similar results using raw video frames in a 3D-CNN, which might be more efficient.

Additionally, the authors suggest this model could be used for lesion studies. By training the model on healthy fish and then testing it on fish with neural damage, researchers could quantitatively measure how well a fish's behavior recovers over time, providing a new metric for studying brain regeneration.

Summary updated

BibTeX
@inproceedings{breier2020analysis,
  author = {Bennet Breier and Arno Onken},
  title = {{Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure