WhaleLM: Finding Structure and Information in Sperm Whale Vocalizations and Behavior with Machine Learning

Pratyusha Sharma, Shane Gero, Daniela Rus, Antonio Torralba, Jacob Andreas

TL;DR

This paper introduces "WhaleLM," a novel application of neural sequence models to decipher the complex communication system of sperm whales. By treating whale vocalizations (codas) as a language, they demonstrate that these sequences exhibit long-range dependencies and encode information about the whales' behavior, paving the way for a new era of understanding animal communication.

What Problem Does This Paper Address?

Animal communication systems are notoriously difficult to decipher. While we know many species use vocalizations to coordinate behavior, the underlying structure and information content of these signals remain largely unknown. Sperm whales, in particular, use intricate sequences of clicks called "codas" for communication, but the patterns governing their use and how these patterns relate to group behavior have been a long-standing mystery. This paper aims to bridge this gap by applying advanced machine learning techniques, specifically neural sequence models, to analyze sperm whale vocalizations and their connection to behavior.

Methods

The core of this research lies in training "sperm whale language models" – neural sequence models – on recorded sperm whale vocalizations and associated behavioral data. The researchers employed an encoder-decoder LSTM architecture, a standard for sequence modeling, to predict future codas and whale behaviors.

A key methodological innovation is their approach to understanding what information is encoded. They systematically "ablate" (remove) specific features of the codas (like rhythm or tempo) from the model's input and observe how this impacts its predictive power. This allows them to identify which acoustic features are crucial for predicting future vocalizations and behaviors. They also compared different model complexities, from simple linear models to more sophisticated LSTMs with attention, to understand the necessary expressiveness for capturing the communication structure.

Datasets

The study utilizes the Dominica Sperm Whale Project (DSWP) dataset. This dataset comprises manually annotated coda clicks and associated accelerometer, gyroscope, and magnetometer data from bio-logging tags deployed on sperm whales in the Eastern Caribbean (EC1 clan) between 2014 and 2018. This rich dataset allows for the computation of whale position over time and the correlation of vocalizations with behavioral states.

Key Insights and Ideas

  • Whale "language" has grammar: Sperm whale vocalizations aren't just random sounds; they exhibit a complex structure with order dependence and long-range dependencies, meaning the meaning or prediction of a coda depends on a sequence of up to eight previous codas (spanning about 30 seconds).
  • Codas encode behavioral context: The sequence of codas is not just about communication between whales; it also contains significant information about the whales' current and future actions, such as their diving behavior.
  • Feature importance for communication: By systematically removing acoustic features (like rhythm and tempo), the researchers identified which elements are most critical for predicting future vocalizations and behaviors, offering insights into the "phonetics" of whale communication.
  • Expressive models are key: More complex neural sequence models, particularly recurrent ones like LSTMs, are significantly better at capturing the nuances of sperm whale communication compared to simpler models like n-grams or linear models.

Key Results

  • Predicting future codas: Models could predict future codas with high accuracy, showing that the sequence of up to eight previous codas is informative. Removing order information from the input increased perplexity (decreased predictability) by up to 22.7%.
  • Predicting behavior:
    • Current behavioral state (descent, ascent, shallow dive) was predicted with 72.8% accuracy.
  • Future behavioral state (e.g., foraging dive vs. other) was predicted with 86.4% accuracy.
  • These accuracies are significantly better than chance baselines (33.3% and 50%, respectively).
  • Feature impact: Tempo features alone were found to be sufficient to match the accuracy of the full model for behavior prediction, while rhythm features provided a reduced but still non-trivial predictive accuracy. This suggests a specific combination of rhythm and tempo is linked to foraging dives.
  • Model comparison: Expressive models with sequential structure (LSTM, LSTM+Attention) outperformed linear models and n-gram models in predicting the next call, though n-gram models performed surprisingly well.

Limitations and Future Work

  • Dataset size and scope: While the DSWP dataset is substantial, the authors note that the "ornaments" feature is rare (4% of the dataset), and the dataset might be too small to fully capture the dynamics of changing rubato. Future work could involve larger datasets or more targeted data collection.
  • Generalizability: The current study focuses on a specific clan (EC1) in the Eastern Caribbean. Future research could explore whether these findings generalize to other sperm whale clans or populations.
  • Interactive playback experiments: The authors suggest that their approach, which identifies informative features offline, can serve as a precursor to interactive playback experiments. This would allow for direct testing of the communicative function of identified vocalization patterns.
  • Broader application: The methods developed are presented as a general framework for using language models to investigate unknown communication systems, suggesting potential applications to other animal species.

Summary updated

Abstract

Sperm whales (Physeter macrocephalus) communicate using patterned click sequences called codas. Whether there are any systematic patterns governing the structure of coda sequences, or how coda production influences group behavior, remain open questions. To answer these questions, we train neural sequence models (“sperm whale language models”) on vocalization and behavior data from a population of sperm whales in the eastern Caribbean. By systematically manipulating models' training data and measuring changes in predictive power, we find that vocalizations exhibit order dependence and long-range dependencies on up to eight previous codas in an exchange. We additionally find that this structure encodes information about behavior: whales' current behavioral context and future actions are predictable with high accuracy from coda sequences. The methods developed for relating vocalization to behavior are general, and offer a flexible framework for using language models to investigate the structure and information content of unknown communication systems.
BibTeX
@inproceedings{sharma2025whalelm,
  author = {Pratyusha Sharma and Shane Gero and Daniela Rus and Antonio Torralba and Jacob Andreas},
  title = {{WhaleLM: Finding Structure and Information in Sperm Whale Vocalizations and Behavior with Machine Learning}},
  booktitle = {AiForAnimalComms @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=ppCEqgYXHd},
}
Figure