Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features

Haixu Liu, Penghao Jiang, Zerui Tao, Muyan Wan, Qiuzhuang Sun

Abstract

Accurate prediction of plant species distribution is a critical component of biodiversity conservation, yet it presents a significant multi-modal challenge for machine learning. We present Tighnari, a framework that secured 3rd place in the GeoLifeCLEF 2024 competition. Tighnari introduces a novel Hierarchical Cross-Attention Mechanism (HCAM) to fuse high-resolution satellite imagery, multi-year climate time series, and sparse environmental metadata. By transforming 1D climate sequences into 2D "temporal cubes" for vision backbones and leveraging a custom graph structure to aggregate neighbor-based ecological data, Tighnari achieves a balance between information-dense visual features and sparse spatial context.

Introduction

Predicting the presence of plant species at specific locations requires synthesizing diverse data types: dense satellite imagery, sequential climatic trends, and tabular environmental indices. Traditional approaches often struggle to fuse these modalities effectively, either overfitting to dense data or ignoring the subtle signals in sparser metadata. Tighnari addresses this by treating the prediction task as a multi-modal fusion problem, prioritizing the hierarchical integration of features based on their information density.

Methodology

The Tighnari architecture is built upon three core innovations designed to handle the heterogeneity of ecological data.

Temporal Cubes and Vision Backbones

Rather than employing standard 1D sequence models like LSTMs for climatic data, Tighnari "folds" 1D time series into 2D matrices. This transformation allows the model to utilize a Swin-Transformer vision backbone. This approach captures the multi-scale spatio-temporal relationships inherent in seasonal shifts more effectively than traditional sequential models.

Climatic feature trends over a 19-year period (precipitation and temperature) used for temporal cube construction.

Graph-Based Feature Construction

To incorporate spatial context, the authors constructed a graph where each survey location serves as a node. Edges connect nodes within a 10km radius that share the same year and region, defining a shared ecological niche. A Graph Feature Vector (GFV) is then generated by aggregating species labels from neighboring nodes using a square-root normalization technique. This allows the model to leverage local biodiversity patterns to inform its predictions.

Spectral band time series for a single node, illustrating the sequential data integrated into the graph features.

Hierarchical Cross-Attention Mechanism (HCAM)

The HCAM module serves as the central fusion engine. Recognizing that simple concatenation often leads to instability when mixing dense and sparse data, HCAM employs a two-stage process:

  1. Stage 1: Satellite image features act as "Keys" and "Values" to be queried by the sparser time-series and tabular data.
  2. Stage 2: The output of the first stage is fused with the GFV using Multi-Head Cross-Attention. This hierarchy ensures that the high-density visual information provides a grounded context for interpreting sparser environmental signals.

Datasets and Experimental Setup

The model was evaluated using the GeoLifeCLEF 2024 dataset, comprising 88,987 training and 4,716 testing plant survey records across Europe. The dataset includes RGB and Near-Infrared satellite imagery, bioclimatic rasters, soil variables, and human footprint data covering 11,255 distinct species.

To optimize performance, the team implemented a Threshold Top-K post-processing method. Instead of a fixed Top-K, a species is only predicted if it ranks within the top 44 candidates and exceeds a probability threshold of 0.23, optimizing the balance between precision and recall.

Results

Tighnari achieved 3rd place in the GeoLifeCLEF 2024 & CVPR-FGVC Competition with a private leaderboard score of 0.36242. Beyond accuracy, the model demonstrated high efficiency; by utilizing optimized backbones like EfficientNet-B0 for imagery and the custom Swin-T for time series, the team reduced training time per epoch by 75% compared to heavier architectures like ResNet18, with negligible impact on performance.

Conclusion and Future Work

Tighnari demonstrates that hierarchical attention and vision-based processing of temporal data are powerful tools for ecological modeling. Future iterations of the model may replace manual graph weighting with Graph Neural Networks (GNNs) to learn spatial relationships dynamically. Additionally, the authors aim to incorporate weakly supervised learning to better utilize abundant but noisy "Presence-Only" data, further enhancing the model's robustness in diverse ecological scenarios.

Summary updated

BibTeX
@inproceedings{liu2024tighnari,
  author = {Haixu Liu and Penghao Jiang and Zerui Tao and Muyan Wan and Qiuzhuang Sun},
  title = {{Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features}},
  booktitle = {CLEF (Working Notes)},
  year = {2024},
}
Figure