Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features

Haixu Liu, Penghao Jiang, Zerui Tao, Muyan Wan, Qiuzhuang Sun

Introduction

Predicting plant species distribution is a critical task for biodiversity conservation, yet it remains a challenging multi-label classification problem. A single location may host dozens of species, influenced by a complex interplay of local climate cycles, soil quality, human activity, and the proximity of existing populations. The "Tighnari" model, which secured 3rd place in the GeoLifeCLEF 2024 competition, addresses these challenges through a multi-modal fusion approach combining satellite imagery, environmental tabular data, and novel graph-based features.

Methodology

The Tighnari architecture introduces several innovations to the standard species distribution modeling pipeline, centered around the Hierarchical Cross-Attention Fusion Mechanism (HCAM).

Time Series as "Cubes"

Traditional 1D sequence models often struggle with long-term climate data. Tighnari "folds" multi-year time series into 2D matrices (cubes), allowing the use of a modified Swin-Transformer to extract temporal features. This approach leverages the vision transformer's ability to capture multi-scale relationships, treating temporal patterns as spatial structures.

Multi-band time series for different survey nodes across 19 years, illustrating the complex temporal data processed by the Swin-Transformer backbone.

Graph-Based Feature Construction

To capture the "power of proximity," survey locations are treated as nodes in a graph. Edges connect surveys within 10km that share the same region and year. A Graph Feature Vector (GFV) is then constructed by aggregating labels from neighboring nodes using a square-root normalization trick, effectively capturing local species clusters.

Hierarchical Cross-Attention (HCAM)

HCAM serves as the integration layer for disparate data types. It uses a hierarchy where dense satellite image features act as "Queries" to extract relevant information from climate and tabular data. The resulting vector then interacts with sparse graph features, ensuring that high-density data does not overwhelm the predictive power of sparse survey records.

Datasets and Data Modalities

The model was evaluated on the GeoLifeCLEF 2024 dataset, comprising 88,987 training records across 11,255 species. Inputs include RGB and Near-Infrared satellite imagery, rasterized environmental variables (soil, land cover), and climate time series.

Seasonal trends in bioclimatic variables (Bio-tasmin) over a 19-year period, representing the environmental data modality.

Results and Discussion

Tighnari achieved a private leaderboard score of 0.36242, placing 3rd in the competition. Ablation studies confirmed that both HCAM and graph-based features are essential for performance. Furthermore, the team optimized training efficiency by using a 10-fold cross-fusion strategy where subsequent folds were fine-tuned from a base model, reducing total training time by 75%.

Conclusion

The success of Tighnari demonstrates that treating temporal data as visual structures and explicitly modeling spatial proximity through graphs can significantly improve species prediction. Future work includes replacing manual graph weights with learned weights via Graph Neural Networks (GNNs) and utilizing Neural Architecture Search (NAS) to further optimize the temporal feature extraction backbone.

Summary updated

BibTeX
@inproceedings{liu2024tighnari,
  author = {Haixu Liu and Penghao Jiang and Zerui Tao and Muyan Wan and Qiuzhuang Sun},
  title = {{Tighnari: Multi-modal Plant Species Prediction Based on Hierarchical Cross-Attention Using Graph-Based and Vision Backbone-Extracted Features}},
  booktitle = {CLEF (Working Notes)},
  year = {2024},
}
Figure