TL;DR
This paper introduces TokenMotion (TMNet), a novel transformer-based approach for video camouflaged object detection (VCOD). TMNet effectively leverages motion cues from adjacent frames by employing a learnable token selection mechanism within its encoder, leading to state-of-the-art performance on the challenging MoCA-Mask dataset.
What Problem Does This Paper Address?
Detecting camouflaged objects in videos (VCOD) is a particularly difficult computer vision task. The core challenge lies in the fact that target objects often share similar textures and appearances with their backgrounds, making them hard to distinguish. This problem is further exacerbated in videos by the complex and often irregular motion patterns of both the objects themselves and the camera. Existing methods struggle with this, either by relying on potentially inaccurate optical flow estimations or by using simple feature correlations that can lead to blurred boundaries and noise. This paper aims to address these limitations by proposing a more robust and effective method for extracting motion-guided features to improve VCOD accuracy.
Methods
The proposed TokenMotion (TMNet) is a transformer-based model designed for VCOD. It utilizes a shared-weight encoder and decoder that process consecutive frames. The key innovations are:
- Motion-Guided Token Extraction Module (MG-TEM): This module operates within each stage of the encoder. It fuses texture features from the current frame with "masked tokens" from the next frame. These masked tokens are hypothesized to carry motion information. The fusion is done via element-wise addition, creating a "motion feature" that guides subsequent feature extraction.
- Token Selection Module (TSM): This module is crucial for identifying and selecting the most relevant tokens from the input feature maps. It uses a Gumbel Softmax trick to achieve a fully differentiable process, allowing it to be jointly trained with the segmentation loss. This enables the model to learn which tokens are most informative for the task, effectively muting or ignoring less relevant ones. The TSM outputs a binary mask indicating selected tokens.
- Neighbor Connection Decoder (NCD): The decoder utilizes the motion-guided features from the encoder. It incorporates a Texture Enhanced Module (TEM) and a Neighbor Connection Decoder (NCD) with group-reversal attention (GRA) to generate fine-grained object edges.
- Pixel Location Aware Loss: The model is trained using a weighted cross-entropy and intersection-over-union loss that specifically increases the weight for hard-to-detect pixels, improving boundary accuracy.
Datasets
The primary dataset used for training and evaluation is the MoCA-Mask dataset. This dataset comprises 87 video sequences and 22,939 frames. For testing, 16 sequences and 3,626 frames were used. The authors note that the MoCA-Mask dataset has a characteristic where many camouflaged objects in the training set are larger than those in the testing set, and some objects might only appear in one set. They also performed an ablation study using a subsampled training set to mitigate potential overfitting issues related to object size.
Key Insights and Ideas
- Motion as Implicit Token Selection: The core idea is that motion information is implicitly encoded within the features of adjacent frames. By using a learnable token selection mechanism, TMNet can effectively "pick out" these motion-guided tokens without explicit optical flow computation.
- Differentiable Token Selection is Key: The use of Gumbel Softmax for token selection is critical. It allows the model to learn which tokens are important in an end-to-end fashion, unlike previous threshold-based methods that were not fully differentiable.
- Neighbor Frame Fusion for Motion: Directly fusing features from adjacent frames within the encoder, rather than relying on separate correlation modules, simplifies the architecture and improves motion feature extraction.
- Addressing Edge Cases: The qualitative results highlight TMNet's ability to handle challenging scenarios like poor lighting, small objects, and similar irrelevant background objects, which are common pitfalls for VCOD methods.
Key Results
TMNet achieves state-of-the-art performance on the MoCA-Mask dataset, significantly outperforming the previous best method (SLT-Net [6]) across several metrics:
- Weighted F-measure: 12.8% improvement.
- S-measure: 8.4% enhancement.
- mean IoU: 10.7% boost.
- MAE: 61.9% improvement.
- mean Dice: 26.7% improvement.
Qualitatively, TMNet produces clearer object boundaries and less noise compared to SLT-Net, particularly in challenging sequences. Ablation studies confirm the effectiveness of both the Motion-Guided Token Extraction Module (MG-TEM) and the Token Selection Module (TSM).
Limitations and Future Work
- Enhanced-alignment Measure (E&): TMNet shows a slight decrease (6.8%) in the Enhanced-alignment measure compared to SLT-Net. The authors attribute this to occasional missed target objects in specific sequences (e.g., copperhead_snake, moth), leading to predictions with all background or foreground pixels at the image level.
- Dataset Imbalance: The authors acknowledge the dataset imbalance in MoCA-Mask regarding object sizes between training and testing sets. While they address this with a subsampling strategy for ablation studies, it suggests a potential area for further improvement in model robustness.
- Future Work: The paper implies that the learned token selection and motion-guided features are highly effective. Future work could explore applying this framework to other video understanding tasks or investigating more sophisticated ways to fuse inter-frame information. The authors also mention that their code will be made available upon paper acceptance.
Summary updated