TL;DR
This paper introduces FENet, a novel dual-branch network for image tampering localization that effectively combines semantic and noise-based features. By employing adaptive feature fusion and an edge-aware attention mechanism, FENet achieves state-of-the-art performance in accurately identifying tampered regions, even under challenging real-world conditions like compression and subtle manipulations.
What Problem Does This Paper Address?
The proliferation of sophisticated image editing tools has made it increasingly difficult to detect and precisely locate tampered regions in digital images. Traditional methods often struggle with robustness against diverse tampering types and post-processing artifacts, while existing deep learning approaches can over-rely on RGB semantic features, leading to poor performance in visually consistent forgeries or when boundary details are obscured. This paper addresses the challenge of achieving both high accuracy and robustness in image tampering localization, particularly focusing on accurately delineating tampered boundaries, which often contain subtle but crucial artifacts.
Methods
FENet employs a Symmetric Dual-Branch Architecture to capture complementary information.
- RGB Branch: Utilizes a pre-trained SegFormer-B2 backbone to extract high-level semantic features, adept at identifying global content inconsistencies.
- SRM Noise Branch: Leverages a fixed-weight SRM high-pass filter to extract noise residual features, sensitive to local statistical anomalies introduced by tampering.
Key novel components include:
- Self-Calibrating Fusion (SCF) Module: This module dynamically adjusts the importance of features from the two branches using adaptive weights. It first unifies feature dimensions, then performs an initial fusion via element-wise addition, followed by a calibration layer that learns weights to dynamically fuse features based on their content, improving efficiency and reducing redundancy.
- Edge-Aware Coordinate Attention (ECAM) Module: This module enhances boundary localization by integrating coordinate attention with edge detection. It explicitly models spatial position and uses Sobel operators to extract edge information, guiding the network to focus on potential tampering boundaries and improving feature representation along edges.
Datasets
The paper uses a combination of datasets for training and evaluation:
- Training: A total of 60,971 synthetic falsified images derived from public datasets and MS-COCO.
- Testing: Five widely adopted datasets were used to evaluate generalization performance:
- CASIA-v1: Copy-move and splicing.
- Columbia: Spliced images.
- NIST16: Simulates real-world, professional-grade tampering scenarios with complex manipulations.
- IMD: Splicing, copy-move, and inpainting.
- DSO: Focuses on small, blurry, and low-contrast tampered regions.
Additionally, experiments were conducted on images uploaded to Facebook, Weibo, and WeChat to simulate real-world social media compression and re-encoding.
Key Insights and Ideas
- Complementary Features are Crucial: The paper strongly advocates for combining semantic (RGB) and statistical (SRM noise) features. This dual-branch approach is key to handling diverse tampering types, especially those that are visually subtle but leave statistical footprints.
- Adaptive Fusion Matters: Simply concatenating or adding features is insufficient. The SCF module's ability to dynamically weight features based on content significantly improves performance, suggesting that the "importance" of different feature types varies with the input image.
- Edges Hold Vital Clues: Tampering often creates anomalies at region boundaries. The ECAM module's explicit focus on edge structures, guided by Sobel filters and attention, proves highly effective in precisely localizing these boundaries, a common weakness in other methods.
- Robustness to Compression: The SRM branch and the adaptive fusion strategy are particularly effective in maintaining accuracy even under severe JPEG compression, a critical factor for real-world social media applications.
Key Results
FENet achieves state-of-the-art performance across multiple benchmark datasets.
- Overall Performance: FENet achieved the best average F1 score, outperforming the second-best method (ConvNeXt) by 3.2%. It consistently ranked among the top two methods in both F1 and IoU metrics.
- DSO Dataset: FENet notably achieved the highest scores on the DSO dataset, which is challenging due to small, low-contrast tampered regions, highlighting its robustness in detecting subtle manipulations.
- Social Media Scenarios: In experiments simulating Facebook, Weibo, and WeChat environments, FENet achieved average F1 scores that were 2.8%, 3%, and 5.6% higher than existing state-of-the-art methods, respectively, demonstrating strong robustness against compression and quality loss.
- Ablation Studies: The SCF module alone improved F1 by 4.2% and IoU by 3.3% over a baseline. The ECAM module further boosted average F1 to 55.3% and IoU to 47.9%, confirming the significant contribution of each component.
Limitations and Future Work
- Columbia Dataset: A slight decline in F1 score was observed on the Columbia dataset due to smooth transitions and low texture contrast, which reduce feature discriminability. However, IoU remained high, indicating accurate boundary localization.
- Future Work: The authors plan to explore deeper correlations between image content and tampering traces, investigate more efficient multi-scale fusion strategies, and develop lightweight network designs for more accurate, efficient, and adaptable image tampering localization models.
Summary updated