TMFNet: Two-Stream Multi-Channels Fusion Networks for Color Image Operation Chain Detection

Yakun Niu, Lei Tan, Lei Zhang, Xianyu Zuo

TL;DR

TMFNet is a novel two-stream deep learning framework designed to uncover the "operation chain"—the sequence of edits like blurring or resizing—performed on a color image. By fusing global spatial artifacts with local noise residuals and specifically leveraging the often-ignored correlations between color channels, it achieves state-of-the-art accuracy and superior generalization across different datasets and resolutions.

What Problem Does This Paper Address?

In the world of multimedia forensics, detecting if an image has been manipulated is only half the battle. To truly verify authenticity, investigators need to reconstruct the operation chain: the specific sequence of processing steps (e.g., Gaussian Blur $\rightarrow$ Median Filtering $\rightarrow$ JPEG Compression) an image has undergone.

Existing methods face three major hurdles:

  1. Generalization: Models trained on one dataset often fail miserably when tested on images from different cameras or at different resolutions.
  2. Information Loss: Standard neural networks use "pooling" layers to reduce data size, but in forensics, pooling acts as a low-pass filter that erases the subtle, high-frequency traces left by editing tools.
  3. Ignoring Color: Most forensic tools treat images as grayscale or process color channels independently, ignoring the rich "inter-channel correlation"—how the Red, Green, and Blue channels relate to one another—which changes predictably after specific edits.

Methods

The authors propose TMFNet (Two-Stream Multi-Channels Fusion Network), which processes images through two specialized pipelines:

  • Spatial Artifact Stream: This stream extracts global features. To prevent the loss of forensic evidence, the authors designed a deep residual architecture that completely avoids pooling in its early stages. This allows the network to learn the global representation of multi-channel correlations without smoothing out the artifacts.
  • Noise Residual Stream: This stream focuses on local, low-level noise. The authors introduced novel, hand-designed RGB filters based on Gaussian distributions and pixel predictors. Unlike standard filters (like SRM) designed for grayscale, these filters are optimized to suppress image content (texture) while highlighting the noise patterns across all three color channels.
  • Feature Fusion: The high-level features from both streams are concatenated and fed into a fusion module. This "best of both worlds" approach allows the model to identify operation traces that might have been weakened or "overwritten" by subsequent steps in the chain.

Datasets

The researchers evaluated TMFNet using several standard forensic benchmarks:

  • RAISE: A dataset of 10,000 high-resolution raw images, used for primary training and testing.
  • UCID & BOSSbase: Used to test cross-dataset generalization (training on RAISE, testing on these).
  • VGGFace2-HQ: A deepfake dataset used to verify the model's effectiveness on AI-generated face swaps.
  • R-SMUD & V-SMUD: Datasets containing images shared across social media platforms (Facebook, Twitter, Flickr) to test "sharing chain" detection.

Key Insights and Ideas

  • Pooling is the Enemy of Forensics: While pooling is a staple of computer vision (like object detection), it is detrimental here. The paper demonstrates that convolution preserves pixel distributions, while pooling destroys them (see Fig. 3 in the paper).
  • Color Correlation as a Fingerprint: The authors proved that different operations affect the statistical correlation between R, G, and B channels in unique ways. TMFNet is one of the first models to explicitly "mine" these latent relationships as a primary forensic clue.
  • Hybrid Filter Design: By combining the statistical strengths of the Spatial Rich Model (SRM) with Gaussian blurring, their custom RGB filters achieve near-zero neighborhood pixel correlation, effectively "blinding" the model to the image's actual content so it can focus purely on the noise left by editing.

Key Results

  • Superior Accuracy: TMFNet achieved an average detection accuracy of 99.19% on grayscale chains and 99.44% on color chains, consistently outperforming previous state-of-the-art models like TransDetect and MISLnet.
  • Robustness to Compression: Even when images were heavily compressed (JPEG Quality Factor of 70), TMFNet maintained 88.97% accuracy, significantly higher than its competitors.
  • Real-World Generalization: In cross-dataset tests, TMFNet improved accuracy by roughly 2.6% to 3.5% over existing methods, suggesting it is much more "field-ready" for diverse image sources.
  • Efficiency: Despite its two-stream complexity, the model is computationally efficient, requiring only 1.08 TFLOPs—about 36% less than the previous leading method (TransDetect).

Limitations and Future Work

The authors acknowledge a classic forensic challenge: Chain Length. As the number of operations in a chain increases (e.g., 5 or more), later operations tend to "erase" the fingerprints of earlier ones. While TMFNet handles short and medium chains exceptionally well, its performance begins to dip as the chain grows longer.

Future work may involve exploring Transformer-based architectures to better capture long-range dependencies in operation sequences or developing methods to localize where in an image specific operations were applied, rather than just identifying the sequence for the whole image.

Summary updated

Abstract

Image operation chain detection techniques have gained increasing attention recently in the field of multimedia forensics. However, existing detection methods suffer from the generalization problem. Moreover, the channel correlation of color images that provides additional forensic evidence is often ignored. To solve these issues, in this article, we propose a novel two-stream multi-channels fusion networks for color image operation chain detection in which the spatial artifact stream and the noise residual stream are explored in a complementary manner. Specifically, we first propose a novel deep residual architecture without pooling in the spatial artifact stream for learning the global features representation of multi-channel correlation. Then, a set of filters is designed to aggregate the correlation information of multi-channels while capturing the low-level features in the noise residual stream. Subsequently, the high-level features are extracted by the deep residual model. Finally, features from the two streams are fed into a fusion module, to effectively learn richer discriminative representations of the operation chain. Extensive experiments show that the proposed method achieves state-of-the-art generalization ability while maintaining robustness to JPEG compression. The source code used in these experiments will be released at https://github.com/LeiTan-98/TMFNet.
BibTeX
@misc{niu2024tmfnet,
  author = {Yakun Niu and Lei Tan and Lei Zhang and Xianyu Zuo},
  title = {{TMFNet: Two-Stream Multi-Channels Fusion Networks for Color Image Operation Chain Detection}},
  year = {2024},
  eprint = {2409.07701},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07701},
}
Figure