收稿日期: 2022-10-28
网络出版日期: 2023-03-28
基金资助
国家自然科学基金面上资助项目(61572308);国家自然科学基金联合基金资助项目(U20B2051)
Multi-granularity fusion-based image inpainting network resistant to deep forensics
Received date: 2022-10-28
Online published: 2023-03-28
数字图像的真伪判别是图像安全领域中的基础问题, 因数字媒体极易被攻击篡改, 针对图像的取证技术得到了广泛的研究. 另一方面, 对图像篡改反取证技术的研究, 不仅追求更逼真的图像篡改操作, 也从相反的方向促进了取证技术的发展. 图像修复作为基础的图像篡改操作, 一直是国内外学者的研究热点. 针对被修复篡改后的图像会被深度取证网络取证的问题, 提出了一种抗深度取证的多粒度融合图像修复(multi-granularity fusion-based image inpainting network resistant to deep forensics, MGFR) 网络. MGFR 网络包括编解码器、 多粒度生成模块以及多粒度注意力模块. 首先, 输入的破损图像被编码器编码成深度特征, 深度特征通过多粒度生成模块生成3 个不同粒度中间特征; 然后, 采用多粒度注意力模块来计算不同粒度中间特征之间的相关性并将其融合; 最后, 融合特征通过解码器生成输出结果. 另外, 所提出的MGFR 网络被重建损失、 模式噪声损失、 深度取证损失以及对抗损失联合监督. 研究结果显示, 所提出的MGFR 网络在拥有较好的修复性能的同时能成功规避深度取证网络的取证.
窦立云, 冯国瑞, 钱振兴, 张新鹏 . 抗深度取证的多粒度融合图像修复网络[J]. 上海大学学报(自然科学版), 2023 , 29(1) : 10 -23 . DOI: 10.12066/j.issn.1007-2861.2456
Considering that it is relatively easy to use technological tools to attack and tamper with digital media, image forensics technology has been studied extensively in the field of image security. In addition to developing more realistic image forgery operations, research on image forgery anti-forensics technology also promotes forensics technology development in the opposite direction. The process of image inpainting has always been a research hotspot. This paper proposes a multi-granularity fusion-based image inpainting network resistant to deep forensics (MGFR) network. MGFR network comprises three parts: a codec, multi-granularity generation module, and a multi-granularity attention module. First, the encoder encodes the input damaged image into depth features, and then the depth features are generated by the multi-granularity generation module into three intermediate features. Subsequently, we use the multi-granularity attention module to fuse the intermediate features of different granularities. As a final step, the fused features are passed through the decoder to produce the output. Additionally, the proposed MGFR is jointly supervised by reconstruction loss, pattern noise loss, deep forensic loss, and adversarial loss. Experimental results reveal that the proposed MGFR avoids the forensics of deep forensics networks while maintaining decent inpainting performance.
Key words： multimedia security; anti-forensics; image inpainting; multi-granularity
| [1] | 谢从华, 张冰, 高蕴梅. 特征点聚类高精度视差图像拼接[J]. 中国图象图形学报, 2000, 25(6): 1180-1189. |
| [2] | Fan Q, Zhang L F. A novel patch matching algorithm for exemplar-based image inpainting[J]. Multimedia Tools Appl, 2018, 77(9): 10807-10821. |
| [3] | 林晓, 周云翔, 李大志, 等. 利用多尺度特征联合注意力模型的图像修复[J]. 计算机辅助设计与图形学学报, 2022, 34(8): 1260-1271. |
| [4] | Itier V, Strauss O, Morel L, et al. Color noise correlation-based splicing detection for image forensics[J]. Multimedia Tools and Applications, 2021, 80(9): 13215-13233. |
| [5] | Jaiswal A K, Srivastava R. Forensic image analysis using inconsistent noise pattern[J]. Pattern Analysis and Applications, 2021, 24(2): 655-667. |
| [6] | Cozzolino D, Verdoliva L. Noiseprint: a CNN-based camera model fingerprint[J]. IEEE Transactions on Information Forensics and Security, 2020, 15: 144-159. |
| [7] | Lin X F, Li C T. On constructing a better correlation predictor for PRNU-based image forgery localization[C]// IEEE International Conference on Multimedia and Expo (ICME). 2021: 1-6. |
| [8] | Dou L Y, Qian Z X, Qin C, et al. Anti-forensics of diffusion-based image inpainting[J]. J Electronic Imaging, 2020, 29(4): 043026. |
| [9] | Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]// The 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000: 417-424. |
| [10] | Nazeri K, Ng E, Joseph T, et al. EdgeConnect: structure guided image inpainting using edge prediction[C]// International Conference on Computer Vision Workshops (ICCV). 2019: 3265-3274. |
| [11] | Wang N, Zhang Y P, Zhang L F. Dynamic selection network for image inpainting[J]. IEEE Transactions on Image Processing, 2021, 30: 1784-1798. |
| [12] | Li H D, Luo W Q, Huang J W. Localization of diffusion-based inpainting in digital images[J]. IEEE Transactions on Information Forensics and Security, 2017, 12(12): 3050-3064. |
| [13] | Zhang Y J, Liu T T, Cattani C, et al. Diffusion-based image inpainting forensics via weighted least squares filtering enhancement[J]. Multimedia Tools and Applications, 2021, 80(20): 30725-30739. |
| [14] | Wu Y, Abd-Almageed W, Natarajan P. Image copy-move forgery detection via an end-to-end deep neural network[C]// The 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 1907-1915. |
| [15] | Mayer O, Stamm M C. Forensic similarity for digital images[J]. IEEE Transactions on Information Forensics and Security, 2020, 15: 1331-1346. |
| [16] | Zhu Y, Chen C F, Yan G, et al. AR-net: adaptive attention and residual refinement network for copy-move forgery detection[J]. IEEE Transactions on Industrial Informatics, 2020, 16(10): 6714-6723. |
| [17] | Wu H W, Zhou J T. IID-Net: image inpainting detection network via neural architecture search and attention[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(3): 1172-1185. |
| [18] | Böhme C, Kirchner M. Counter-forensics: attacking image forensics[C]// Digital Image Forensics. 2012: 327-366. |
| [19] | Barni M, Stamm M C, Tondi B. Adversarial multimedia forensics: overview and challenges ahead[C]// The 26th European Signal Processing Conference (EUSIPCO). 2018: 962-966. |
| [20] | Qian Z X, Zhang X P. Improved anti-forensics of JPEG compression[J]. Journal of Systems and Software, 2014, 91: 100-108. |
| [21] | Wu Z H, Stamm M C, Liu K J R. Anti-forensics of median filtering[C]// IEEE International Conference on Acoustics, Speech, and Signal Processing. 2013: 3043-3047. |
| [22] | Stamm M C, Tjoa S K, Lin W S, et al. Anti-forensics of JPEG compression[C]// The IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2010: 1694-1697. |
| [23] | Fan Z G, Queiroz R L D. Identification of bitmap compression history: JPEG detection and quantizer estimation[J]. IEEE Transactions on Image Processing, 2003, 12(2): 230-235. |
| [24] | Fan W, Wang K, Cayre F, et al. Median filtered image quality enhancement and anti-forensics via variationa[J]. IEEE Transactions on Information Forensics and Security, 2015, 10(5): 1076-1091. |
| [25] | Amerini I, Barni M, Caldelli R, et al. Counter-forensics of SIFT-based copy-move detection by means of keypoint classification[J]. EURASIP Journal on Image and Video Processing, 2013, 18: 1-17. |
| [26] | Gloe T, Kirchner M, Winkler A, et al. Can we trust digital image forensics?[C]// The 15th ACM International Conference on Multimedia. 2007: 78-86. |
| [27] | Goljan M, Fridrich J, Chen M. Sensor noise camera identification: countering counter-forensics[C]// SPIE Media Forensics and Security Ⅱ. 2010: 75410S. |
| [28] | KarakÜçÜk A, Dirik A. Adaptive photo-response non-uniformity noise removal against image source attribution[J]. Digital Investigation, 2015, 12: 66-76. |
| [29] | Huang Y Z. Can digital image forgery detection be unevadable? A case study: color filter array interpolation statistical feature recovery[C]// SPIE Visual Communications and Image Processing. 2005: 980-991. |
| [30] | Kirchner M, Böhme R. Synthesis of color filter array pattern in digital images[C]// SPIE Media Forensics and Security Ⅰ. 2009: 72540K. |
| [31] | Stamm M C, Liu K J. Protection against reverse engineering in digital cameras[C]// The International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2013: 8702-8706. |
| [32] | Zhang X, Karaman S, Chang S F. Detecting and simulating artifacts in GAN fake images[C]// IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-6. |
| [33] | Du M, Pentyala S, Li Y N, et al. Towards generalizable deepfake detection with locality-aware AutoEncoder[C]// The 29th ACM International Conference on Information & Knowledge Management (CIKM). 2020: 325-334. |
| [34] | Wang S Y, Wang O, Zhang R, et al. CNN-generated images are surprisingly easy to spot for now[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 8692-8701. |
| [35] | Neves J C, Tolosana R, Vera-Rodríguez R, et al. GANprintR: improved fakes and evaluation of the state of the art in face manipulation detection[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(5): 1039-1048. |
| [36] | Cozzolino D, Ties J, Rossler A, et al. SpoC: spoofing camera fingerprints[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 990-1000. |
| [37] | Wu J, Sun W. Towards multi-operation image anti-forensics with generative adversarial networks[J]. Computers & Security, 2021, 100: 102083. |
| [38] | Yu J H, Lin Z, Yang J M, et al. Free-form image inpainting with gated convolution[C]// IEEE/CVF International Conference on Computer Vision (ICCV). 2019: 4470-4479. |
| [39] | Wang P Q, Chen P F, Yuan Y, et al. Understanding convolution for semantic segment-ation[C]// IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 1451-1460. |
| [40] | Hu X F, Zhang Z H, Jiang Z Y, et al. SPAN: spatial pyramid attention network for image manipulation localization[C]// European Conference on Computer Vision (ECCV). 2020: 312-328. |
| [41] | Park D Y, Lee K H. Arbitrary style transfer with style-attentional networks[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019: 5880-5888. |
| [42] | Huang X, Belongie S J. Arbitrary style transfer in real-time with adaptive instance normalization[C]// IEEE International Conference on Computer Vision (ICCV). 2017: 1510-1519. |
| [43] | Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. 2018: 3-19. |
| [44] | Lukas J, Fridrich J, Goljan M. Digital camera identification from sensor noise[J]. IEEE Transactions on Information Forensics and Security, 2006, 1(2): 205-214. |
| [45] | Wang Y, Tao X, Qi X J, et al. Image inpainting via generative multi-column convolutional neural networks[C]// Advances in Neural Information Processing Systems (NIPS). 2018: 329-338. |
| [46] | Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of Wasserstein GANs[C]// Advances in Neural Information Processing Systems (NIPS). 2017: 5767-5777. |
| [47] | Zhou B, Lapedriza A, Khosla A, et al. Places: a 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464. |
| [48] | Liu G L, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C]// European Conference on Computer Vision. 2018: 89-105. |
| [49] | Li J Y, Wang N, Zhang L F, et al. Recurrent feature reasoning for image inpainting[C]// The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 7757-7765. |
| [50] | Wu H W, Zhou J T, Li Y M. Deep generative model for image inpainting with local binary pattern learning and spatial attention[J]. IEEE Transactions on Multimedia, 2022, 24: 4016-4027. |
| [51] | Hong X, Xiong P F, Ji R H, et al. Deep fusion network for image completion[C]// The 27th ACM International Conference on Multimedia (MM). 2019: 2033-2042. |
| [52] | Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. |
/
| 〈 |
|
〉 |
