中文

Journal of Shanghai University >

2023 , Vol. 29 >Issue 1: 10 - 23

DOI: https://doi.org/10.12066/j.issn.1007-2861.2456

Multimedia Information Processing

Multi-granularity fusion-based image inpainting network resistant to deep forensics

Expand
  • 1. College of Communication and Information Engineering, Shanghai University, Shanghai 200444, China
    2. School of Computer Science and Technology, Fudan University, Shanghai 200433, China

Received date: 2022-10-28

  Online published: 2023-03-28

Fold

Abstract

Considering that it is relatively easy to use technological tools to attack and tamper with digital media, image forensics technology has been studied extensively in the field of image security. In addition to developing more realistic image forgery operations, research on image forgery anti-forensics technology also promotes forensics technology development in the opposite direction. The process of image inpainting has always been a research hotspot. This paper proposes a multi-granularity fusion-based image inpainting network resistant to deep forensics (MGFR) network. MGFR network comprises three parts: a codec, multi-granularity generation module, and a multi-granularity attention module. First, the encoder encodes the input damaged image into depth features, and then the depth features are generated by the multi-granularity generation module into three intermediate features. Subsequently, we use the multi-granularity attention module to fuse the intermediate features of different granularities. As a final step, the fused features are passed through the decoder to produce the output. Additionally, the proposed MGFR is jointly supervised by reconstruction loss, pattern noise loss, deep forensic loss, and adversarial loss. Experimental results reveal that the proposed MGFR avoids the forensics of deep forensics networks while maintaining decent inpainting performance.

Key words： multimedia security; anti-forensics; image inpainting; multi-granularity

Cite this article

DOU Liyun, FENG Guorui, QIAN Zhenxing, ZHANG Xinpeng . Multi-granularity fusion-based image inpainting network resistant to deep forensics[J]. Journal of Shanghai University, 2023 , 29(1) : 10 -23 . DOI: 10.12066/j.issn.1007-2861.2456

References

[1] 谢从华, 张冰, 高蕴梅. 特征点聚类高精度视差图像拼接[J]. 中国图象图形学报, 2000, 25(6): 1180-1189.
[2] Fan Q, Zhang L F. A novel patch matching algorithm for exemplar-based image inpainting[J]. Multimedia Tools Appl, 2018, 77(9): 10807-10821.
[3] 林晓, 周云翔, 李大志, 等. 利用多尺度特征联合注意力模型的图像修复[J]. 计算机辅助设计与图形学学报, 2022, 34(8): 1260-1271.
[4] Itier V, Strauss O, Morel L, et al. Color noise correlation-based splicing detection for image forensics[J]. Multimedia Tools and Applications, 2021, 80(9): 13215-13233.
[5] Jaiswal A K, Srivastava R. Forensic image analysis using inconsistent noise pattern[J]. Pattern Analysis and Applications, 2021, 24(2): 655-667.
[6] Cozzolino D, Verdoliva L. Noiseprint: a CNN-based camera model fingerprint[J]. IEEE Transactions on Information Forensics and Security, 2020, 15: 144-159.
[7] Lin X F, Li C T. On constructing a better correlation predictor for PRNU-based image forgery localization[C]// IEEE International Conference on Multimedia and Expo (ICME). 2021: 1-6.
[8] Dou L Y, Qian Z X, Qin C, et al. Anti-forensics of diffusion-based image inpainting[J]. J Electronic Imaging, 2020, 29(4): 043026.
[9] Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]// The 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000: 417-424.
[10] Nazeri K, Ng E, Joseph T, et al. EdgeConnect: structure guided image inpainting using edge prediction[C]// International Conference on Computer Vision Workshops (ICCV). 2019: 3265-3274.
[11] Wang N, Zhang Y P, Zhang L F. Dynamic selection network for image inpainting[J]. IEEE Transactions on Image Processing, 2021, 30: 1784-1798.
[12] Li H D, Luo W Q, Huang J W. Localization of diffusion-based inpainting in digital images[J]. IEEE Transactions on Information Forensics and Security, 2017, 12(12): 3050-3064.
[13] Zhang Y J, Liu T T, Cattani C, et al. Diffusion-based image inpainting forensics via weighted least squares filtering enhancement[J]. Multimedia Tools and Applications, 2021, 80(20): 30725-30739.
[14] Wu Y, Abd-Almageed W, Natarajan P. Image copy-move forgery detection via an end-to-end deep neural network[C]// The 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 1907-1915.
[15] Mayer O, Stamm M C. Forensic similarity for digital images[J]. IEEE Transactions on Information Forensics and Security, 2020, 15: 1331-1346.
[16] Zhu Y, Chen C F, Yan G, et al. AR-net: adaptive attention and residual refinement network for copy-move forgery detection[J]. IEEE Transactions on Industrial Informatics, 2020, 16(10): 6714-6723.
[17] Wu H W, Zhou J T. IID-Net: image inpainting detection network via neural architecture search and attention[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(3): 1172-1185.
[18] Böhme C, Kirchner M. Counter-forensics: attacking image forensics[C]// Digital Image Forensics. 2012: 327-366.
[19] Barni M, Stamm M C, Tondi B. Adversarial multimedia forensics: overview and challenges ahead[C]// The 26th European Signal Processing Conference (EUSIPCO). 2018: 962-966.
[20] Qian Z X, Zhang X P. Improved anti-forensics of JPEG compression[J]. Journal of Systems and Software, 2014, 91: 100-108.
[21] Wu Z H, Stamm M C, Liu K J R. Anti-forensics of median filtering[C]// IEEE International Conference on Acoustics, Speech, and Signal Processing. 2013: 3043-3047.
[22] Stamm M C, Tjoa S K, Lin W S, et al. Anti-forensics of JPEG compression[C]// The IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2010: 1694-1697.
[23] Fan Z G, Queiroz R L D. Identification of bitmap compression history: JPEG detection and quantizer estimation[J]. IEEE Transactions on Image Processing, 2003, 12(2): 230-235.
[24] Fan W, Wang K, Cayre F, et al. Median filtered image quality enhancement and anti-forensics via variationa[J]. IEEE Transactions on Information Forensics and Security, 2015, 10(5): 1076-1091.
[25] Amerini I, Barni M, Caldelli R, et al. Counter-forensics of SIFT-based copy-move detection by means of keypoint classification[J]. EURASIP Journal on Image and Video Processing, 2013, 18: 1-17.
[26] Gloe T, Kirchner M, Winkler A, et al. Can we trust digital image forensics?[C]// The 15th ACM International Conference on Multimedia. 2007: 78-86.
[27] Goljan M, Fridrich J, Chen M. Sensor noise camera identification: countering counter-forensics[C]// SPIE Media Forensics and Security Ⅱ. 2010: 75410S.
[28] KarakÜçÜk A, Dirik A. Adaptive photo-response non-uniformity noise removal against image source attribution[J]. Digital Investigation, 2015, 12: 66-76.
[29] Huang Y Z. Can digital image forgery detection be unevadable? A case study: color filter array interpolation statistical feature recovery[C]// SPIE Visual Communications and Image Processing. 2005: 980-991.
[30] Kirchner M, Böhme R. Synthesis of color filter array pattern in digital images[C]// SPIE Media Forensics and Security Ⅰ. 2009: 72540K.
[31] Stamm M C, Liu K J. Protection against reverse engineering in digital cameras[C]// The International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2013: 8702-8706.
[32] Zhang X, Karaman S, Chang S F. Detecting and simulating artifacts in GAN fake images[C]// IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-6.
[33] Du M, Pentyala S, Li Y N, et al. Towards generalizable deepfake detection with locality-aware AutoEncoder[C]// The 29th ACM International Conference on Information & Knowledge Management (CIKM). 2020: 325-334.
[34] Wang S Y, Wang O, Zhang R, et al. CNN-generated images are surprisingly easy to spot for now[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 8692-8701.
[35] Neves J C, Tolosana R, Vera-Rodríguez R, et al. GANprintR: improved fakes and evaluation of the state of the art in face manipulation detection[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(5): 1039-1048.
[36] Cozzolino D, Ties J, Rossler A, et al. SpoC: spoofing camera fingerprints[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 990-1000.
[37] Wu J, Sun W. Towards multi-operation image anti-forensics with generative adversarial networks[J]. Computers & Security, 2021, 100: 102083.
[38] Yu J H, Lin Z, Yang J M, et al. Free-form image inpainting with gated convolution[C]// IEEE/CVF International Conference on Computer Vision (ICCV). 2019: 4470-4479.
[39] Wang P Q, Chen P F, Yuan Y, et al. Understanding convolution for semantic segment-ation[C]// IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 1451-1460.
[40] Hu X F, Zhang Z H, Jiang Z Y, et al. SPAN: spatial pyramid attention network for image manipulation localization[C]// European Conference on Computer Vision (ECCV). 2020: 312-328.
[41] Park D Y, Lee K H. Arbitrary style transfer with style-attentional networks[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019: 5880-5888.
[42] Huang X, Belongie S J. Arbitrary style transfer in real-time with adaptive instance normalization[C]// IEEE International Conference on Computer Vision (ICCV). 2017: 1510-1519.
[43] Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. 2018: 3-19.
[44] Lukas J, Fridrich J, Goljan M. Digital camera identification from sensor noise[J]. IEEE Transactions on Information Forensics and Security, 2006, 1(2): 205-214.
[45] Wang Y, Tao X, Qi X J, et al. Image inpainting via generative multi-column convolutional neural networks[C]// Advances in Neural Information Processing Systems (NIPS). 2018: 329-338.
[46] Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of Wasserstein GANs[C]// Advances in Neural Information Processing Systems (NIPS). 2017: 5767-5777.
[47] Zhou B, Lapedriza A, Khosla A, et al. Places: a 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464.
[48] Liu G L, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C]// European Conference on Computer Vision. 2018: 89-105.
[49] Li J Y, Wang N, Zhang L F, et al. Recurrent feature reasoning for image inpainting[C]// The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 7757-7765.
[50] Wu H W, Zhou J T, Li Y M. Deep generative model for image inpainting with local binary pattern learning and spatial attention[J]. IEEE Transactions on Multimedia, 2022, 24: 4016-4027.
[51] Hong X, Xiong P F, Ji R H, et al. Deep fusion network for image completion[C]// The 27th ACM International Conference on Multimedia (MM). 2019: 2033-2042.
[52] Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
Options
Outlines

/

〈 〉