基于像素流的视频彩色化

doi:10.12066/j.issn.1007-2861.2119

Abstract

Abstract:

Video colourisation methods that transfer colour information in keyframes based on traditional optical flow are time-consuming, while those relying on global colour transfer are prone to desaturation. This paper proposes a new video colourisation method based on voxel flow. In the proposed method, the reference and target images are both converted to the lab colour space, before a double-channel voxel flow is obtained by feeding the luminance channels of the images into a neural network. The voxel flow values indicate the positional colour correspondence between the target frame and the reference frame. Then, the colour of the target frame is obtained by bilinear interpolation of the reference frame utilising the voxel flow. Finally, the colour and luminance channels are combined to synthesise the final colourised image. Experimental results show that the proposed video colourisation method maintains the saturation of the reference image, while also maintaining edge sharpness. Compared with rival video colourisation methods based on traditional optical flow, the proposed method yields a higher peak signal-to-noise ratio (PSNR) and offers a shorter runtime.

Key words: colourisation, voxel flow, deep learning, neural network, optical flow

CLC Number:

TP391.41

CHEN Yu, DING Youdong, YU Bing, XU Min. Video colourisation based on voxel flow[J]. Journal of Shanghai University（Natural Science Edition）, 2021, 27(1): 18-27.

Figures/Tables 7

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Fig.5

Fig.6

References 34

[1]	Levin A, Lischinski D, Weiss Y. Colorization using optimization[J]. ACM Transactions on Graphics, 2004,23(3):689-694. doi: 10.1145/1015706.1015780
[2]	Lucas B D, Kanade T. An iterative image registration technique with an application to stereo vision[C] // 7th International Joint Conference on Artificial Intelligence (IJCAI). 1981: 674-679.
[3]	Irony R, Cohen-Or D, Lischinski D. Colorization by example[C] // Proceedings of the 16th Eruographics Conference on Rendering Techniques. 2005: 201-210.
[4]	Zhen Z, Yan G, Li Z M. An automatic image, video colorization algorithm based on pattern continuity[C] // International Conference on Audio, Language, Image Processing. 2012: 531-536.
[5]	Lang M, Wang O, Aydin T, et al. Practical temporal consistency for image-based graphics applications[J]. ACM Transactions on Graphics, 2012,31:34-42.
[6]	Yatziv L, Sapiro G. Fast image and video colorization using chrominance blending[J]. IEEE Transactions on Image Processing, 2006,15(5):1120-1129. doi: 10.1109/tip.2005.864231 pmid: 16671293
[7]	Sheng B, Sun H, Chen S, et al. Colorization using the rotation-invariant feature space[J]. IEEE Computer Graphics Applications, 2011,31(2):24-35. doi: 10.1109/MCG.2011.18 pmid: 24808026
[8]	Manjunath B S, Ma W Y. Texture features for browsing, retrieval of image data[J]. IEEE Trans Pattern Anal Machine Intelligence, 1996,18(8):837-842 doi: 10.1109/34.531803
[9]	Heu J H, Hyun D Y, Kim C S, et al. Image, video colorization based on prioritized source propagation[C] // IEEE International Conference on Image Processing. 2009: 465-468.
[10]	Paul S, Bhattacharya S, Gupta S. Spatiotemporal colorization of video using 3D steerable pyramids[J]. IEEE Transactions on Circuits, Systems for Video Technology, 2017,27(8):1605-1619. doi: 10.1109/TCSVT.2016.2539539
[11]	Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images[J]. ACM Trans Graphics, 2002,21(3):277-280. doi: 10.1145/566654.566576
[12]	Sykora D, Burianek J, Zara J. Unsupervised colorization of black-and-white cartoons[C] // The 3rd International Symposium on Non-photorealistic Animation, Rendering. 2004: 121-127.
[13]	Pan Z, Dong Z, Zhang M. A new algorithm for adding color to video or animation clips[J]. Journal of WSCG, 2004,12(1/2/3):515-519.
[14]	Teng S, Shen Y, Zhao Z, et al. An interactive framework for video colorization[C] // IEEE International Conference on Image, Graphics. 2013: 89-94.
[15]	Xia S, Liu J, Fang Y, et al. Robust, automatic video colorization via multiframe reordering refinement[C] // IEEE International Conference on Image Processing. 2016: 4017-4021.
[16]	Pierre F, Aujol J F, Bugeau A, et al. Interactive video colorization within a variational framework[J]. SIAM Journal on Imaging Sciences, 2017,10(4):2293-2325. doi: 10.1137/16M1108625
[17]	Jampani V, Gadde R, Gehler P V. Video propagation networks[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2017: 3154-3164.
[18]	Liu S, Zhong G, de Mello S, et al. Switchable temporal propagation network[C] // 15th European Conference on Computer Vision. 2018: 89-104.
[19]	Meyer S, Cornillère V, Djelouah A, et al. Deep video color propagation [EB/OL]. (2018-08-09) [2019-01-11]. https://arxiv.org/abs/1808.03232.
[20]	Lefkimmiatis S. Universal denoising networks: a novel CNN architecture for image denoising[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2018: 3204-3213.
[21]	Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2018: 5505-5514.
[22]	Chen Y S, Wang Y C, Kao M H, et al. Deep photo enhancer: unpaired learning for image enhancement from photographs with GANs[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2018: 6306-6314.
[23]	Niklaus S, Mai L, Liu F. Video frame interpolation via adaptive separable convolution[C] // IEEE International Conference on Computer Vision. 2017: 261-270.
[24]	Liu Z, Yeh R A, Tang X, et al. Video frame synconfproc using deep voxel flow[C] // IEEE International Conference on Computer Vision. 2017: 4473-4481.
[25]	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2016: 770-778.
[26]	He K, Zhang X, Ren S, et al. Delving deep into rectifiers: surpassing human-level performance on imagenet classification[C] // IEEE International Conference on Computer Vision. 2015: 1026-1034.
[27]	Long G, Kneip L, Alvarez J M, et al. Learning image matching by simply watching video[C] // European Conference on Computer Vision. 2016: 434-450.
[28]	Perazzi F, Pont-Tuset J, McWilliams B, et al. A benchmark dataset, evaluation methodology for video object segmentation[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2016: 724-732.
[29]	Pont-Tuset J, Perazzi F, Caelles S, et al. The 2017 davis challenge on video object segmentation [EB/OL]. (2018-03-01) [2019-01-11]. https://arxiv.org/abs/1704.00675.
[30]	Su S, Delbracio M, Wang J, et al. Deep video deblurring for hand-held cameras[C] // IEEE Conference on Computer Vision, Pattern Recognition. 2017: 237-246.
[31]	Kingma D P, Ba J. Adam: a method for stochastic optimization [EB/OL]. (2017-01-30) [2019-01-11]. https://arxiv.org/abs/1412.6980v9.
[32]	Soomro K, Zamir A R, Shah M. UCF101: a dataset of 101 human actions classes from videos in the wild [EB/OL]. (2012-12-03) [2019-01-11]. https://arxiv.org/abs/1212.0402.
[33]	Li Y, Liu M Y, Li X, et al. A closed-form solution to photorealistic image stylization [EB/OL]. (2018-02-19) [2019-01-11]. https://arxiv.org/abs/1802.06474.
[34]	Zhang R, Zhu J Y, Isola P, et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Transactions on Graphics, 2017,36(4):1-11.

Video colourisation based on voxel flow

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 34

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	HUANG Jiabao, ZHU Yonghua, ZHOU Jiting, GAO Wenjing. Fine-grained image classification of succulents with convolutions [J]. Journal of Shanghai University（Natural Science Edition）, 2020, 26(2): 283-291.
[2]	Lei WANG, Jingui LU, Lewei LI. Neural network for fatigue life of wind turbine blade numerical calculation of approximate model [J]. Journal of Shanghai University（Natural Science Edition）, 2019, 25(6): 870-878.
[3]	Jiacheng HU, Xiangyang WANG, Han LIU. Defect detection of continuous casting slabs based on deep learning [J]. Journal of Shanghai University（Natural Science Edition）, 2019, 25(4): 445-452.
[4]	CHEN Gaopan, XU Meihua, WANG Qi, GUO Aiying. Front-vehicle detection method based on monocular vision [J]. Journal of Shanghai University（Natural Science Edition）, 2019, 25(1): 56-65.
[5]	XU Zhengze, ZHANG Wenjun. Hand segmentation from a single depth image based on histogram threshold selection and shallow CNN [J]. Journal of Shanghai University（Natural Science Edition）, 2018, 24(5): 675-685.
[6]	ZHOU Jingyi, GUO Yan, DING Youdong. Sentiment analysis of Chinese movie reviews based on deep learning [J]. Journal of Shanghai University（Natural Science Edition）, 2018, 24(5): 703-712.
[7]	XIE Zhifeng, YE Guanhua, YAN Shuqi, HE Shaorong, DING Youdong. HDR image style transfer technique based on generative adversarial networks [J]. Journal of Shanghai University（Natural Science Edition）, 2018, 24(4): 524-534.
[8]	RAN Peng, WANG Ling, LI Xin, LIU Pengwei. Improved softmax classifier for deep convolution neural networks and its application in face recognition [J]. Journal of Shanghai University（Natural Science Edition）, 2018, 24(3): 352-366.
[9]	LIU Huibin, CHEN Qiang, WU Fei, ZHAO Yi. Classification of individual objects in focused group based on convolutional neural network [J]. Journal of Shanghai University（Natural Science Edition）, 2017, 23(6): 874-.
[10]	LANG Yue1, ZHOU Jiting1, LIANG Xiaolong2, ZHANG Wenjun1. Automatic annotation for film and Television drama shots and recut system based on face identification [J]. Journal of Shanghai University（Natural Science Edition）, 2017, 23(3): 353-363.
[11]	LIU Yanping, TIAN Jinpeng, CHEN Yong. RF fingerprinting identification with feature fusion [J]. Journal of Shanghai University（Natural Science Edition）, 2017, 23(3): 408-413.
[12]	SHEN Wenfeng, ZHANG Jianlei, ZHOU Dingqian, CHEN Shengbo, QIU Feng. Recognition of Chinese characters on license plates based on big data [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 88-96.
[13]	BU Feng, LI Chuan-jiang, CHEN Jia-jia, LI Huan, GUO Wei-hai. Design of Electromyography Prosthesis Controller Based on ARM [J]. Journal of Shanghai University（Natural Science Edition）, 2014, 20(4): 442-449.
[14]	XU Qin-le1, ZHANG Jin-yi,1,2,3, XU De-zheng1, LI Ruo-han2, ZHANG Jing-jing2. Multi-sensor Data Fusion Algorithm for High Accuracy Arc Fault Detection [J]. Journal of Shanghai University（Natural Science Edition）, 2014, 20(2): 165-173.
[15]	HUANG Qing-hua, LI Lin, LAI Shi-cun. Personalized Modeling of Head-Related Transfer Function Based on RBF Neural Network [J]. Journal of Shanghai University（Natural Science Edition）, 2014, 20(2): 157-164.