提出了一种面向数字孪生的深层特征融合低照度图像去雨线方法.首先,创建动态线性卷积(dynamic line convolution,DLConv),学习雨线的先验知识,提高对雨线的特征提取能力,解决伪影问题.其次,结合先验知识和坐标注意力(coordinate attention,CA)机制,改进U形模型,生成雨线的深层特征,解决细节丢失问题.最后,利用图像的亮度信息进一步细化加强深层特征,得到每一级的无雨图像.在此基础上,各级之间融合无雨图像的特征,并通过跨级特征融合(cross-stage feature fusion,CSFF)传递多尺度特征,实现低照度图像去雨线.在最新的夜间雨天图像数据集上进行实验,结果表明,本方法在峰值信噪比(peak signal-to-noise ratio,PSNR)、结构相似性(structural similarity,SSIM)、学习感知图像块相似度(learned perceptual image patch similarity,LPIPS)和自然图像质量评估器(natural image quality evaluator,NIQE)指标上分别达到了38.522 9 dB、0.974 5、0.061 9和4.207 2,优于现有的主流方法,证明了本方法能有效改善伪影和细节丢失问题,大大提升了低照度图像的可用性.
A deep feature fusion method was proposed for low-light images rain streak removal in digital twin applications. First, the dynamic line convolution（DLConv） was created to learn the prior knowledge of rain streaks, enhancing the ability to extract rain streak features to address the artifact problem. Second, by combining prior knowledge with the coordinate attention（CA） mechanism, the U-shaped model was improved to generate deep features of rain streaks, thereby addressing the issue of detail loss. Finally, the brightness information of the images was utilized to further refine and enhance the deep features and obtain rain-free images at the current stage, where the features of the rainfree images were fused between the stages, and multiscale features were transferred through cross-stage feature fusion, achieving rain streak removal for low-light images. Experimental results on the latest low-light rainy image dataset showed that the proposed method achieved peak signal-to-noise ratio（PSNR）、structural similarity（SSIM）、learned perceptual image patch similarity（LPIPS） and natural image quality evaluator（NIQE） values of 38.522 9 dB, 0.974 5, 0.061 9 and 4.207 2, respectively, outperforming existing mainstream methods. This demonstrated the effectiveness of the proposed approach in alleviating artifacts and detail loss, thereby significantly improving the usability of low-light images.
[1] Zhang Z, Wei Y, Zhang H, et al. Data-driven single image deraining:a comprehensive review and new perspectives[J]. Pattern Recognition, 2023, 143:109740.
[2] Ren D, Zuo W, Hu Q, et al. Progressive image deraining networks:a better and simpler baseline[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019:3937-3946.
[3] 柳长源, 王琪, 毕晓君. 基于多通道多尺度卷积神经网络的单幅图像去雨方法[J]. 电子与信息学报, 2020, 42(9):2285-2292.
[4] 张学锋, 李金晶. 基于双注意力残差循环单幅图像去雨集成网络[J]. 软件学报, 2021, 32(10):3283-3292.
[5] Zamir S W, Arora A, Khan S, et al. Multi-stage progressive image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021:14821-14831.
[6] Ba Y, Zhang H, Yang E, et al. Not just streaks:towards ground truth for single image deraining[C]//European Conference on Computer Vision Cham. 2022:723-740.
[7] Liang Y, Anwar S, Liu Y. DRT:a lightweight single image deraining recursive transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022:589-598.
[8] Chen X, Li H, Li M, et al. Learning a sparse transformer network for efiective image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023:5896-5905.
[9] Zhang F, You S, Li Y, et al. Learning rain location prior for nighttime deraining[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023:13148-13157.
[10] 汤红忠, 熊珮全, 王蔚, 等. 基于双阶段特征解耦网络的单幅图像去雨方法[J]. 计算机辅助设计与图形学学报, 2024, 36(2):273-282.
[11] Chen X, Pan J, Dong J. Bidirectional multi-scale implicit neural representations for image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024:25627-25636.
[12] Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision. 2017:764-773.
[13] Qi Y, He Y, Qi X, et al. Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023:6070-6079.
[14] Hu J, Shen L, Sun G. Squeeze-and-excitation net-works[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018:7132-7141.
[15] Woo S, Park J, Lee J Y, et al. CBAM:convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision. 2018:3-19.
[16] Hou Q, Zhou D, Feng J. Coordinate attention for ecient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021:13713-13722.
[17] Jiang K, Wang Z, Yi P, et al. Multi-scale progressive fusion network for single image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:8346-8355.
[18] Yang W, Tan R T, Feng J, et al. Deep joint rain detection and removal from a single image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:1357-1366.
