为了充分挖掘光场极平面视图(epipolar plane image,EPI)之间的内在相关性,同时强化对空间信息的有效捕捉,提出了一种多尺度EPI信息融合的密集光场解耦重建方法.该方法对空间维度和极平面维度进行了更深层次的特征利用,能够更好地捕捉子孔径视图间的角度相关性,通过解耦与融合多种信息,提高了光场重建的精度和效果.首先,在四维光场数据的基础上,增加了密集的空间维度,提升了网络的泛化能力,并增强了其对图像局部结构和纹理信息的理解.其次,为了更好地补充和增强极平面间的信息互补性,设计了一个极平面融合模块,并提出了一种新的多尺度卷积注意力机制来融合特征信息.该注意力机制通过多尺度特征提取与全局关注机制,能有效捕捉角度信息,增强重要特征表达并抑制冗余内容.最后,在HCInew、HCIold和Stanford等光场数据集上进行实验.结果表明,本方法在峰值信噪比(peak signal-to-noise ratio,PSNR)和结构相似性(structural similarity,SSIM)等评价指标上优于现有的最先进的SOTA (state-of-the-art)方法,在大多数测试场景中实现了更好的重建效果.
To fully exploit the inherent correlation between light field epipolar plane images（EPI） and strengthen the effective capture of spatial information, this study proposed a dense light field decoupling reconstruction method based on multiscale EPI information fusion. This method utilized the spatial and epipolar plane dimensions to enable a better capture of the angular correlations between subaperture views. By decoupling and fusing multiple types of information, the accuracy and effectiveness of light field reconstruction could be enhanced. First, based on four-dimensional light field data, additional dense spatial dimensions were introduced to improve the generalization capability of the network and enhanced its understanding of local structures and texture details in images. Second, to better complement and enhance the mutual information among epipolar planes, an epipolar plane fusion module was designed along with a novel multiscale convolutional attention mechanism to integrate feature information. This attention mechanism effectively captured angular correlations through multiscale feature extraction and a global attention mechanism, thereby enhancing the expression of critical features while suppressing redundant content. Finally, experiments conducted on light field datasets, such as HCInew, HCIold,and Stanford, demonstrated that the proposed method outperformed existing state-of-theart（SOTA） approaches in terms of evaluation metrics including the peak signal-to-noise ratio（PSNR） and structural similarity（SSIM）. The proposed method achieved superior reconstruction performance in most test scenarios.
[1] Cui Z, Sheng H, Yang D, et al. Light field depth estimation for non-lambertian objects via adaptive cross operator[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(2):1199-1211.
[2] Chao W, Wang X, Wang Y, et al. Learning sub-pixel disparity distribution for light field depth estimation[J]. IEEE Transactions on Computational Imaging, 2023, 9:1126-1138.
[3] Yang D, Cui Z, Sheng H, et al. An occlusion and noise-aware stereo framework based on light field imaging for robust disparity estimation[J]. IEEE Transactions on Computers, 2023, 73(3):764-777.
[4] Chen Y, Jiang G, Yu M, et al. Learning zero-shot dense light field reconstruction from heterogeneous imaging[J]. Information Fusion, 2024, 103:102088.
[5] Kwon K H, Erdenebat M U, Kim N, et al. Image quality enhancement of 4D light field microscopy via reference impge propagation-based one-shot learning[J]. Applied Intelligence, 2023, 53(20):23834-23852.
[6] Piao Y, Jiang Y, Zhang M, et al. PANet:patch-aware network for light field salient object detection[J]. IEEE Transactions on Cybernetics, 2021, 53(1):379-391.
[7] Zhang M, Ji W, Piao Y, et al. LFNet:light field fusion network for salient object detection[J]. IEEE Transactions on Image Processing, 2020, 29:6276-6287.
[8] Ma J, Li Z, Cheng J, et al. Light field image super-resolution based on dual learning and deep Fourier channel attention[J]. Optics Letters, 2024, 49(11):2886-2889.
[9] Zhou S, Hu L, Wang Y, et al. AIF-LFNet:all-in-focus light field super-resolution method considering the depth-varying defocus[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8):3976-3988.
[10] Wang Y, Wang L, Wu G, et al. Disentangling light fields for super-resolution and disparity estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1):425-443.
[11] Zhang S, Chang S, Lin Y. End-to-end light field spatial super-resolution network using multiple epipolar geometry[J]. IEEE Transactions on Image Processing, 2021, 30:5956-5968.
[12] Liu G, Yue H, Wu J, et al. Efficient light field angular super-resolution with sub-aperture feature learning and macro-pixel upsampling[J]. IEEE Transactions on Multimedia, 2022, 25:6588-6600.
[13] Jin J, Hou J, Yuan H, et al. Learning light field angular super-resolution via a geometry-aware network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020:11141-11148.
[14] Wanner S, Goldluecke B. Variational light field analysis for disparity estimation and superresolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(3):606-619.
[15] Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras[J]. ACM Transactions on Graphics, 2016, 35(6):1-10.
[16] Shi J, Jiang X, Guillemot C. Learning fused pixel and feature-based view reconstructions for light fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:2555-2564.
[17] Meng N, Li K, Liu J, et al. Light field view synthesis via aperture disparity and warping confidence map[J]. IEEE Transactions on Image Processing, 2021, 30:3908-3921.
[18] Liu G, Yue H, Li K, et al. Adaptive pixel aggregation for joint spatial and angular superresolution of light field images[J]. Information Fusion, 2024, 104:102183.
[19] Shi L, Hassanieh H, Davis A, et al. Light field reconstruction using sparsity in the continuous Fourier domain[J]. ACM Transactions on Graphics, 2014, 34(1):12-24.
[20] Wu G, Zhao M, Wang L, et al. Light field reconstruction using deep convolutional network on EPI[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:6319-6327.
[21] Yeung H W F, Hou J, Chen J, et al. Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues[C]//Proceedings of the European Conference on Computer Vision. 2018:137-152.
[22] Wu G, Liu Y, Dai Q, et al. Learning sheared EPI structure for light field reconstruction[J]. IEEE Transactions on Image Processing, 2019, 28(7):3261-3273.
[23] Honauer K, Johannsen O, Kondermann D. A dataset and evaluation methodology for depth estimation on 4D light fields[M]. Berlin:Springer International Publishing, 2017:19-34.
[24] Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields[J]. Vision, Modeling & Visualization, 2013, 13:225-226.
[25] Wang Y, Liu F, Wang Z, et al. End-to-end view synthesis for light field imaging with pseudo 4DCNN[C]//Proceedings of the European Conference on Computer Vision. 2018:333-348.
[26] Jin J, Hou J, Chen J, et al. Deep coarse-to-fine dense light field reconstruction with flexible sampling and geometry-aware fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(4):1819-1836.
[27] Wu G, Liu Y, Fang L, et al. Light field reconstruction using convolutional network on EPI and extended applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(7):1681-1694.
[28] Wang Y, Liu F, Zhang K, et al. High-fidelity view synthesis for light field imaging with extended pseudo 4DCNN[J]. IEEE Transactions on Computational Imaging, 2020, 6:830-842.
[29] Zhong L, Zong B, Wang Q, et al. Implicit epipolar geometric function based light field continuous angular representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023:3463-3472.
[30] Xia P, Lu Y, Zhang S, et al. Revisiting large kernel convolution for light field image angular super-resolution[C]//2024 IEEE International Conference on Multimedia and Expo. 2024:1-6.
