多尺度EPI融合的密集光场解耦重建

doi:10.12066/j.issn.1007-2861.2663

上海大学学报(自然科学版) ›› 2025, Vol. 31 ›› Issue (3): 530-542.doi: 10.12066/j.issn.1007-2861.2663

多尺度EPI融合的密集光场解耦重建

曹捷^1,2, 吴玉静³, 张倩^1,2, 孟春丽¹, 严涛⁴

1. 上海师范大学信息与机电工程学院, 上海 200234;
2. 上海智能教育大数据工程技术研究中心, 上海 200234;
3. 复旦大学附属中学, 上海 200433;
4. 莆田学院人工智能学院, 福建莆田 351100

收稿日期:2024-10-22 出版日期:2025-06-30 发布日期:2025-07-22
通讯作者: 张倩(1983-),女,副教授,研究方向为视频和图像信息处理. E-mail:qianzhang@shnu.edu.cn
基金资助:
国家自然科学基金资助项目(62301320);福建省自然科学基金资助项目(2023J011009);青少年身心成长的智能监测与评估资助项目(2023YFC3305802);莆田市科技局资助项目(2021G2001ptxy08);莆田学院人才科研启动经费资助项目(2019003)

Dense light field decoupling reconstruction based on multiscale EPI fusion

CAO Jie^1,2, WU Yujing³, ZHANG Qian^1,2, MENG Chunli¹, YAN Tao⁴

1. The College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai 200234, China;
2. Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai 200234, China;
3. High School Affiliated to Fudan University, Shanghai 200433, China;
4. School of Artificial Intelligence, Putian University, Putian 351100, Fujian, China

Received:2024-10-22 Online:2025-06-30 Published:2025-07-22

摘要/Abstract

摘要： 为了充分挖掘光场极平面视图(epipolar plane image,EPI)之间的内在相关性,同时强化对空间信息的有效捕捉,提出了一种多尺度EPI信息融合的密集光场解耦重建方法.该方法对空间维度和极平面维度进行了更深层次的特征利用,能够更好地捕捉子孔径视图间的角度相关性,通过解耦与融合多种信息,提高了光场重建的精度和效果.首先,在四维光场数据的基础上,增加了密集的空间维度,提升了网络的泛化能力,并增强了其对图像局部结构和纹理信息的理解.其次,为了更好地补充和增强极平面间的信息互补性,设计了一个极平面融合模块,并提出了一种新的多尺度卷积注意力机制来融合特征信息.该注意力机制通过多尺度特征提取与全局关注机制,能有效捕捉角度信息,增强重要特征表达并抑制冗余内容.最后,在HCInew、HCIold和Stanford等光场数据集上进行实验.结果表明,本方法在峰值信噪比(peak signal-to-noise ratio,PSNR)和结构相似性(structural similarity,SSIM)等评价指标上优于现有的最先进的SOTA (state-of-the-art)方法,在大多数测试场景中实现了更好的重建效果.

关键词: 光场, 光场解耦, 密集特征提取, 极平面融合, 角度超分辨率

Abstract: To fully exploit the inherent correlation between light field epipolar plane images（EPI） and strengthen the effective capture of spatial information, this study proposed a dense light field decoupling reconstruction method based on multiscale EPI information fusion. This method utilized the spatial and epipolar plane dimensions to enable a better capture of the angular correlations between subaperture views. By decoupling and fusing multiple types of information, the accuracy and effectiveness of light field reconstruction could be enhanced. First, based on four-dimensional light field data, additional dense spatial dimensions were introduced to improve the generalization capability of the network and enhanced its understanding of local structures and texture details in images. Second, to better complement and enhance the mutual information among epipolar planes, an epipolar plane fusion module was designed along with a novel multiscale convolutional attention mechanism to integrate feature information. This attention mechanism effectively captured angular correlations through multiscale feature extraction and a global attention mechanism, thereby enhancing the expression of critical features while suppressing redundant content. Finally, experiments conducted on light field datasets, such as HCInew, HCIold,and Stanford, demonstrated that the proposed method outperformed existing state-of-theart（SOTA） approaches in terms of evaluation metrics including the peak signal-to-noise ratio（PSNR） and structural similarity（SSIM）. The proposed method achieved superior reconstruction performance in most test scenarios.

Key words: light field, light field decoupling, dense feature extraction, epipolar plane images（EPI） fusion, angular super-resolution

中图分类号:

TP919.81

曹捷, 吴玉静, 张倩, 孟春丽, 严涛. 多尺度EPI融合的密集光场解耦重建[J]. 上海大学学报(自然科学版), 2025, 31(3): 530-542.

CAO Jie, WU Yujing, ZHANG Qian, MENG Chunli, YAN Tao. Dense light field decoupling reconstruction based on multiscale EPI fusion[J]. Journal of Shanghai University（Natural Science Edition）, 2025, 31(3): 530-542.

参考文献

[1] Cui Z, Sheng H, Yang D, et al. Light field depth estimation for non-lambertian objects via adaptive cross operator[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(2):1199-1211.
[2] Chao W, Wang X, Wang Y, et al. Learning sub-pixel disparity distribution for light field depth estimation[J]. IEEE Transactions on Computational Imaging, 2023, 9:1126-1138.
[3] Yang D, Cui Z, Sheng H, et al. An occlusion and noise-aware stereo framework based on light field imaging for robust disparity estimation[J]. IEEE Transactions on Computers, 2023, 73(3):764-777.
[4] Chen Y, Jiang G, Yu M, et al. Learning zero-shot dense light field reconstruction from heterogeneous imaging[J]. Information Fusion, 2024, 103:102088.
[5] Kwon K H, Erdenebat M U, Kim N, et al. Image quality enhancement of 4D light field microscopy via reference impge propagation-based one-shot learning[J]. Applied Intelligence, 2023, 53(20):23834-23852.
[6] Piao Y, Jiang Y, Zhang M, et al. PANet:patch-aware network for light field salient object detection[J]. IEEE Transactions on Cybernetics, 2021, 53(1):379-391.
[7] Zhang M, Ji W, Piao Y, et al. LFNet:light field fusion network for salient object detection[J]. IEEE Transactions on Image Processing, 2020, 29:6276-6287.
[8] Ma J, Li Z, Cheng J, et al. Light field image super-resolution based on dual learning and deep Fourier channel attention[J]. Optics Letters, 2024, 49(11):2886-2889.
[9] Zhou S, Hu L, Wang Y, et al. AIF-LFNet:all-in-focus light field super-resolution method considering the depth-varying defocus[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8):3976-3988.
[10] Wang Y, Wang L, Wu G, et al. Disentangling light fields for super-resolution and disparity estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1):425-443.
[11] Zhang S, Chang S, Lin Y. End-to-end light field spatial super-resolution network using multiple epipolar geometry[J]. IEEE Transactions on Image Processing, 2021, 30:5956-5968.
[12] Liu G, Yue H, Wu J, et al. Efficient light field angular super-resolution with sub-aperture feature learning and macro-pixel upsampling[J]. IEEE Transactions on Multimedia, 2022, 25:6588-6600.
[13] Jin J, Hou J, Yuan H, et al. Learning light field angular super-resolution via a geometry-aware network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020:11141-11148.
[14] Wanner S, Goldluecke B. Variational light field analysis for disparity estimation and superresolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(3):606-619.
[15] Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras[J]. ACM Transactions on Graphics, 2016, 35(6):1-10.
[16] Shi J, Jiang X, Guillemot C. Learning fused pixel and feature-based view reconstructions for light fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:2555-2564.
[17] Meng N, Li K, Liu J, et al. Light field view synthesis via aperture disparity and warping confidence map[J]. IEEE Transactions on Image Processing, 2021, 30:3908-3921.
[18] Liu G, Yue H, Li K, et al. Adaptive pixel aggregation for joint spatial and angular superresolution of light field images[J]. Information Fusion, 2024, 104:102183.
[19] Shi L, Hassanieh H, Davis A, et al. Light field reconstruction using sparsity in the continuous Fourier domain[J]. ACM Transactions on Graphics, 2014, 34(1):12-24.
[20] Wu G, Zhao M, Wang L, et al. Light field reconstruction using deep convolutional network on EPI[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:6319-6327.
[21] Yeung H W F, Hou J, Chen J, et al. Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues[C]//Proceedings of the European Conference on Computer Vision. 2018:137-152.
[22] Wu G, Liu Y, Dai Q, et al. Learning sheared EPI structure for light field reconstruction[J]. IEEE Transactions on Image Processing, 2019, 28(7):3261-3273.
[23] Honauer K, Johannsen O, Kondermann D. A dataset and evaluation methodology for depth estimation on 4D light fields[M]. Berlin:Springer International Publishing, 2017:19-34.
[24] Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields[J]. Vision, Modeling & Visualization, 2013, 13:225-226.
[25] Wang Y, Liu F, Wang Z, et al. End-to-end view synthesis for light field imaging with pseudo 4DCNN[C]//Proceedings of the European Conference on Computer Vision. 2018:333-348.
[26] Jin J, Hou J, Chen J, et al. Deep coarse-to-fine dense light field reconstruction with flexible sampling and geometry-aware fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(4):1819-1836.
[27] Wu G, Liu Y, Fang L, et al. Light field reconstruction using convolutional network on EPI and extended applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(7):1681-1694.
[28] Wang Y, Liu F, Zhang K, et al. High-fidelity view synthesis for light field imaging with extended pseudo 4DCNN[J]. IEEE Transactions on Computational Imaging, 2020, 6:830-842.
[29] Zhong L, Zong B, Wang Q, et al. Implicit epipolar geometric function based light field continuous angular representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023:3463-3472.
[30] Xia P, Lu Y, Zhang S, et al. Revisiting large kernel convolution for light field image angular super-resolution[C]//2024 IEEE International Conference on Multimedia and Expo. 2024:1-6.

多尺度EPI融合的密集光场解耦重建

Dense light field decoupling reconstruction based on multiscale EPI fusion

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	刘发国, 白晓琦, 张倩, 王斌, 司文. 基于多特征融合和几何感知网络的光场图像编码[J]. 上海大学学报(自然科学版), 2024, 30(4): 669-681.
[2]	章介伦,沈文达. 具有q^4的非谐振子与谐振子的坐标二阶耦合系统的演化[J]. 上海大学学报(自然科学版), 1995, 1(2): 133-137.