中文

Journal of Shanghai University >

2022 , Vol. 28 >Issue 3: 427 - 439

DOI: https://doi.org/10.12066/j.issn.1007-2861.2379

Data Collection, Database and Data Processing

Kalman filter based method for processing small noisy sample data

Expand
  • 1. School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China
    2. School of Materials Science and Engineering, Shanghai University, Shanghai 200444, China
    3. Center of Materials Informatics and Data Science, Materials Genome Institute, Shanghai University, Shanghai 200444, China
    4. National Engineering Research Center for Magnesium Alloys, School of Materials Science and Engineering, Chongqing University, Chongqing 400044, China
    5. Zhejiang Laboratory, Hangzhou 311100, Zhejiang, China

Received date: 2022-03-15

  Online published: 2022-05-27

Fold

Abstract

A small sample noisy data processing method based on Kalman filter and extended Kalman filter has been proposed. The core idea was to establish a system model using physical models or empirical formula, then used the system model to predict the model data, and finally used the observation data to correct the model data and achieve the effect of smoothing data noise. Experimental results showed that when using the autoregressive integrated moving average (ARIMA) model and random forest (RF) model to predict the corrosion weight gain of weather steel BC500, the coefficient of determination $R^{2}$ was increased by an average of 6.4% after Kalman filter denoising, while the $R^{2}$ was increased by an average of 4.9% after extended Kalman filter. These results verified the effectiveness of the proposed methods.

Key words： Kalman filter; extended Kalman filter; data denoising; small samples; corrosion data

Cite this article

LIU Fen, FAN Hongqiang, LÜ Tao, LI Qian, QIAN Quan . Kalman filter based method for processing small noisy sample data[J]. Journal of Shanghai University, 2022 , 28(3) : 427 -439 . DOI: 10.12066/j.issn.1007-2861.2379

References

[1] 王石, 李玉忱, 刘乃丽, 等. 在属性级别上处理噪声数据的数据清洗算法[J]. 计算机工程, 2005, 31(9): 3.
[2] Kerber R. ChiMerge: discretization of numeric attributes[C]// Proceedings of the 10th National Conference on Artificial Intelligence. 1992: 123-128.
[3] Catlett J. On changing continuous attributes into ordered discrete attributes[M]. New York: Springer-Verlag, 1991.
[4] 傅涛, 孙文静, 孙亚民, 等. 基于分箱统计的 FCM 算法及其在网络入侵检测中的应用[J]. 计算机科学, 2008, 35(4): 4.
[5] Box G, Jenkins G M, Reinsel G C. Time series analysis forecasting and control[J]. Journal of Time, 1976, 31(2): 238-242.
[6] Li D. Support vector regression based image denoising[J]. Image and Vision Computing, 2009, 27(6): 623-627.
[7] López-Rubio E, Florentín-Núñez M N. Kernel regression based feature extraction for 3D MR image denoising[J]. Medical Image Analysis, 2011, 15(4): 498-513.
[8] Huang P, Yang Z J, Wang W B, et al. Denoising low-rank discrimination based least squares regression for image classification[J]. Information Sciences, 2022, 587: 247-264.
[9] Bai X R, Peng X. Radar image series denoising of space targets based on Gaussian process regression[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7): 4659-4669.
[10] 孙添, 何国良. 基于 K-均值聚类和泰森多边形的异常值检测方法[J]. 重庆文理学院学报(社会科学版), 2016, 35(2): 4.
[11] Dong W, Xin L, Lei Z, et al. Sparsity-based image denoising via dictionary learning and structural clustering[C]// 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2011: 457-464.
[12] 胡姣姣. 基于深度学习的飞行器遥测时间序列数据异常检测与预测方法研究[D]. 西安: 西安理工大学, 2019.
[13] Zhang S, Salari E. Image denoising using a neural network based non-linear filter in wavelet domain[C]// IEEE International Conference on Acoustics, Speech, and Signal Processing. 2005: 855947.
[14] Mozhde F P, Mansour V, Mohammad A G. Lung sound signal denoising using discrete wavelet transform and artificial neural network[J]. Biomedical Signal Processing and Control, 2022, 72: 103329.
[15] Yao C, Tang Y B, Sun J, et al. Multiscale residual fusion network for image denoising[J]. IET Image Process, 2022, 16(3): 878-887.
[16] Basar T. A new approach to linear filtering and prediction problems[M]. New Jersey: Wiley-IEEE Press, 2009.
[17] Kalman R E. New results in linear filtering and prediction theory[J]. Journal of Fluids Engineering, 1961, 83(1): 95-108.
[18] Huang J F, Sun I W. Nonanomalous electrodeposition of zinc-iron alloys in an acidic zinc chloride-1-ethyl-3-methylimidazolium chloride ionic liquid[J]. Journal of the Electrochemical Society, 2004, 151(1): C8-C14.
[19] Bajat J B, Miškovivić-Stanković V B, Kačrević-Popović Z. The influence of steel surface modification by electrodeposited Zn-Fe alloys on the protective behaviour of an epoxy coating[J]. Progress in Organic Coatings, 2003, 47(1): 49-54.
[20] Bajat J B, Miskovic-Stankovic V B, Md M, et al. Electrochemical deposition and characterization of Zn-Fe alloys[J]. Journal of the Serbian Chemical Society, 2004, 69(10): 807-815.
[21] 杨仲年. 耐候钢和 Zn 与 Zn-Fe 合金镀层的腐蚀电化学行为研究[D]. 杭州: 浙江大学, 2006.
[22] Shuichi H, Takayuki K, Hideaki M, et al. Taxonomy for protective ability of rust layer using its composition formed on weathering steel bridge[J]. Corrosion Science, 2007, 49(3): 1131-1142.
[23] 梁彩凤, 侯文泰. 合金元素对碳钢和低合金钢在大气中耐腐蚀性的影响[J]. 中国腐蚀与防护学报, 1997, 17(2): 6.
[24] 刘国超, 董俊华, 韩恩厚, 等. 耐候钢锈层研究进展[J]. 腐蚀科学与防护技术, 2006, 18(4): 5.
[25] Townsend H E. Effects of alloying elements on the corrosion of steel in industrial atmospheres[J]. Corrosion, 2001, 57: 497-501.
[26] Ma Y, Ying L, Wang F. The atmospheric corrosion kinetics of low carbon steel in a tropical marine environment[J]. Corrosion Science, 2010, 52(5): 1796-1800.
[27] Hao L, Zhang S, Dong J, et al. Atmospheric corrosion resistance of MnCuP weathering steel in simulated environments[J]. Corrosion Science, 2011, 53(12): 4187-4192.
[28] Lan T, Thoa N, Nishimura R, et al. Atmospheric corrosion of carbon steel under field exposure in the southern part of Vietnam[J]. Corrosion Science, 2006, 48(1): 179-192.
[29] Rajamani M R. Data-based techniques to improve state estimation in model predictive control[D]. Madison: The University of Wisconsin, 2007.
[30] Murali R R, Rawlings J B. Estimation of the disturbance structure from data using semidefinite programming and optimal weighting[J]. Automatica, 2009, 45(1): 142-148.
[31] Bania P, Baranowski J. Field Kalman filter and its approximation[C]// Decision & Control. 2016: 2875-2880.
Options
Outlines

/

〈 〉