针对当前古汉语机器翻译任务中平行语料较少、因词义多变产生的模型翻译用词精确度低的问题,提出了一种基于Transformer模型的预训练特征中心算法模型.首先,训练语言模型和提取预训练特征中心,以提高在平行语料不足时的泛化能力;其次,Transformer模型在特征中心空间生成查询点,依据查询点确定生成在特征中心中的距离,以提高模型对一词多义的理解能力.实验结果表明,该方法在翻译准确率上总体高于基准模型,在西汉时期的数据集中BLEU 1~4分数分别高出Transformer模型6.1、4.9、3.7和2.8,在五代十国数据集中BLEU 1~4分数分别高出Transformer模型4.2、4.2、3.3和2.6,在唐朝数据集中BLEU 1~4分数表现最好.与此同时,相较于Transformer模型,该方法参数量下降了5.73 M,充分证明该算法模型的优势.
To solve the low accuracy of model translation words owing to the lack of parallel corpora and variable meanings in the current ancient Chinese machine translation model, a Transformer-based pre-trained feature center algorithm model is proposed. This method first trains the language model to extract pretrained feature centers to improve generalization ability when parallel corpora are insufficient. Second, the Transformer generates queries in the feature center space and determines the generation based on the distance between the queries in the feature center to improve the model’s understanding of polysemy. The experimental results indicate that the generation accuracy of the proposed method is generally higher than that of the benchmark model. In the XiHan Datasets, BLEU 1~4 scores were 6.1, 4.9, 3.7, and 2.8 higher than those of the Transformers. In the WuDaiShiGuo Datasets, the BLEU 1~4 scores were 4.2, 4.2, 3.3, and 2.6 higher than those of the Transformers. In the Tang Datasets, the BLEU 1~4 scores were the best. Simultaneously, compared with the Transformer, the parameter count of the proposed method decreased by 5.73 M, demonstrating its advantages.
[1] Kalchbrenner N, Blunsom P. Recurrent continuous translation models [C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013: 1700-1709.
[2] Cho K, van Merrienboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1724-1734.
[3] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks [J]. Advances in Neural Information Processing Systems, 2014, 27: 3104-3122.
[4] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.
[5] Mnih V, Heess N, Graves A. Recurrent models of visual attention [J]. Advances in Neural Information Processing Systems, 2014, 27: 2204-2212.
[6] Larochelle H, Erhan D, Bengio Y. Zero-data learning of new tasks [C]// AAAI. 2008: 646-651.
[7] Palatucci M, Pomerleau D, Hinton G E, et al. Zero-shot learning with semantic output codes [J]. Advances in Neural Information Processing Systems, 2009, 22: 1410-1418.
[8] Kenton J D M W C, Toutanova L K. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of NAACL-HLT. 2019: 4171-4186.
[9] Brown T, Mann B, Ryder N, et al. Language models are few-shot learners [J]. Advances in Neural Information Processing Systems, 2020, 33: 1877-1901.
[10] Rajpurkar P, Zhang J, Lopyrev K, et al. SQuAD: 100000+ questions for machine comprehension of text [C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016: 2383-2392.
[11] Zhang X, Lapata M. Chinese poetry generation with recurrent neural networks [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 670-680.
[12] Wang D Z, He W, Wu H, et al. Chinese poetry generation with planning based neural network [C]// The 26th International Conference on Computational Linguistics: Technical Papers. 2016: 1051-1060.
[13] Yi X Y, Sun M S, Li R Y, et al. Chinese poetry generation with a working memory model [C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018: 4553- 4559.
[14] Ghazvininejad M, Shi X, Choi Y, et al. Generating topical poetry [C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016: 1183-1191.
[15] 韩芳, 杨天心, 宋继华. 基于句本位句法体系的古汉语机器翻译研究[J]. 中文信息学报, 2015, 29(2): 103-110; 117.
[16] Yan R. I. Poet: automatic poetry composition through recurrent neural networks with iterative polishing schema [C]// IJCAI 2016. 2016: 2238-2244.
[17] Yang Z C, Cai P S, Feng Y S, et al. Generating classical Chinese poems from vernacular Chinese [C]// EMNLP-IJCNLP 2019. 2019: 6155-6164.
[18] Jin P, Wang H L, Ma L M, et al. Translating classical Chinese poetry into modern Chinese with transformer [C]// 22nd Chinese Lexical Semantics Workshop (CLSW). 2022: 474-480.
[19] Holtzman A, Buys J, Du L, et al. The curious case of neural text degeneration [C]// ICLR. 2020: 1-14.
[20] Papineni K, Roukos S, Ward T, et al. BLEU: a method for automatic evaluation of machine translation [C]// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002: 311-318.
[21] Ott M, Edunov S, Baevski A, et al. Fairseq: a fast, extensible toolkit for sequence modeling [C]// Proceedings of the 2019 Conference of the North. 2019: 48-53.
[22] Ramachandran P, Liu P J, Le Q. Unsupervised pretraining for sequence to sequence learning [C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 383-391.
