面向大数据应用挑战的超级计算机设计

doi:10.3969/j.issn.1007-2861.2015.03.014

Abstract

Abstract:

Because traditional supercomputer is designed for high-performance computing, big data processing applications brings some software and hardware challenges including compute, storage, communication and programming. This paper introduces optimization methods of Tianhe-2 supercomputer system to process big data, such as a new heterogeneous polymorphic architecture, custom high-speed TH-Express 2+ interconnection network, hybrid hierarchical storage system and hybrid computing pattern framework.These efforts maybe make help for how to design supercomputers in the age of big data.

Key words: big data, supercomputer, Tianhe-2

LIAO Xiangke, TAN Yusong, LU Yutong, XIE Min, ZHOU Enqiang, HUANG Jie. On the challenge for supercomputer design in the big data era[J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 3-16.

References

[1] Big data and big data analytics: significance for the hardcopy industry [EB/OL]. [2015-10-19]. http://www.idc.com/getdoc.jsp?containerId=246831.
[2] IDC’s worldwide big data taxonomy [EB/OL]. [2015-10-19]. http://www.idc.com/getdoc.jsp?containerId=254052.
[3] Graph500 [EB/OL]. [2015-10-19]. http://www.graph500.org.
[4] Jing N F, Shen Y, Lu Y, et al. An energy-efficient and scalable eDRAM-based register file architecture for GPGPU [C]//Processing of International Symposiumon Computer Architecture. 2013: 344-355.
[5] Lam C H. Storage class memory [C]//Processing of 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT). 2010: 1080-1083.
[6] Dhiman G, Ayoub R, Rosing T. PDRAM: a hybrid PRAM and DRAM main memory system [C]// Processing of Design Automation Conference. 2009: 664-669.
[7] Ham T J, Chelepalli B K, Neng X, et al. Disintegrated control for energy-efficient and heterogeneous memory systems [C]// Processing of IEEE International Symposium on High Performance Computer Architecture. 2013: 424-435.
[8] Ghemawat S, Gobioff H, Leung S T. The google file system [C]// Processing of ACM SIGOPS Operating Systems Review. 2003: 29-43.
[9] HDFS [EB/OL]. [2015-10-19]. http://hadoop.apache.org/hdfs/docs/current/hdfs design.html.
[10] ADFS [EB/OL]. [2015-10-19]. http://github.com/taobao/ADFS.

[11] Zhou W, Pierre G, Chi C H. Scalable transactions for web applications in the cloud [J]. Lecture Notes in Computer Science, 2009, 5(4): 525-539.
[12] Decandia G, Hastorun D, Jampani M, et al. Dynamo: amazon’s highly available key-value store [J]. Proc Sosp, 2007, 41(6): 205-220.
[13] Cassandra [EB/OL]. [2015-10-19]. http://cassandra.apache.rog.
[14] Chen C, Hsiao M. Bigtable: a distributed storage system for structured data [C]//Proceedings of OSDI. 2006: 205-218.
[15] Peng D, Dabek F. Large-scale incremental processing using distributed transactions and notifications [C]//Usenix Symposium on Operating Systems Design and Implementation. 2010: 4-6.
[16] Cooper B F. Spanner: Google’s globally-distributed database [C]// Proceedings of the 6th International Systems and Storage Conference. 2013: 1-10.
[17] Hbase [EB/OL]. [2015-10-19]. http://hbase.apache.org/.
[18] OceanBase [EB/OL]. [2015-10-19]. http://alibaba.github.io/oceanbase.
[19] Memcached [EB/OL]. [2015-10-19]. http ://www.memcached.org/.
[20] Redis [EB/OL]. 2013-03-25 [2015-10-19]. http://www.redis.io/.
[21] Ousterhout J, Agrawal P, Erickson D, et al. The case for RAMCloud [J]. Communications of the Acm, 2011, 54(7): 121-130.
[22] He B S, Fang W B, Luo Q, et al. Mars: a MapReduce framework on graphics processors [C]// Processing of 17th International Conference on Parallel Architectures and Compilation Techniques (PACT). 2008: 260-269.
[23] Hong C T, Chen D H, Chen W G, et al. MapCG: writing parallel program portable between CPU and GPU [C]// PACT. 2010: 217-226.
[24] Basaran C, Kang K D. Grex: an e cient Map/Reduce framework for graphics processing units [J]. Journal of Parallel and Distributed Computing, 2013, 73(4): 522-533.
[25] Farivar R, Verma A, Chan E M, et al. MITHRA: multiple data independent tasks on a heterogeneous resource architecture [C]// Processing of IEEE Conference on Cluster Computing. 2009: 1-10.
[26] Stuart J A, Owens J D. Multi-GPU MapReduce on GPU clusters [C]// IPDPS. 2011: 1068-1079.
[27] Lu M, Zhang L, Huynh H P, et al. Optimizing the Map/Reduce framework on intel xeon phi coprocessor [C]//2013 IEEE International Conference on Big Data. 2013: 125-130.
[28] Lu M, Liang Y, Huynh H, et al. Mrphi: an optimized Map/Reduce framework on intel xeon phi coprocessors [J]. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(11): 3066-3078.
[29] Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing [C]// Proceedings of the 9th USENIX Conference
on Networked Systems Design and Implementation. 2012: 141-146.
[30] Yoo R M, Romano A, Kozyrakis C. Phoenix rebirth: scalable Map/Reduce on a largescale shared-memory system [C]//Proceedings of IEEE International Symposium on Workload
Characterization. 2009: 198-207.

[31] Gates A F, Natkovich O, Chopra S, et al. Building a high-level dataflow system on top of Map-Reduce: the Pig experience [C]// Proceedings of the VLDB Endowment 2.2. 2009: 1414-1425.
[32] Thusoo A, Sarma J S, Jain N, et al. Hive: a warehousing solution over a map-reduce framework [C]// Proceedings of the VLDB Endowment 2.2. 2009: 1626-1629.
[33] Taz [EB/OL]. [2015-10-19]. http://hortonworks.com/blog/introducing-tez-faster-hadoopprocessing/.
[34] Pang Z B, Xie M, Zhang J, et al. The TH Express high performance interconnect networks [J]. Frontiers of Computer Science, 2014, 8(3): 357-366.
[35] Liao X K, Pang Z B, Wang K F, et al. High performance interconnect network for Tianhe system [J]. Journal of Computer Science and Technology, 2015, 30(2): 259-272.
[36] Huang W, Gao Q, Liu J, et al. High performance virtual machine migration with RDMA over modern interconnects [C]// 2007 IEEE International Conference on Cluster Computing. 2007: 11-20.
[37] Gavrilovska A, Kumar S, Sundaragopalan S, et al. Platform overlays: enabling in-network stream processing in large-scale distributed applications [C]// Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video. 2005: 171-176.
[38] Regnier G, Makineni S, Illikkal R, et al. TCP onloading for data center servers [J]. Computer, 2004(11): 48-58.
[39] Zhang Q, Cheng L, Boutaba R. Cloud computing: state-of-the-art and research challenges [J]. Journal of Internet Services and Applications, 2010, 1(1): 7-18.
[40] Wang W Z, Wu Q B, Tan Y S, et al. Optimizing the MapReduce framework for CPU-MIC heterogeneous cluster [C]//2015 International Conference on Advanced Parallel Processing Technology. 2015: 33-44.

On the challenge for supercomputer design in the big data era

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

References

Related Articles 8

Recommended Articles

Metrics

Comments

[1]	GUO Yike1,2, YANG Xian2. Precision medicine and big data [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 17-27.
[2]	GUO Yike. Let's go big data [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 1-2.
[3]	QI Lianyong1,2, DOU Wanchun1, ZHOU Yuming1. A context-aware weighting approach for big data of quality ratings in E-commerce [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 36-44.
[4]	LU Dongfang, XU Junfu, XIANG Chaojuan, XIE Jiang. Survey of clustering methods for big data in biology [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 45-57.
[5]	GU Qi1,2, ZHU Can1, CAO Jian1. Product matching based on Internet and its implementation [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 58-68.
[6]	YAN Zhiguo, XU Zheng, MEI Lin, HU Chuanping. Big data analysis of next generation video surveillance system for public security [J]. Journal of Shanghai University（Natural Science Edition）, 2016, 22(1): 81-87.
[7]	SONG Yu-fei, ZHANG Jun-jie, WANG Kai, LI Jia-qi, XUE Zi-wei, ZHENG Yue. Design and implementation of FPGA 10 Gbit/s reliable interconnect communication based on IP protocol [J]. Journal of Shanghai University（Natural Science Edition）, 2015, 21(5): 570-578.
[8]	ZHANG Qian1,2, JIN Yi1,2, SONG Kai1,2,3, GAO Huan1,2. MPI Programming Based on Ternary Optical Computer in Supercomputer [J]. Journal of Shanghai University（Natural Science Edition）, 2014, 20(2): 180-189.