On the challenge for supercomputer design in the big data era

Expand
  • School of Computer, National University of Defense Technology, Changsha 410073, China

Received date: 2015-12-23

  Online published: 2016-02-29

Abstract

Because traditional supercomputer is designed for high-performance computing, big data processing applications brings some software and hardware challenges including compute, storage, communication and programming. This paper introduces optimization methods of Tianhe-2 supercomputer system to process big data, such as a new heterogeneous polymorphic architecture, custom high-speed TH-Express 2+ interconnection network, hybrid hierarchical storage system and hybrid computing pattern framework.These efforts maybe make help for how to design supercomputers in the age of big data.

Cite this article

LIAO Xiangke, TAN Yusong, LU Yutong, XIE Min, ZHOU Enqiang, HUANG Jie . On the challenge for supercomputer design in the big data era[J]. Journal of Shanghai University, 2016 , 22(1) : 3 -16 . DOI: 10.3969/j.issn.1007-2861.2015.03.014

References

[1] Big data and big data analytics: significance for the hardcopy industry [EB/OL]. [2015-10-19]. http://www.idc.com/getdoc.jsp?containerId=246831.
[2] IDC’s worldwide big data taxonomy [EB/OL]. [2015-10-19]. http://www.idc.com/getdoc.jsp?containerId=254052.
[3] Graph500 [EB/OL]. [2015-10-19]. http://www.graph500.org.
[4] Jing N F, Shen Y, Lu Y, et al. An energy-efficient and scalable eDRAM-based register file architecture for GPGPU [C]//Processing of International Symposiumon Computer Architecture. 2013: 344-355.
[5] Lam C H. Storage class memory [C]//Processing of 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT). 2010: 1080-1083.
[6] Dhiman G, Ayoub R, Rosing T. PDRAM: a hybrid PRAM and DRAM main memory system [C]// Processing of Design Automation Conference. 2009: 664-669.
[7] Ham T J, Chelepalli B K, Neng X, et al. Disintegrated control for energy-efficient and heterogeneous memory systems [C]// Processing of IEEE International Symposium on High Performance Computer Architecture. 2013: 424-435.
[8] Ghemawat S, Gobioff H, Leung S T. The google file system [C]// Processing of ACM SIGOPS Operating Systems Review. 2003: 29-43.
[9] HDFS [EB/OL]. [2015-10-19]. http://hadoop.apache.org/hdfs/docs/current/hdfs design.html.
[10] ADFS [EB/OL]. [2015-10-19]. http://github.com/taobao/ADFS.

[11] Zhou W, Pierre G, Chi C H. Scalable transactions for web applications in the cloud [J]. Lecture Notes in Computer Science, 2009, 5(4): 525-539.
[12] Decandia G, Hastorun D, Jampani M, et al. Dynamo: amazon’s highly available key-value store [J]. Proc Sosp, 2007, 41(6): 205-220.
[13] Cassandra [EB/OL]. [2015-10-19]. http://cassandra.apache.rog.
[14] Chen C, Hsiao M. Bigtable: a distributed storage system for structured data [C]//Proceedings of OSDI. 2006: 205-218.
[15] Peng D, Dabek F. Large-scale incremental processing using distributed transactions and notifications [C]//Usenix Symposium on Operating Systems Design and Implementation. 2010: 4-6.
[16] Cooper B F. Spanner: Google’s globally-distributed database [C]// Proceedings of the 6th International Systems and Storage Conference. 2013: 1-10.
[17] Hbase [EB/OL]. [2015-10-19]. http://hbase.apache.org/.
[18] OceanBase [EB/OL]. [2015-10-19]. http://alibaba.github.io/oceanbase.
[19] Memcached [EB/OL]. [2015-10-19]. http ://www.memcached.org/.
[20] Redis [EB/OL]. 2013-03-25 [2015-10-19]. http://www.redis.io/.
[21] Ousterhout J, Agrawal P, Erickson D, et al. The case for RAMCloud [J]. Communications of the Acm, 2011, 54(7): 121-130.
[22] He B S, Fang W B, Luo Q, et al. Mars: a MapReduce framework on graphics processors [C]// Processing of 17th International Conference on Parallel Architectures and Compilation Techniques (PACT). 2008: 260-269.
[23] Hong C T, Chen D H, Chen W G, et al. MapCG: writing parallel program portable between CPU and GPU [C]// PACT. 2010: 217-226.
[24] Basaran C, Kang K D. Grex: an e cient Map/Reduce framework for graphics processing units [J]. Journal of Parallel and Distributed Computing, 2013, 73(4): 522-533.
[25] Farivar R, Verma A, Chan E M, et al. MITHRA: multiple data independent tasks on a heterogeneous resource architecture [C]// Processing of IEEE Conference on Cluster Computing. 2009: 1-10.
[26] Stuart J A, Owens J D. Multi-GPU MapReduce on GPU clusters [C]// IPDPS. 2011: 1068-1079.
[27] Lu M, Zhang L, Huynh H P, et al. Optimizing the Map/Reduce framework on intel xeon phi coprocessor [C]//2013 IEEE International Conference on Big Data. 2013: 125-130.
[28] Lu M, Liang Y, Huynh H, et al. Mrphi: an optimized Map/Reduce framework on intel xeon phi coprocessors [J]. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(11): 3066-3078.
[29] Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing [C]// Proceedings of the 9th USENIX Conference
on Networked Systems Design and Implementation. 2012: 141-146.
[30] Yoo R M, Romano A, Kozyrakis C. Phoenix rebirth: scalable Map/Reduce on a largescale shared-memory system [C]//Proceedings of IEEE International Symposium on Workload
Characterization. 2009: 198-207.

[31] Gates A F, Natkovich O, Chopra S, et al. Building a high-level dataflow system on top of Map-Reduce: the Pig experience [C]// Proceedings of the VLDB Endowment 2.2. 2009: 1414-1425.
[32] Thusoo A, Sarma J S, Jain N, et al. Hive: a warehousing solution over a map-reduce framework [C]// Proceedings of the VLDB Endowment 2.2. 2009: 1626-1629.
[33] Taz [EB/OL]. [2015-10-19]. http://hortonworks.com/blog/introducing-tez-faster-hadoopprocessing/.
[34] Pang Z B, Xie M, Zhang J, et al. The TH Express high performance interconnect networks [J]. Frontiers of Computer Science, 2014, 8(3): 357-366.
[35] Liao X K, Pang Z B, Wang K F, et al. High performance interconnect network for Tianhe system [J]. Journal of Computer Science and Technology, 2015, 30(2): 259-272.
[36] Huang W, Gao Q, Liu J, et al. High performance virtual machine migration with RDMA over modern interconnects [C]// 2007 IEEE International Conference on Cluster Computing. 2007: 11-20.
[37] Gavrilovska A, Kumar S, Sundaragopalan S, et al. Platform overlays: enabling in-network stream processing in large-scale distributed applications [C]// Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video. 2005: 171-176.
[38] Regnier G, Makineni S, Illikkal R, et al. TCP onloading for data center servers [J]. Computer, 2004(11): 48-58.
[39] Zhang Q, Cheng L, Boutaba R. Cloud computing: state-of-the-art and research challenges [J]. Journal of Internet Services and Applications, 2010, 1(1): 7-18.
[40] Wang W Z, Wu Q B, Tan Y S, et al. Optimizing the MapReduce framework for CPU-MIC heterogeneous cluster [C]//2015 International Conference on Advanced Parallel Processing Technology. 2015: 33-44.

Outlines

/