上海大学学报(自然科学版) ›› 2016, Vol. 22 ›› Issue (1): 28-35.doi: 10.3969/j.issn.1007-2861.2015.04.020

• 大数据 • 上一篇    下一篇

大规模数据集聚类的K邻近均匀抽样数据预处理算法

吉成恒, 雷咏梅   

  1. 上海大学 计算机工程与科学学院, 上海 200444
  • 收稿日期:2015-11-20 出版日期:2016-02-29 发布日期:2016-02-29
  • 通讯作者: 雷咏梅(1965—), 女, 教授, 博士生导师, 博士, 研究方向为高性能计算、大数据处理等.E-mail: lei@shu.edu.cn
  • 基金资助:

    上海市教委重点学科资助项目(12ZZ09); 上海市科委资助项目(13DZ118800)

KNN-based even sampling preprocessing algorithm for big dataset

JI Chengheng, LEI Yongmei   

  1. School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China
  • Received:2015-11-20 Online:2016-02-29 Published:2016-02-29

摘要:

为解决基于密度的聚类算法处理大规模数据集效率低和存储开销大的问题, 提出一种分片的基于K邻近关系的空间均匀抽样算法作为聚类应用的数据预处理过程, 将数据集分片,按密度降序方式去除数据集中部分样本的K邻居, 将剩余样本作为抽样样本, 在保证精度的同时, 可以降低数据规模, 提升计算效率. 实验结果表明, 在数据规模较大且保证聚类结果准确性的前提下, 通过降低聚类数据规模, 可以有效提升聚类效率.

关键词: K邻近, 聚类, 空间均匀抽样, 密度降序

Abstract:

To solve the problem of low efficiency and high storage overheads in densitybased clustering algorithms, an algorithm of even data sampling based on K nearest neighbors (KNN) is proposed as a data preprocessing method of clustering applications. The sampling algorithm slices dataset and gets samples evenly. After slicing a dataset, for part of the samples, the algorithm removes each sample’s K nearest neighbors in a descending order according to the density. The remaining samples are then used as the sample dataset. Experimental results show that, with the increase of data size and the guaranteed accuracy, the sampling algorithm can effectively improve efficiency of clustering by reducing the amount of data needed in clustering.

Key words: K nearest neighbors (KNN), clustering, density descending order, spatial even sampling