kNN-IS: an iterative spark-based design of the k-nearest neighbors classifier for big data

Tools

Maillo, Jesus, Ramirez, Sergio, Triguero, Isaac and Herrera, Francisco (2016) kNN-IS: an iterative spark-based design of the k-nearest neighbors classifier for big data. Knowledge-Based Systems . ISSN 1872-7409 (In Press)

Full text not available from this repository.

Abstract

The k-Nearest Neighbors classifier is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies.

In this work we provide a new solution to perform an exact k-nearest neighbor classification based on Spark. We take advantage of its in-memory operations to classify big amounts of unseen cases against a big training dataset. The map phase computes the k-nearest neighbors in different training data splits. Afterwards, multiple reducers process the definitive neighbors from the list obtained in the map phase. The key point of this proposal lies on the management of the test set, keeping it in memory when possible. Otherwise, it is split into a minimum number of pieces, applying a MapReduce per chunk, using the caching skills of Spark to reuse the previously partitioned training set. In our experiments we study the differences between Hadoop and Spark implementations with datasets up to 11 million instances, showing the scaling-up capabilities of the proposed approach. As a result of this work an open-source Spark package is available.

Item Type: Article
RIS ID: https://nottingham-repository.worktribe.com/output/795290
Keywords: K-nearest neighbors; Big data; Apache Hadoop; Apache Spark; MapReduce
Schools/Departments: University of Nottingham, UK > Faculty of Science > School of Computer Science
Identification Number: https://doi.org/10.1016/j.knosys.2016.06.012
Depositing User: Triguero, Isaac
Date Deposited: 15 Jun 2016 08:26
Last Modified: 04 May 2020 17:56
URI: https://eprints.nottingham.ac.uk/id/eprint/34013

Actions (Archive Staff Only)

Edit View Edit View