Impacts of sample design for validation data on the accuracy of feedforward neural network classification

Foody, Giles (2017) Impacts of sample design for validation data on the accuracy of feedforward neural network classification. Applied Sciences, 7 (9). 888/1-888/15. ISSN 2076-3417

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Available under Licence Creative Commons Attribution.
Download (705kB) | Preview

Abstract

Validation data are often used to evaluate the performance of a trained neural network and used in the selection of a network deemed optimal for the task at-hand. Optimality is commonly assessed with a measure, such as overall classification accuracy. The latter is often calculated directly from a confusion matrix showing the counts of cases in the validation set with particular labelling properties. The sample design used to form the validation set can, however, influence the estimated magnitude of the accuracy. Commonly, the validation set is formed with a stratified sample to give balanced classes, but also via random sampling, which reflects class abundance. It is suggested that if the ultimate aim is to accurately classify a dataset in which the classes do vary in abundance, a validation set formed via random, rather than stratified, sampling is preferred. This is illustrated with the classification of simulated and remotely-sensed datasets. With both datasets, statistically significant differences in the accuracy with which the data could be classified arose from the use of validation sets formed via random and stratified sampling (z = 2.7 and 1.9 for the simulated and real datasets respectively, for both p < 0.05%). The accuracy of the classifications that used a stratified sample in validation were smaller, a result of cases of an abundant class being commissioned into a rarer class. Simple means to address the issue are suggested.

Item Type: Article
Keywords: cross-validation; multi-layer perceptron; remote sensing; classification error; sample design; machine learning
Schools/Departments: University of Nottingham, UK > Faculty of Social Sciences > School of Geography
Identification Number: 10.3390/app7090888
Depositing User: Eprints, Support
Date Deposited: 11 Sep 2017 10:00
Last Modified: 13 Oct 2017 07:57
URI: http://eprints.nottingham.ac.uk/id/eprint/45703

Actions (Archive Staff Only)

Edit View Edit View