Two step clustering for data reduction combining DBSCAN and kmeans clustering
Abstract
A novel combination of two widelyused clustering algorithms is proposed here for the detection and reduction of high data density regions. The Density Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used for the detection of high data density regions and the kmeans algorithm for reduction. The proposed algorithm iterates while successively decrementing the DBSCAN search radius, allowing for an adaptive reduction factor based on the effective data density. The algorithm is demonstrated for a physics simulation application, where a surrogate model for fusion reactor plasma turbulence is generated with neural networks. A training dataset for the surrogate model is created with a quasilinear gyrokinetics code for turbulent transport calculations in fusion plasmas. The training set consists of model inputs derived from a repository of experimental measurements, meaning there is a potential risk of overrepresenting specific regions of this input parameter space. By applying the proposed reduction algorithm to this dataset, this study demonstrates that the training dataset can be reduced by a factor ~20 using the proposed algorithm, without a noticeable loss in the surrogate model accuracy. This reduction provides a novel way of analyzing existing highdimensional datasets for biases and consequently reducing them, which lowers the cost of repopulating that parameter space with higher quality data.
 Publication:

arXiv eprints
 Pub Date:
 November 2021
 arXiv:
 arXiv:2111.12559
 Bibcode:
 2021arXiv211112559K
 Keywords:

 Physics  Computational Physics;
 Computer Science  Databases;
 Physics  Plasma Physics;
 I.0