Systematic Evaluation of Computational Topology as an Effective Methodology for Improving Solar Eruption Prediction

We propose a systematic approach to evaluate the predictive power of engineered features from the SDO HMI dataset in the context of a multi-layer perceptron approach for the solar eruption prediction problem. In our previous work (Deshmukh et. al, 2020), we proposed a novel featurization technique on the SDO HMI magnetogram image data using Topological Data Analysis (TDA) for solar eruption prediction. We further demonstrated that these novel topology-based features show an improvement in the True Skill Statistic (TSS) prediction score over the SHARPs physics-based features using a fixed multi-layer perceptron (MLP) model. In this work, we further methodically validate these conclusions using a model hyperparameter tuning approach. For a given combination of training and test sets and for each engineered feature set, we determine an optimal set of model hyperparameters that maximizes the average TSS score using a k-fold cross validation method. We then evaluate the model on the test set with the optimal hyperparameters using a bootstrap testing method. While ensuring fairness across the feature sets with hyperparameter tuning, we confirm our previous conclusions in a statistically significant manner on 10 different training/test set combinations. Finally, we compare our results with the recent publication by Leka et al. (2019) using a dataset and evaluation strategies very similar to theirs to establish a standard for systematically evaluating solar eruption forecasting systems. The results of this work will be used for developing a comprehensive deep learning eruption forecasting model that leverages features extracted from the raw SDO image data by convolutional neural network models in combination with the engineered features presented here.