What good is a STONE curve: Feature set analysis of this new metric

We apply idealized scatter-plot distributions to The Sliding Threshold of Observation for Numeric Evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point-spread distribution. The STONE curve is based on the relative operating characteristic (ROC) curve but is developed to work with a continuous-valued set of observations. This is particularly useful for space weather research, which is often interested in model predictions of time series data. While the ROC curve defines events in the observational data set and then sweeps the event identification threshold for only the model data set, the STONE curve utilizes the continuous nature of the observational set and slides the threshold simultaneously for both the observational and model data sets. The identical sweep of both the model and observational thresholds results in changes to both the modeled and observed event states as the quadrant boundaries shift. The changes in a data-model pair's event states result in noisy and nonmonotonic features to appear in the STONE curve when compared to a ROC curve for the same observational and model data sets. Such features can be indicative of characteristics in the underlying distributions of the data and model values. Many combinations of idealized observational and model datasets were created with different known distribution functions for the data-model pairs, revealing that certain scatter-plot features result in distinct STONE curve signatures. A comprehensive suite of feature-signature combinations is presented, including their relationship to many other metrics. In addition, the similarities and differences between STONE and ROC curve features for the same data-model number set are discussed. The results of this study allow for easier interpretation of the STONE curve in Earth and space science data-model comparisons, especially space weather applications.