Why do some probabilistic forecasts lack reliability?
Abstract
In this work, we investigate the reliability of the probabilistic binary forecast. We mathematically prove that a necessary, but not sufficient, condition for achieving a reliable probabilistic forecast is maximizing the Peirce Skill Score (PSS) at the threshold probability of the climatological base rate. The condition is confirmed by using artificially synthesized forecast-outcome pair data and previously published probabilistic solar flare forecast models. The condition gives a partial answer as to why some probabilistic forecast system lack reliability, because the system, which does not satisfy the proved condition, can never be reliable. Therefore, the proved condition is very important for the developers of a probabilistic forecast system. The result implies that those who want to develop a reliable probabilistic forecast system must adjust or train the system so as to maximize PSS near the threshold probability of the climatological base rate.
- Publication:
-
Journal of Space Weather and Space Climate
- Pub Date:
- April 2019
- DOI:
- 10.1051/swsc/2019016
- arXiv:
- arXiv:1904.08791
- Bibcode:
- 2019JSWSC...9A..17K
- Keywords:
-
- probabilistic forecast;
- reliability;
- necessary condition;
- Peirce Skill Score;
- forecast model;
- Physics - Space Physics;
- Astrophysics - Earth and Planetary Astrophysics;
- Astrophysics - Solar and Stellar Astrophysics;
- Physics - Atmospheric and Oceanic Physics;
- Physics - Data Analysis;
- Statistics and Probability
- E-Print:
- 12 pages, 6 figures, 1 table, accepted for publication in the Journal of Space Weather and Space Climate (JSWSC)