Evaluating Simulation-Based Earthquake Forecasts with Examples from UCERF3-ETAS
Abstract
The Collaboratory for the Study of Earthquake Predictability (CSEP) supports an international effort to conduct and rigorously evaluate earthquake forecasting experiments. CSEP has concluded its first phase of testing (CSEP1) with testing centers operating in California, New Zealand, Japan, China, and Europe with 442 models under prospective evaluation. CSEP1 experiments evaluate forecasts expressed as expected rates in small space-time-magnitude bins that can be updated at regular intervals. This experiment design is simple and allows a wide range of models to participate. However, recently developed forecast models, including models for governmental Operational Earthquake Forecasting (OEF), can simulate thousands of synthetic seismicity catalogs (stochastic event sets), which express important space-time dependency structures between triggered earthquakes. The CSEP1 tests do not consider these dependencies. Here, we present new evaluation metrics for forecasts that produce stochastic events. We show evaluations formulated for both one-point and distribution-based statistics, and present the theoretical framework so bespoke tests can be developed based on various other useful statistics. In addition to this traditional statistical hypothesis-testing, we present graphical tools to help effectively communicate the performance of the forecasts with both the modelers and the greater community. For example, in applications of operational aftershock forecasting, understanding the accuracy of the spatial distribution of forecasted seismicity can provide officials with meaningful information to make informed decisions. This information must be presented in an accessible way so that understanding the performance of a forecast does not require intimate familiarity with the underlying statistical methodology used to evaluate it. Both the quantitative and qualitative methods are demonstrated through forecasts generated by the Uniform California Earthquake Rupture Forecast - Version 3 (UCERF3-ETAS) during notable mainshock/aftershock sequences in the California testing region. These methods will be made available to the community through an open-source python package to aid modelers with their model development and design.
- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2019
- Bibcode:
- 2019AGUFM.S13D0464S
- Keywords:
-
- 4306 Multihazards;
- NATURAL HAZARDS;
- 4315 Monitoring;
- forecasting;
- prediction;
- NATURAL HAZARDS;
- 7209 Earthquake dynamics;
- SEISMOLOGY;
- 7223 Earthquake interaction;
- forecasting;
- and prediction;
- SEISMOLOGY