Reproducible earthquake forecasting experiments with pyCSEP

William H. Savran, José A. Bayona, Maximilian J. Werner, & Philip J. Maechling

Published August 16, 2021, SCEC Contribution #11535, 2021 SCEC Annual Meeting Poster #263

An important goal of the Collaboratory for the Study of Earthquake Predictability (CSEP) is to
facilitate transparent and reproducible earthquake forecasting experiments. To reproduce an experiment, we must obtain consistent results, at least statistically, when the experiment is re-run using the same data and computational methods. For over a decade, earthquake forecasting experiments were computed within CSEP testing centers, which are dedicated servers and software to compute and evaluate probabilistic earthquake forecasting models against observations. Recently, CSEP has developed pyCSEP, an open-source Python library, to provide forecast evaluation software used in the CSEP testing centers to the research community. pyCSEP enables users to conduct forecasting experiments on their own machines. This flexibility, however, also makes it difficult to maintain a strictly controlled computing environment for these bespoke experiments. Previously, CSEP testing centers had maintained such a controlled environment to ensure the reproducibility and unbiased nature of CSEP experiments. Thus, user-run experiments do not replace testing centers, but rather supplement them. In this work, we show how transparent and reproducible forecasting experiments can be conducted using freely available tools such as git, Zenodo, and Docker. We demonstrate this approach by reproducing results from the Regional Earthquake Likelihood Model (RELM) forecasting experiment published by Zechar et al. (2013). We reproduce summary statistics of the forecast and CSEP consistency tests for forecasts in the mainshock+aftershock class, as defined by Zechar et al. (2013). By substituting transparency for a controlled environment, we can uphold the reproducibility requirements for CSEP experiments. This new approach has advantages, namely that all research artifacts and code are made publicly available rather than stored on managed CSEP servers. In addition, external entities manage and store the data and meta-data. The focus on reproducibility and data management in this approach should be standard practice for future CSEP experiments, and should guide future CSEP infrastructure developments.

Key Words
earthquake forecasting, csep, oef, reproducibility

Citation
Savran, W. H., Bayona, J. A., Werner, M. J., & Maechling, P. J. (2021, 08). Reproducible earthquake forecasting experiments with pyCSEP. Poster Presentation at 2021 SCEC Annual Meeting.

Related Projects & Working Groups
Earthquake Forecasting and Predictability (EFP)