Opening the Black Box of the Radiation Belt Machine Learning Model
Abstract
Many Machine Learning (ML) systems, especially deep neural networks, are fundamentally regarded as black boxes since it is difficult to fully grasp how they function once they have been trained. Here, we tackle the issue of the interpretability of a high-accuracy ML model created to model the flux of Earth's radiation belt electrons. The Outer Radiation belt Electron Neural net model (ORIENT) uses only solar wind conditions and geomagnetic indices as inputs. Using the Deep SHAPley additive explanations (DeepSHAP) method, for the first time, we show that the `black box' ORIENT model can be successfully explained. Two significant electron flux enhancement events observed by Van Allen Probes during the storm time of 17-18 March 2013 and non-storm time of 19-20 September 2013 are investigated using the DeepSHAP method.
The DeepSHAP method successfully quantified the feature attribution for the different inputs. For the storm time event, the strong enhancement of solar wind pressure contributed to the rapid dropout seen at higher L-shells, consistent with the magnetopause shadowing effect and outward radial diffusion process. The acceleration of electron flux at higher L-shells was contributed dominantly by clusters of AL peaks while at lower L-shell, the acceleration was mainly contributed by the SYM-H index. The rapid decrease of the SYM-H index (indicative of enhanced convection) also contributed to the dropout of the fluxes at high L-shell during storm time. Different contributions to the fluxes from SYM-H at high and low L-shells were seen to be consistent with the well known 'Dst effect'. Regarding the non-storm time event, the acceleration was found to be clearly correlated to the substorm injection process. These findings, which are highly consistent with current physical understanding, not only demonstrate the reliability of the interpretation method, but also its ability to point to the discovery of potentially missing physical processes, and additionally demonstrate not only the accuracy of our ML model but also most importantly that most of the physical processes are 'baked in' to the ORIENT model. Our study thus provides the framework and encouragement for a new way of modeling and explaining radiation belt dynamics and other similar ML models.- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2022
- Bibcode:
- 2022AGUFMNG45A..06M