Big Data Assimilation Incorporating Deep Learning with Phased Array Radar Data and Numerical Weather Prediction

The Japan's Big Data Assimilation (BDA) project started in October 2013 and ended its 5.5-year period in March 2019. The direct follow-on project was accepted and started in April 2019 under the Japan Science and Technology Agency (JST) AIP Acceleration Research, with emphases on the connection with AI technologies, in particular, an integration of DA and AI with high-performance computation (HPC). The BDA project aimed to fully take advantage of "big data" from advanced sensors such as the phased array weather radar (PAWR) and Himawari-8 geostationary satellite, which provide two orders of magnitude more data than the previous sensors. We have achieved successful case studies with newly-developed 30-second-update, 100-m-mesh numerical weather prediction (NWP) system based on RIKEN's SCALE model and local ensemble transform Kalman filter (LETKF) to assimilate PAWR in Osaka and Kobe. We also developed two precipitation nowcasting systems with the every-30-second PAWR data: one with an optical-flow-based system, the other with a deep-learning-based system. We chose the convolutional Long Short Term Memory (Conv-LSTM) as a deep learning algorithm, and found it effective for precipitation nowcasting. The results show that the Conv-LSTM-based nowcasting outperforms the optical-flow-based nowcasting. The Conv-LSTM takes inputs from most recent PAWR data and produces future data as the prediction. For further improvement, the Conv-LSTM is designed to take inputs from future data. Here, we use prediction data from 30-second-update NWP. The Conv-LSTM system taking inputs from both past PAWR data and NWP forecast data can be considered as a step toward the integration of DA and AI with HPC. This presentation will include an overview of the BDA project under the new AIP Acceleration Research scheme, and recent progress on the Conv-LSTM toward a DA-AI-HPC integration.