CASM: A Long-term Consistent Artificial Intelligence-based Soil Moisture Dataset

Consistent AI-based Soil Moisture (CASM) dataset is a global soil moisture (SM) dataset created using machine learning, based on the NASA Soil Moisture Active Passive (SMAP) SM data, and is aimed at extrapolating SMAP-like quality SM data back in time. Patching various remote sensing and in-situ SM measurements can introduce major biases in the mean or variability of the data, and statistical merging methods cannot retain the data quality and amplify uncertainty. Utilizing a neural network (NN), the new CASM dataset was created using high-quality SMAP SM as a target and brightness temperatures from SMOS or AMSRE/2 as input. CASM represents SM in the top soil layer, defined on a global 25 km EASE-2 grid and covering 2002-2020 with a 3-day temporal resolution. The resulting dataset exhibits excellent spatial and temporal homogeneity, without compromising interannual variability, and agreement with the SMAP data (with a mean correlation of 0.97 between the SMAP SM and CASM SM for the period when the two overlap). Moreover, the input and target signals were divided into a seasonal cycle and residuals, with the NN trained on the residuals. This approach ensures that the high performance doesn't mask a simple seasonal cycle matching but rather exemplifies the skill in predicting extremes; with the NN achieving a correlation of 0.75 on the test data for the residuals. A comparison to 367 global in-situ SM monitoring sites shows a SMAP-like median correlation of 0.66. Additionally, the SM product uncertainty was assessed, and both aleatoric and epistemic uncertainties are included in the dataset. Mean epistemic uncertainty, related to the NN model structure, is ranging from 0.007 m3/m3 to 0.014 m3/m3 (the difference is due to multi-step NN training necessary under the input data availability) and on average is close to a desired SM product stability threshold of 0.01 m3/m3 per year. Aleatoric uncertainty, defined as input noise propagated through the system, depends on the introduced level of noise. With 10% noise applied to the residuals, the resulting mean standard deviation of the model outputs rises from 0.005 to 0.007 m3/m3. Further studies will focus on the assessment of CASM consistency and gaining new insights into the global interannual SM dynamic.