Watershed process understanding through automated multi source data ingestion and model-data coupling

There is an increasing need for actionable, near real time understanding of watershed processes. Such an understanding requires assimilation, processing and analysis of heterogeneous, multi- domain (hydrological, geochemical, remote sensing) and multi- source (US federal and state agencies, international organizations, private industry, academia and citizen science) datasets.

Multiple tools exist which can be used in developing such process understanding. These include (but are not limited to) information models for earth science observations (ODM2), modeling (PFLOTRAN, MODFLOW, LANDLAB, HSPF, SWAT) and parameter estimation (PEST, DAKOTA, UCODE).

These tools are very powerful and have led to increased understanding of watershed scale processes but a primary challenge facing watershed scientists is that current approaches for collecting data and applying analysis tools on this data are resource intensive and do not scale.

Our team's objective is to develop and apply tools which can provide near real time understanding for many (hundreds to thousands) of sites. This requires that data ingestion and analysis happens essentially autonomous in near real time. For this we have developed a modular vertically integrated cloud-based software application which was designed specifically for process monitoring. This software (PAF - Predictive Assimilation Framework) is a multi tenant web applicationg. It was designed using a modular architecture using modern software practices and leverages multiple open source components.

In the last year we have extended PAF to be able to automatically locate, ingest and harmonize (using a controlled vocabulary for parameters and rule-based time zone mapping) many different data sets from public and private providers. This allows us to use this data as input for automated model building and parameterization (e.g. for PFLOTRAN, MODFLOW and Landlab models) as well as subsequent data analysis flows. All of these can be parameterized, run and visualized through a web interface and can be invoked programmatically through an API from other codes. Such a standardization and automation of data ingestion, analysis and model-data coupling reduces the resources required for site study and enables rapid site investigation and comprehensive cross site comparison.