From field to cloud: a collaborative software tool to manage hydrological observatories

Managing data collection, assessment, storage, and analysis in hydrological observatories is challenging: Many processes can only be detected when long-term time series are being analysed, but temporary staff like postgraduates perform the measurements. Naturally the students focus on the data needed for their project and do not particularly care about the long-term availability of the data. Data providing new process insights gets often lost in unmaintainable spreadsheets with no clear distinction between raw, error controlled and derived data. Data warehouse systems, like the one developed by the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) and strict data management guide lines by funding institutions, intend to mediate this effect. However, data warehouse structures are optimized for write once / read often use and require rigorous quality control and metadata description prior to the upload. Our experience shows a risk for data loss at this stage: Data collected at the end of a project is not reviewed and never enters the database and gets lost with the expiring position. As a solution to this kind of problems, we suggest to enter observation early, if possible online, and perform the review process in the system. We are presenting a new collaborative tool for managing hydrological observatories in a standardized and well documented manner directly from the point of data production, the field. Beside the observation data the system stores the observatory management tasks to ensure regular sampling or sensor maintenance. A second benefit of logging management actions together with observations, is the possibility to interpret side effects of sampling or maintenance actions on measurements. In difference to data warehouse systems, the users do data quality control and sensor calibration directly in the online system. The raw data is not changed but augmented by calibration equations and faulty data points are not deleted but marked as an error. Transformation functions convert direct observation to derived data, like discharge, on the fly. Improved stage-discharge relations apply directly to older measurements. The management system consists of a web portal, plotting and mapping facilities, import and export functions, an image database, and a management tool to assign tasks. A transparent link to CUAHSI Hydrological Information System (HIS), a data sharing portal, is currently under development using the standardized WaterML interface. The system is freely available and built upon open source tools. The system is in operational use for three observatories located in Germany, Ecuador and Kenya holding 10 to 50 Million records.