Lessons learned from a use case of Dendra, a sensor observatory curation system, for a complex highly instrumented water quality buoy
Abstract
Dendra is a cloud-based, multi-organizational system, designed to support massive and permanent environmental monitoring efforts. Dendra abstracts QA/QC editing of raw data as metadata, applying them as transformations on views of raw data. Dendra was used in collaboration with the University of New Hampshire (UNH) and the Piscataqua Region Estuaries Partnership (PREP) to make available online 7 years worth of data from a buoy in the Great Bay of New Hampshire. This buoy was equipped with 11 different water quality sensors, including several instruments that were in beta tests. Meteorological data were also collected. A large team of field technicians, lab technicians, data scientists and programmers collaborated to perform an extensive data QA/QC review. Metadata about the instruments, instrument swaps, and data quality were collected and submitted to Dendra via a REST API. Data and data QA/QC annotations are now available via the cloud for analysis of this previously unreleased dataset. Due to the complex and experimental instrumentation, extensive QA/QC annotations were created. The extent and complexity of these annotations illustrated edge cases and limitations of the Dendra system, which led to new designs for QA/QC annotations within Dendra. Google collab hosted Jupyter Notebooks and RapidMiner processes were used to facilitate both detailed records of data provenance and editing as well as rapid visual assessment of trends and relationships among data streams.
- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2021
- Bibcode:
- 2021AGUFMIN22B..02L