A comprehensive data qa/qc strategy for data from autonomous point sensors: design, implementation and examples

There is an exponential increase in the use of autonomous point sensors (sensors which collect and transmit data without any human intervention) across all of geosciences. Point sensors include both physical and chemical sensors. Such data is typically stored in relational databases, from which data is subsequently polled for a range of different purposes. One of the fundamental challenges for end users is to assess the confidence in specific measurements. Historically (when sensor owners, sensor installers, data managers and data users were associated with one research group or institution) there might have been an intuitive (if poorly quantifiable) feel for such confidence. However, in the current environment these roles are often filled by people who are geographically separated and in different organizations. In addition, while historically such data was subject to semi manual review, this is becoming less and less practical. Finally, there is more and more a desire to use data in near real time. Consequently, challenges exist on how to automate all aspects of data qa/qc and validation for autonomous sensors. Data validation can result either in a confidence range and/or a Boolean indicator (good/bad data). We have developed and implemented a comprehensive, multi level data validation strategy. This strategy progresses from analysis of the most recent data received from a sensor (typically one to hundreds of measurements) to an analysis of the recent data in the context of the historic data received from the sensor, to an analysis of data received by other sensors (which compares trends and patterns), to a simple model based analysis. The outcome of this analysis (which is performed as soon as new data arrives) results in a quality indicator which is made available to the user with the data. In this talk I will provide examples of this approach for a number of currently operating monitoring networks as well as a discussion on how to easily implement this strategy for existing point sensor networks.