Big Data Challenges in a Data Center Workflow

The Mission of NOAA's National Geophysical Data Center (NGDC) is to provide long-term scientific data stewardship for the Nation's geophysical data, ensuring quality, integrity, and accessibility.NGDC provides stewardship, products, and services for geophysical data from our Sun to Earth and Earth's sea floor and solid earth environment, including Earth observations from space. As part of its mission NGDC executes preservation workflows which include, ingest, quality control, metadata generation, product generation and development of access methods for diverse data types. Each of the phases of proper stewardship involves challenges when it comes to Big Data. This presentation will look at Big Data as it interacts and is supported by the stewardship workflow of a National Data Center. We will present tools and techniques as well as identify remaining challenges as we continue to march into the big data era.