Strategic Intelligence and the Preservation of Solar Data

What response is required when a user asks an archive for solar data? Which requests must be satisfied with an immediate download of full-resolution data and which could be served by metadata? This is becoming more of an issue as science data archives continue to grow in volume and complexity. The study of sunspots presents an excellent example of this conundrum. The Sunspot Number V2 (S) is a metadata representation of the many images that were analyzed by numerous researchers over the past 300 years. The daily version of S has some 75,000 points since 01-Jan-1818 and a file size of roughly 3 MB. The annual average data has ∼320 points (since 1700) and a file size of ∼9 kB. Either of those datasets is easier to analyze than if each user had to start from the original images whenever they wanted to analyze the S time series? Of course, the ongoing recalibration of S requires some to re-examine the original data and the steps to create S. Strategic Intelligence faces similar issues and deals with it by assigning a lifetime to data, after which it becomes stale and less useful. Can solar data use a hierarchy of original, high-volume, high-resolution data and along with metadata derived from that data where users are encouraged to use the metadata for their studies? What would that metadata consist of and who would produce it? How would users be guided to the metadata? Some thoughts from SI will provide guidance in these questions.