The Data Transport Network: A Usenet-Based Approach For Data Retrieval From Remote Field Sites

The Data Transport Network coordinates the collection of scientific data, instrument telemetry and post-processing for the delivery of real-time results over the Internet from instruments located at remote field sites with limited or unreliable network connections. The system was originally developed in 1999 for the distribution of large data sets collected by the radar, lidars and imagers at the NSF upper atmosphere research facility in Sondrestrom, Greenland. The system helped to mitigate disruptions in network connectivity and optimized transfers over the site's low-bandwidth satellite link. The core idea behind the system is to transfer data files as attachments in Usenet messages. The messages collected by a local news server are periodically transmitted to other servers on the Internet when link conditions permit. If the network goes down, data files continue to be stored locally and the server will periodically attempt to deliver the files for upwards of two weeks. Using this simple approach, the Data Transport Network is able to handle a large number of independent data streams from multiple instruments. Each data stream is posted into a separate news group. There are no limitations to the types of data files that can be sent and the system uses standard Internet protocols for encoding, accessing and transmitting files. A common framework allows for new data collection or processing programs to be easily integrated. The two-way nature of the communications also allows for data to be delivered to the site as well, a feature used for the remote control of instruments. In recent years, the Data Transport Network has been applied to small, low-power embedded systems. Coupled with satellite-based communications systems such as Iridium, these miniature Data Transport servers have found application in a number of remote instrument deployments in the Arctic. SRI's involvement as a team member in Veco Polar Resources, the NSF Office of Polar Programs Arctic Logistics contractor, has provided us the opportunity to transition and apply the system to other NSF-funded projects, including an autonomous instrument platform in Ivotuk, Alaska, and an Iridium-based MACCS magnetometer array in Canada. The Data Transport Network has also been scaled to serve as the telemetry backbone for the NSF Advanced Modular Incoherent Scatter Radar (AMISR). This project has tremendous data handling requirements. Status information must be collected from over 12,000 antenna elements and 400 control computers located at multiple locations around the globe. Instrumentation displays, system monitoring, and data collection are all handled by an extensive Data Transport application. In addition, the Data Transport Network supplies the necessary tools to support remote operations, a fundamental aspect of the AMISR. For more information, including access to the open-source software, please visit http://transport.sri.com.