The Unidata LDM Data Distribution System
Abstract
The Unidata LDM is a near real-time, event-driven system for transmitting frequently-generated data-products, 24/7, from a producer to multiple subscribers using the Internet. Once received, a data-product is processed according to user specifications. A data-product can be anything up to 4 gigabytes in size. Downstream LDM-s register a regular expression based selection predicate with upstream LDM-s. Network topologies include point-to-point, star, and tree. Based on ONC RPC, the LDM system is extremely robust and efficient. Since its initial release in 1994, a network of LDM-s called the Internet Data Distribution (IDD) system has been the primary means by which many if not most Earth Sciences departments in the US obtain and process meteorological data (up to 20 GB/hour and 250k products/hour) with latencies measured in seconds or less. Data-products include numerical model output, radar data, WMO bulletins, and lightning data. Users of the LDM also include the international atmospheric science university community, NOAA, NASA, USGS, the US military, ECMWF, and the meteorological agencies of China, Australia, Brazil, South Korea, and Vietnam. The LDM is the highest volume advanced application on Internet-2 (currently averaging 27 terabytes per week). The LDM history and architecture is presented together with an analysis of its strengths and weaknesses.
- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2010
- Bibcode:
- 2010AGUFMIN31A1274E
- Keywords:
-
- 1914 INFORMATICS / Data mining;
- 1940 INFORMATICS / Machine-to-machine communication;
- 1964 INFORMATICS / Real-time and responsive information delivery;
- 1998 INFORMATICS / Workflow