The Earth Observation Depot Network (EODN)
A distributed cloud, open-content distribution network for remote sensing data
Download the EODN Flyer Here
The Earth Observation Depot Network (EODN) is collaboration among the United States Geological Survey (USGS), AmericaView and the DLT development team. EODN is designed to expand use of large remote sensing datasets by timely distribution and removing access barriers inherent in current distribution schemes. EODN is focused on three prevailing issues plaguing the remote sensing community: availability latency – the delay between acquisition and availability of data, download speed – exacerbated in some cases by rate-limiting policies of large satellite data providers, and connectivity, particularly in areas of sub-standard or damaged infrastructure.
The USGS has complete responsibility for acquiring, archiving and distributing much of the nations publicly-owned satellite imagery (e.g. Landsat data.) As such, USGS is interested in EODN as a mechanism to reduce the load on USGS servers and networks.
AmericaView is a USGS sponsored organization of state-based, university-led consortia dedicated to the promotion of remote sensing education, research and outreach. AmericaView first became interested in data logistics in 2006 through the TexasView’s involvement in the NSF-funded REDDnet project. In 2007, TexasView developed a demonstration project using REDDnet to distribute Landsat Data, and demonstrated the capability widely to USGS and other organizations. The EODN project is a direct result of that demonstration, driven by development of the DLT. AmericaView has continued to support development of data logistics-based remote sensing data distribution since that time. In 2014, AmericaView adopted EODN as a national-level project, providing funding to various state-level members for participation by installing DLT instances to join the EODN, as well as giving exposure and guidance to the project.
The DLT development team has worked closely with AmericaView members to implement the EODN. In return, AmericaView members have served as a testing resource for DLT enhancements and package releases. The partnership has produced a pre-production, functional EODN providing access to the most recent 30 days of Landsat 7 and Landsat 8 data for the United States and other select areas. Data are harvested from the USGS EROS Center in Sioux Falls, SD by an automated process and uploaded to EODN. A web-based portal provides access to the data, as well as network status and visualization services. The system is integrated with the USGS shopping cart system so that users can use USGS Earth Explorer to select Landsat scenes and download them from EODN.
Typical Workflow
- An application or workflow process subscribes to image data of interest. For example, Landsat scenes over a geographic area.
- Satellite sensor data is periodically “harvested” from the USGS EROS data center in South Dakota.
- Global policy on scene selection and distribution is set by the IDMS dispatcher process.
- Downloaded scene data is staged and replicated within the depot network based on the requested dispatcher policy.
- Once a new scene is available, metadata about the file is published to UNIS, the network information service responsible for tracking files within the depot network. The data manager service (DMS) monitors active depots and data distribution in conjunction with UNIS.
- Downloads to the application server start as soon as a notification is received over the subscription channel.
- Processed images are uploaded to the RealEarth service and made available via a browser-based interface.
- The IDMS dispatcher maintains the state of files within the depot network.
GENI-IDMS
The Intelligent Data Movement System (IDMS) was a GENI “Shakedown Experiment” designed to evaluate a distributed data storage service as a long-running experiment on the GENI infrastructure. The packaged DLT components were used as building blocks for deploying appliance images that were instantiated across GENI aggregates, and the practical deployment knowledge gained through IDMS helped improve the robustness of the DLT software, and the development of the IDMS dispatcher and policy engine. Storage depots in IDMS are discoverable alongside community deployed DLT depots but may additionally make use of a provisioned data plane for inter-GENI connectivity.
The IDMS is now incorporated into the DLT, and it also serves as an extension of the EODN, where dynamically provisioned GENI resources may be used to improve transfer performance and data availability of earth sciences data. The DLT/IDMS and AmericaView teams have collaborated in demonstrating the effectiveness of an integrated DLT deployment.
L-Store
The ability to dynamically blacklist problematic drives has paid big dividends for the CMS T2. Even though individual drives in the large L-Store storage array fail at the rate of nearly one per day, there is little if any impact on the operation, reliability and performance of the T2.
The ability to rebalance data across drives of unequal size and ability to migrate data off resources has been extremely important to the operation of the CMS T2 at Vanderbilt. We have demonstrated the ability to do this on multiple occasions and with large amounts of data (PBs).
The DLT software stack has been in production use under the demanding conditions of the Vanderbilt CMS T2 for several years now, and the experience has led to many improvements to the performance and reliability of the code.
Additional Resources:
