Advancing Airborne Hyperspectral Data Processing for Sustainable Agriculture with Parsl

Airborne hyperspectral remote sensing data can provide rich spectral signatures and high spatial resolution to accurately quantify diverse agroecosystem variables (such as crop traits, soil properties, and management practices). These data, like data in many other fields, can have high data dimensions and require complex procedures with extensive computation for data processing. Once the raw data are obtained, they are run through a pipeline that processes the raw airborne hyperspectral imagery through radiometric, geometric, and atmospheric corrections to obtain high-quality surface reflectance, which will be further used to quantify agroecosystem variables. This extensive computation on large data sets is both complex to program and time- and resource-intensive to execute. To address these challenges, we have leveraged the advanced parallel computation Python package, Parsl, to enable the processing of airborne hyperspectral data in a high-throughput manner. Using Parsl has allowed us to process our data on a range of HPC architectures, in an adaptably parallel, and scalable fashion, reducing runtimes by over 75% using a single 24-core system when compared to a completely serial run on 1 core. This can be reduced further by optimizing the chunking of the data, utilizing in-memory files instead of files on disk, and using multiple systems in parallel.