Toward a Virtual Laboratory to Assess Biodiversity from Data Produced by an Underwater Microscope

Real-time data from sensors deployed in the ocean are increasingly available online for broad use by scientists, educators, and the public. Such data have previously been limited to physical parameters, but data for biological parameters are becoming more prevalent with the development of new submersible instruments. Imaging FlowCytobot (IFCB), for example, automatically and rapidly acquires images of microscopic algae (phytoplankton) at the base of the food web in marine ecosystems. These images and products from image processing and automated classification are accessible via web services from an IFCB dashboard. However, until now, to process these data further into results representing the biodiversity of the phytoplankton required a complex workflow that could only be executed by scientists involved in the instrument development. Also, because these data have been collected near continuously for a decade, a number of "big data" challenges arise in attempting to implement and reproduce the workflow. Our research is geared toward the development of a virtual laboratory to enable other scientists and educators, as new users of data from this underwater microscope, to generate biodiversity data products. Our solution involves an electronic notebook (Jupyter Notebook) that can be re-purposed by users with some Python programming experience. However, when we scaled the virtual laboratory to accommodate a 2-month example time series (thousands of binned files each representing thousands of images), we needed to expand the execution environment to include batch processing outside of the notebook. We will share how we packaged these tools to share with other scientists to perform their own biodiversity assessment from data available on an IFCB dashboard. Additional outcomes of software development in this project include a prototype for time-series visualizations to be generated in near-real-time and recommendations for new products accessible via web services from the IFCB dashboard.