Running the SkyMapper Science Data Pipeline: To be a Big Fish in a Small Pond, or a Small Fish in a Big Ocean?
Abstract
We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio.
- Publication:
-
Astronomical Data Analysis Software and Systems XXV
- Pub Date:
- December 2017
- Bibcode:
- 2017ASPC..512..393L