Python Code Parallelization, Challenges and Alternatives
Abstract
In the last few years the development of Python code for science and data reduction purposes has gained significant popularity. ESO itself uses a Python-based archiving system for VLT and ALMA data. Also the data reduction suite for ALMA data is python-based. Rapid development is fostered by a big community and a wide range of already available packages. However Python enforces locking mechanisms, to ensure thread safety, that effectively reduce the capacity of Python to use only one core. In this context a number of alternatives have been developed by the community to emulate actual multi-threading and make parallel processing easier to use from Python, preserving interactivity.
- Publication:
-
Astronomical Data Analysis Software and Systems XXVI
- Pub Date:
- October 2019
- Bibcode:
- 2019ASPC..521..515G