The Chandra Source Catalog Pipeline: the Yin Yang of a Challenging Project

Production of release 2.0 of the Chandra Source Catalog (CSC 2.0) started in Spring of 2015νll. Development of the processing pipeline is the work of a small team of Chandra X-ray Center (CXC) scientists and software developers, each contributing their knowledge and expertise to the requirements, design, and implementation of the pipelines, tools, and archive infrastructure needed to produce the catalog. CSC 2.0 will more than triple the size of the original catalog released in 2009 to at least 300,000 sources. The source increase is due to co-adding multiple observations, the use of new source detection and background algorithms to include the faintest (∼5 net counts) sources, and 5 more years of data (observations released publicly during 2010-2014). The images and source detections of the deepest and most crowded fields are spectacular, and very faint sources will be found! The Chandra Source Catalog is innovative in that data products (e.g., images, exposure maps) are saved in the archive, along with tabular data (e.g., source positions, position error) for user access and analysis. The pipeline goes well beyond the traditional data reduction thread and well into detailed data analysis. We had to find the balance in developing a pipeline that runs 24/7 against the intricacies of analysis required for source detection of co-added fields. We had to evaluate where to draw the line on developing new and novel scientific algorithms since additional research impacted the project in both time and resources. We had to measure hardware costs and estimated computation time against affordability and project pressures to complete the catalog. In this paper, we provide an overview of the technical challenges we met in developing the CSC 2.0 pipelines, and review the end products available to the catalog user. We highlight the coordination and management challenges in developing the catalog against the responsibilities of a team in the out years of a mission. We also highlight lessons learned — both positive and negative — so they can benefit others on a similar path.