Closing the loop: surveying PIs who have not published their data

With high over-subscription rates and significant operational costs, observatories must ensure that their operations are efficient and effective. A number of key performance indicators are generally used to evaluate the observatory's performance among which are the numbers of publications and citations of refereed journal articles to measure the overall scientific impact. Those measures, however, are broad and can not assess whether the observatory was successful on a project-by-project basis to deliver data to the PIs enabling them to carry out their science and to publish their results. In particular the reasons that prevented PIs from publishing remain hidden. Understanding and acting upon those reasons, however, have the potential to substantially improve the observatory's operational model. Of course not every approved project even should lead to a publication. Indeed, the risk of not finding the expected (or any unexpected) science in the data the PI receives is an inherent and indispensable part of the scientific process. But even here, measuring the fraction of such projects can lead to valuable insights which might then be used to instruct future proposal review committees. To fully close the loop on the end-to-end data-flow, ALMA has started in March 2015 to send survey questions to PIs where two years after the end of the proprietary period no publication making use of the delivered data could be identified. We describe our method as well as the type of conclusions we hope to be able to draw once a statistically relevant sample of answers has been received.