Eleven Ways to Measure the Immeasurable and Count the Incalculable

In 2017 Ocean Networks Canada (ONC), a research infrastructure operator, sought to redefine its core reporting metrics. We asked, "which metrics should we hold as key, essential metrics to drive our organizational priorities and decision making?" This question helped us define a collection of eleven sets of yardsticks, some inward-looking, others squarely focused on societal outcomes.

This presentation will introduce the individual metrics we adopted, the insights they are helping us glean and some of their inherent challenges and ambiguities. We will also discuss the larger, more complex and interdependent impacts we lack the wherewithal to measure.

ONC's core funding agency, the Canada Foundation for Innovation (CFI), continues to emphasize scientific output as a primary criterion. We measure this by counting peer-reviewed presentations and publications resulting from use the facility and ONC's data archives. But this seemingly clear-cut metric has been a thorny one to define, track and grow. Training and support for post-secondary students is another core reporting metric, however this measurement is also fraught with ambiguities.

Some of the easier metrics to track are those specifically related to facility operations, such as reliability and user satisfaction. But we were perplexed by the question of how to measure "optimal use" of the facility, as mandated by CFI. Optimal use is hard to define for an underwater infrastructure design like ONC's, which can be flexibly extended with no hard limits on hardware capacity, archive volume or data access.

When it comes to societal benefit, our approach has been twofold. One set of metrics examines technology transfer, grants and contracts. Another set focuses on our engagements and active collaborations with governmental, indigenous and non-governmental organizations.

However, some outcomes remain challenging to measure. While it is straightforward to count up our external interactions and collaborations, how can we quantify their current and future societal impact? Additionally, while transdisciplinarity is measurable, and seems to be a worthy goal, what is its demonstrable value and how can it be measured? These and related questions will be explored.