Evaluating Broader Impacts: Issues and Opportunities (Invited)

The NSF expects that funded projects will include activities that have intrinsic Intellectual Merit and that promote the Broader Impacts (BI) of this work to those outside the immediate research community. These BI activities take many forms, but often involve collaborations with schools, informal science centers, and media developers in efforts to promote the public understanding of science, encourage a talented and diverse pool of students to pursue careers in science, and illustrate the benefits society derives from scientific discoveries. A critical question is how to evaluate individual BI activities and the overall portfolio of BI activities. What are the metrics for success? How can evaluation results be used to improve the BI portfolio? Evaluation of BI activities is complicated by several factors, including: [1] The scope of BI activities is highly variable across different types of NSF-funded projects. Individual research projects typically have limited BI activities, with only modest funding (<10%) designated for these efforts, and evaluation efforts are rare. On the other hand, large projects such as research centers and major facilities typically have dedicated BI/education specialists and formal evaluation expertise. An additional complication is that many BI activities are unique or have novel aspects that reflect the local circumstances and opportunities, but make it difficult to develop broadly-applicable evaluation instruments;[2] There is not consensus on the perspective from which the evaluation should be conducted. Scientists, participants (teachers, students, museum/aquarium personnel), and the funding agencies typically have differing objectives and metrics for BI projects. [3] The timeframe for conducting any evaluation is frequently limited to a few years, placing limitations on the scope of the evaluation effort. Long-term learning, career impacts, and changes in cultural attitudes or perceptions are difficult to assess under this circumstance; [4] Limited financial resources sharply constrain the depth of inquiry for many studies, and, consequently, most BI efforts have not been formally evaluated or have only been through relatively simple formative evaluation surveys; [5] There is not a well-developed community effort focused on using evaluation results to identify the most creative and sustainable BI activities that have high impact, and, as a result, the importance of a thoughtful evaluation effort is often underplayed. There are many notable BI activities that have developed from the creativity of NSF-supported researchers, but it has proven difficult for these individual efforts to coalesce into a community-wide national endeavor that is recognized for its creativity and effectiveness. Recent advances in the learning and communication possibilities associated with broadband-hosted hand-held devices amplify the critical need to capitalize on these local and individual successes and bring them to a national scale. The evaluation community is poised to play a critical role in this process by working with the various BI stakeholders (scientists, participants, agencies) to identify the most successful activities for dissemination/refinement.