Alternative Evaluation Designs for Data-Centered Technology-Based Geoscience Education Projects

This paper will present different strategies for how to evaluate contrasting K-12 geoscience classroom-based interventions with different goals, leveraging the first author's experiences as principal investigator of four NSF and NASA-funded geoscience education projects. Results will also be reported. Each project had its own distinctive features but all had in common the broad goal of bringing to high school classrooms uses of real place-based geospatial data to study the relationships of Earth system phenomena to climate change and sustainability. The first project's goal was to produce templates and exemplars for curriculum and assessment designs around studying contrasting geoscience topics with different data sets and forms of data representation. The project produced a near transfer performance assessment task in which students who studied climate trends in Phoenix turned their attention to climate in Chicago. The evaluation looked at the technical quality of the assessment instrument as measured by inter-rater reliability. It then analyzed the assessment results against student responses to the instructional tasks about Phoenix. The evaluation proved useful in pinpointing areas of student strength and weakness on different inquiry tasks, from simple map interpretation to analysis of contrasting claims about what the data indicate. The goal of the second project was to produce an exemplar curriculum unit that bridges Western science and traditional American Indian ecological knowledge for student learning and skill building about local environmental sustainability issues. The evaluation looked at the extent to which Western and traditional perspectives were incorporated into the design of the curriculum. The curriculum was not constructed with a separate assessment, yet evidence centered design was utilized to extrapolate from the exemplar unit templates for future instructional and assessment tasks around other places, other sustainability problems, and other repositories of traditional ecological knowledge. The goals of the two other projects, in progress, are to build forms of support and access by teachers to complex scientific geospatial data sets concerning climate change and a myriad of related Earth system phenomena for which measurements are available from different government agencies such as NASA, NCAR, and the USGS. The driving philosophy of these projects has been that teachers are more likely to use these powerful resources when provided with curricula and educative supports, yet have the option of implementing the curricula as written, adapting the curricula, or developing their own curricula provided that they on at least some of the data about the local region. These projects are being evaluated on the extent to which this model of flexible implementation is bearing fruit in teacher capacity building and student learning outcomes. Hence, teachers are being provided with a set of pre post assessment options that they can choose from, including for example selected response items on Earth system variables of their choice, map interpretation items, and open-ended constructed response items about the weather, climate, and ecosystem concepts that they select to focus on with their students. Teacher capacity building is being measured through oral and written documentation of the teachers' evolving learning about the data resources and evolving decision-making about how to use the resources.