Improving Student Outcomes Through Informed Use of Learning Analytics

Although computer learning systems can provide extensive information about student behavior and success or failure in certain learning activities, it is far from trivial to utilize this data for effective student success interventions at large scale. Here we detail a successful learning analytics-based intervention in Habitable Worlds, a fully online science course for undergraduate non-science majors.

Learning analytics from past offerings of the course show performance on "training" lessons to be a consistent discriminator between students who succeed and fail. Most of the course material consists of paired "training" and "application" lessons, with trainings graded for completeness and applications graded for correctness. There are two notable differences between successful and unsuccessful students. First, students who do not complete the first week's training material earn grades more than one letter grade lower than those who do. Second, students who pass the course are more likely to complete the trainings and applications in their assigned order during the semester, while students who fail are more likely to attempt the application lessons first. Together these show that timely completion of the trainings is closely linked to course success, suggesting interventions that encourage this strategy will be successful.

In Spring 2018, we employed two such interventions. First, the instructor sent a targeted email to students who had not completed the first week's training lessons. This email noted the historical evidence that completing training lessons was highly correlated with success. Second, there was a course policy change to offer a small bonus point award to students who complete each week's training lessons early. Using a regression analysis, we compared Spring 2018 to the prior two spring terms (n = 1046). Results show positive shifts of 0.3-0.4 standard deviations on training and application lesson scores as well as course grades. Notably, failure to complete the week one training lessons (the trigger for the email intervention) was half a letter grade less predictive of course grade in the Spring 2018 term. This successful intervention demonstrates a model for use of key behavioral metrics from course analytics to design informed and effective interventions that improve student outcomes.