Learning as it happens: A decade of analyzing and shaping a large-scale online learning system

With the advent of computers in education, and the ample availability of online learning and practice environments, enormous amounts of data on learning become available. The purpose of this paper is to present a decade of
experience with analyzing and improving an online practice environment for math, which has thus far recorded over a billion responses. We present the methods we use to both steer and analyze this system in real-time, using
scoring rules on accuracy and response times, a tailored rating system to provide both learners and items with current ability and difficulty ratings, and an adaptive engine that matches learners to items. Moreover, we explore the quality of fit by means of prediction accuracy and parallel item reliability. Limitations and pitfalls are discussed by diagnosing sources of misfit, like violations of unidimensionality and unforeseen dynamics. Finally, directions
for development are discussed, including embedded learning analytics and a focus on online experimentation to evaluate both the system itself and the users’learning gains. Though many challenges remain open, we believe
that large steps have been made in providing methods to efficiently manage and research educational big data from a massive online learning system.