Our series on The Learning Analytics Roadmap endeavors to provide a set of practical thinking posts to help you start an analytics-based practice. While analytics are not new in learning, the effort today is about fostering a critical mass among education professionals. Therefore, this series is not intended to require any particular background, other than a proven desire to inform, with the hopes to enhance, teaching.
Analytics have been in educational institutions for some time. In Australia, the Office for Learning and Teaching has made and attempt to follow-up on the efforts and results of applied analytics in the classroom, with the “Completing the Loop” Project. The most meaningful outcome of the project is this year’s release of a handbook with lessons learned and practical applications, available as a PDF file. Unfortunately, it reports only on a limited pilot that took place as part of the project.
“Completing the Loop” went from early 2014 to mid-2016, encompassing three phases. Phase 1 consisted of surveying teachers’ perspectives. Phase 2 entailed a development effort built on top of Blackboard and Moodle, which helped teachers design and implement a strategy with an analytics component. Phase 3 attempted to apply analytics-based learning from the previous phase into a limited sample of pilot courses.
The main phase-1 finding was the “relatively basic in nature” requests and expectations of analytics, by surveyed teachers. Familiarity with analytics was varied, but in general the handbook reports a missing “sense of what was possible in terms of available data and suitable analysis techniques“. Considerations about engagement, for instance, often took the form of “how long students persisted with watching [a video]“.
Nevertheless, finding of an open mind to the possibilities of analytics allowed in Phase 2 for the set up and fine-tuning of the “Loop Tool”, a piece of guiding software to “enable teachers to articulate the connections between learning outcomes“, among other functions. Loop packaged some data visualization.
The Loop Tool is freely available at its GitHub Repository. It requires Moodle or Blackboard.
Monitoring the implementation of the Tool in pilot classrooms was the main focus of Phase 3. Evaluation assessments provided by the tool did not seem to extend beyond teachers’ opinions on usability, which was considered “satisfying”, “a nine out of ten” and that “did not represent a steep learning curve“. Teachers expressed willingness to continue using the Tool beyond the pilot.
Short of a survey on actual analytics usage in the past years, “Completing the Loop” serves an encouraging role especially for those in an early stage of interest about using analytics. From the point of view of the ROI on Learning Pyramid (discussed previously in the Series), the Loop Tool belongs to the base, namely the “learning (reaction and) satisfaction stage”, since the only assessment found in the handbook was the pilot teachers’ personal testimony.
While the community is looking forward to increased roles of analytics and supporting technologies in learning interventions, it is critical to look for lessons learned. One of which seems, judging by the “Completing the Loop” handbook, that general surveys do not help uncover remarkable accounts of successes or insight from the past.
The full document reviews the methods and outcomes of the research, critical pedagogical takeaways, details on the Loop Tool functionality and technical notes for IT staff. It also introduces some theoretical background.