Blackboard’s John Whitmer Startles You Into Analytics. MoodleMoot US 2017-NOLA

2293

“Dr. John” jumped for a few minutes from his Blackboard’s Data Science milieu and back to the Moodleverse to ask a sobering question:

WIRIS

“I won’t make you say this out loud, but how many of your courses don’t use the higher-level functionality within your LMS?”

Despite the improvements and competition among services, Whitmer testifies to the percentage of users who see Moodle, Moodlerooms or Blackboard Learn as a glorified file repository. A situation he is aware of, almost painfully. In Higher Ed, the main focus of Whitmer’s research, most courses are put together haphazardly, as if neglecting –or mistrusting?– the LMS. At the same time, he feels a sense of responsibility as he is, to some extent, holding the keys to a door with plenty of answers. His research can inform your course design, but he also needs your help.

“We’re putting so much time and money into these systems. And we know they are recording the transactions and interactions [teachers and students have] with these systems. Wouldn’t it be nice if we could wrangle that data into something that’s useful and meaningful, so we have a sense of what people did, whether the efforts paid off, and how we can do better?”

His talk in New Orleans is an update on the research he has done in recent years trying to respond to these concerns. Some of it has been readily available; in this research he not only makes the case for better use of the already available data, but also for empowerment. Including data reviews in higher ed, in a way not unlike the way a company reviews statements monthly or quarterly, should be a more common practice, and one Moodle and similarly featured LMS are ready to support this new practice.

There are also advantages of using the recordings LMS generate as opposed to surveys, which require extra effort and expense and offer significant results about the segment of the population who answers surveys. In the case of Blackboard, surveys would have hardly procured the team with data about 1.2 million students in more than 34 thousand courses taught at 788 institutions through Blackboard Learn or Moodlerooms. An impressive sample that perhaps only Moodle can potentially match. But while the massive data set gives results more statistically trustworthy results, they felt discouraging at first. On average, spending one more hour using the LMS explains less than 1% of the final grade.

For many reasons, Whitmer expected a similar result, given how 53% use LMS in a “Supplemental” way. Focusing on the ways LMS use does have an effect in educational outcomes led to the creation of additional categories for the remaining 47%. The different groups helped him look more closely into the circumstances in which LMS use is associated with performance. Whitmer also thought it important to look into the usage pattern of the different activities within the groups. That way, the research would avoid a tendency to nudge course designers away from the supplemental level and towards a higher involvement with the LMS. In the end, these are the five categories of usage and their patterns:

  • Supplemental (53% of courses in the sample): Low interaction, use of LMS almost exclusive to accessing course content.
  • Complementary (24%): 1-way use. Course content is the main focus, but other activities gain prominence based on the whim of the teacher.
  • Social (11%): Messaging activity becomes significantly more important, second to course content. At this point, there seems to be another finding, namely a correlation between LMS involvement and class size.
  • Evaluative (10%): Assessment activities win over course content.
  • Holistic (2%): Other activities becomes important again, but assessment also overtakes course content. However, at this point the sample might be too small to generalize.

Whitmer admits the research poses more questions than it answers. In particular, the idea of personalization, as a particular case of a holistic level of LMS use, deserves pause. He closes by coming back to his initial question, suggesting that before we commit even more resources to gain higher engagement in institutional LMS, we have to make sure it makes sense to do so. Holistic engagement might not suit every case. As he joked, it might be too Californian.

“LMS use is not a proxy for effort,” Whitmer warns conclusively. Pushing users to spend more time on an LMS not only could be wasteful, but it could override natural and useful engagement and end up contradicting personalization.


eThink LogoThis Moodle Practice related post is made possible by: eThink Education, a Certified Moodle Partner that provides a fully-managed Moodle experience including implementation, integration, cloud-hosting, and management services. To learn more about eThink, click here.