It’s hauntingly hard to find an optimal Learning Management System. By which, I mean to rely on one source of information, or a ratings and review portals. (We all know the problem with ratings.) Feature-based evaluations hint at capability, but it’s all too easy to see two LMS listing the same feature and get two completely different functionality and experience. It’s part of the reason we opted for considering analysis based on user stories, with reasonable success; except that it’s main takeaway is that two different user stories are really hard to compare from a statistically significant perspective.
Here’s where sentiment analysis come in. By looking into the history of public mentions for these products on social media, and using a bit of NLP-fu to assess whether these mentions were favorable of unfavorable, we came up with a list we’re comfortable with.
Full disclosure: When it comes to LMS evaluation, especially for large-scale and long-term procurement, Sentiment Analysis should be a key criteria among many. Hands-on pilots with key stakeholders remain optimal methods of evaluation, which of course are largely dependent on the organization’s budget and time frames. But by evaluating user-generated activity, it is possible to identify potential issues during regular use of the platform. At the very least, Sentiment Analysis plays a role in the quantitative risk management component of an LMS procurement process.
How the LMS Space Race Sentiment Analysis ranking was made
We looked into the vast array of Sentiment Analysis tools to make sure they provided realistic and valid measures. We soon realized the potential disparities between tools. Eventually we preferred tools who were more open about the sources of the sentiment assessment.
For this ranking, we settled on two tools: socialmention.com and social-searcher.com. We had to make key choices that might seem arbitrary and affect the list, which we explain below. But ultimately, the LMS in this list have overwhelmingly positive social media sentiment in almost every case. (The sentiment viz tool by North Carolina State University looked the most promising, but it was largely unresponsive.)
Here are some caveats before the results:
- The main issue is the relatively limited sample from which to draw conclusions. LMS are just not a consistently exciting topic to cover on social media. (Not yet anyway.) But even in more mainstream topics, there is always a high likelihood that the internet is not real life.
- Another unfortunate finding is that volume is associated with positive sentiment as well as with longevity. Meaning that, all things equal, an older brand will likely rank higher than a newer one.
- Related to the above, social media savvy companies will likely rank higher. But that’s okay. Ultimately one of the main draws of social media as a marketing space is the potential of —at least a little bit— more transparent relationships between brands and consumers, compared with traditional mass media.
- Some brands have too generic a name, making things ambiguous.
- There is, however, at least one benefit to unpopularity: Fewer incentives to manipulate the conversation. Sentiment analysis tools do account for phenomena like bots, astroturfing and brigading, but they are imperfect. A lower volume of social media activity diminishes manipulative effects.
- For most of the LMS, we looked into the website and leadership to gauge at least some subjective level of personality and trust, that would lead us to think any efforts to artificially influence sentiment scores were less likely. (Though never impossible.) We gave positive mentions a higher weight than negative ones. This is based on the well-supported finding that people are more negative on social media than in real life, as they believe, rightly it seems, that companies react quickly to customer problems when raised in public.
- Finally, an excess of positive results is not only suspicious, but has a higher potential for bias. So if two companies have the same sheer number of positive reviews, a small amount of negative posts gives more credibility to the positive ones. It sound paradoxical: Brands with a slightly lower ratio of positive to negative mentions (nothing beyond than 1 mention or 5% of all mentions) will rank higher than brands with no negative mentions at all.
Without further ado:
№20. Instructure Canvas
A surprising placement for one of the most active communities online. The latest financial developments explain most of it. As the ownership transfer completes and users realize whether the new arrangement continues to deliver on the experience that still Canvas at the top of the U.S. Higher Ed food chain.
№19. Google Classroom
A dominant force in the K-12 space, it can be difficult to separate the tool from the behemoth, creating a play of two forces: Whatever people’s feelings have towards Google get imprinted on the public debate about Classroom; and the efforts in distancing Classroom from Google, some in more justifiable ways than others. (Sometimes by Google and the Classroom team themselves.)
Not exactly an LMS, but one that across online conversations continues to be debated as one.
№17. Open LMS
Disclaimer: LMSPulse is an Open LMS, Learning Technologies Group Company.