A review published by the US’s National Bureau of Economic research offers the first known attempt to organize the existing scientific literature about educational technologies (EdTech). By focusing exclusively on research that applied randomized controlled trial or regression discontinuity design techniques, the review combines experimental and observational data to “add a productive voice to broader and more methodologically-diverse policy research dialogues in an environment characterized by complex tangles of cause and effect.” Given the specificity of the available research, it is not advisable to draw conclusions for specific scenarios, perhaps other than stressing the importance of more rigorous impact evaluation in the field.
According to the goals of the items in review, authors classified each into one of four categories and then gave a general assessment about the findings for the individual categories.
“Access to technology”: Does the mere introduction of computer-based technology help student outcomes? A definitive “Yes” regarding computer skills, not guaranteed regarding academic achievement. The studies with the largest samples in this category show either “null effects” or inconclusive results. 15 items belong in this category.
“Computer-Assisted Learning”: Does specialized computer-assisted software help student outcomes? A 20-strong majority of findings says yes, with just 8 concluding no effects and one claiming negative outcomes. The largest positive effect reported was an increase of 0.63 standard deviations on Texan kids studying 7th grade math. There seems to be no correlation between time spent using the software and higher scores. Sample sizes here were on the small size, particularly when compared to “Access to technology.” There were 29 items in this category.
“Behavioral interventions”: Do learning technologies based on behavioral economic theory help student outcomes and the quality of their decisions? While behavioral economics might suggest a level of sophistication in the software evaluated, it also includes simple parental engagement reminders. But as it turns out, simpler is better to evaluate. This category offers even wider diversity of subjects and methods with the authors detailing some of the items individually. These suggest readers might want to read items relevant to them, beginning on page 56. 47 items were available.
“Online courses”: Does online education lead to student outcomes at least as good as through face-to-face settings? Probably. Education with lessons taking place in physical settings yielded higher results, but never more than about 5% on average. 9 items were available.
Some noteworthy caveats about the research include:
- All the studies included in the review were conducted in developed countries, although the authors’ definition allowed for places like Chile and Romania alongside the US or the Netherlands.
- Most of the research took place after 2010, with few articles published as early as 2003, and several undated.
- The review featured articles published in peer-reviewed journals as well as working papers, evaluation reports, and unpublished manuscripts. The review itself is a working paper.
- The limited availability of research items did not allow the authors to control for student attributes such as age, level of education, or family income.