A recent Education Next piece from Laurence Holt highlights some promising research about technology-fueled learning apps:
In August 2022, three researchers at Khan Academy, a popular math practice website, published the results of a massive, 99-district study of students. It showed an effect size of 0.26 standard deviations (SD)—equivalent to several months of additional schooling—for students who used the program as recommended.
A 2016 Harvard study of DreamBox, a competing mathematics platform, though without the benefit of Sal Khan’s satin voiceover, found an effect size of 0.20 SD for students who used the program as recommended. A 2019 study of i-Ready, a similar program, reported an effect size in math of 0.22 SD—again for students who used the program as recommended. And in 2023 IXL, yet another online mathematics program, reported an effect size of 0.14 SD for students who used the program as designed.
These are great results, right?
Except, did you catch the caveat in all of them? The positive effects were limited to “students who used the program as recommended.”
So the key question is not whether these types of digital programs “work,” but, “how many students actually stick with them?” And the reality is not that many. Holt notes that, in the case of the Khan Academy study, it was 4.7% of students, and the highest participation rates were among student groups that were already the highest performing.
So how do schools and districts crack this “5% problem?” Holt offers one clue when he notes that, “student usage was driven more by “teacher- and school-level practices” than by “student preferences.” In other words, teachers and district leaders will need to be intentional about how to maximize the value of the tools they’re buying.
A recent study out of Utah shows some support for this theory. The state provided competitive funding for districts that were interested in adopting one of six pre-approved vendors offering digital reading platforms for K-3 students. To participate in the program, districts had to submit an application and identify which of the programs they intended to use. When the state funded an evaluation of the program, it found that students who engaged with the tools for more time saw better results. Additionally, the program had especially strong results for English Learners, students with disabilities, and low-income students who were served by the program. However, only 41% of eligible students met the vendors’ recommended dosage. That’s still not a lot, but it’s far higher than some of the voluntary results.
The engagement problem is prevalent in other promising interventions as well. Tutoring, for instance, can help students make strong gains, but when it’s voluntary, the students who need the most help are the ones who are least likely to use it. This is one of the reasons I’m so bullish on districts tackling this challenge by using performance-based contracts, where the vendor only gets paid if the students actually learn.
Similarly, there’s a lot of hype and interest around AI-backed tutoring programs like Khanmigo. But even if it’s able to eliminate more of its math errors, it will still run into usage and persistence problems, just like the original Khan Academy did.
As one leading AI researcher put it in describing an earlier effort that failed, “We missed something important. At the heart of education, at the heart of any learning, is engagement. And that’s kind of the Holy Grail.”