Episode 6:

An Education Researcher Sounds the Alarm on Learning Loss

Transcript:

Jim Cowen This is Jim Cowen from the Collaborative for Student Success, and this is the “Route K-12: Exploring Education Recovery” podcast. Each week, we travel the country on a kind of “road trip” to talk about the ways federal recovery dollars are being used in states to reshape education. Along the way, we’ll hold up the best examples with the hope that those practices are repeated in other schools.

Our guest today is Tom Kane, faculty director at Harvard’s Center for Education and Policy Research. In collaboration with Dartmouth, the University of Washington, and NWEA, Tom co-authored one of the strongest reports yet on the impact of the pandemic on learning loss and the potential interventions required to help students recover. His findings paint a pretty dismal picture and he’s sounding the alarm on how important it is to address this issue while there are federal funds available to do so. He talks about how district leaders need to be honest with parents about how bad the problem is, what it’s going to take to get these students caught up, and what investments would be most effective in doing so.

Jim Cowen Tom, thanks for joining us today.

Tom Kane Thanks Jim. I’m happy to be here.

Jim Cowen The theme for our little podcast is “road tripping” across the country and so before we get into this information and we get too depressed talking about your research, I hope we could start this on a bit of a high note. And so, I’ll ask you, what was your most memorable road trip and is there a song that sort of encapsulates that?

Tom Kane Yes, so the road trip that immediately comes to mind was a trip I took with like a dear friend of mine from high school. It was back in the late ‘80s. I was in grad school by then, and he was moving from our hometown, Winston Salem, all the way out to Hermosa Beach, California and I-40 is the interstate that runs almost that whole trip. And so, we just had the best time driving across country and the song that comes to mind, which if I were a baseball player might be my walk-up song, is Katrina & The Waves – Walking on Sunshine.

Jim Cowen That’s a good one. That’s a great one. I like that a lot.

So, Tom, I want to start this off with an acknowledgement that we have partnered with our friends at Georgetown’s Edunomincs Lab and the Center on Reinventing Public Education, CRPE, on something that we call the Education Recovery Hub. And the whole idea there is that we’re scouring the country—this is part of that promotion, this kind of road trip idea—and we’re holding up examples of what we think are good investments that are happening. Good practices, good intentions, good spirit in states on their use of federal recovery dollars.

We’ve seen things from focusing on accelerated learning, using data to drive decisions, promoting family engagement, strengthening teacher voice; all look like they’re promising at efforts. However, if I look at your recent results from your research correctly, you’re saying “That might be well and good, Jim, but those practices simply aren’t going to be enough to get kids back on track after the pandemic.” Is that fair? Can you outline your study and your overall findings?

Tom Kane Sure. Let me first outline it and then explain why I’m trying to sound the alarm now. So, in our study, we had data from 49 states plus the District of Columbia. So, one of our research partners was NWEA, which administers their assessment to districts all around the country. And we added in data on the number of weeks that schools were remote during 2021. And we learned some really important and interesting things.

Number one is that even for schools that went back to in-person throughout 2021, they lost ground because remember everybody was out in the spring of 2020. So, it’s just that, you know, they lost the equivalent of about 10 weeks worth of learning. In those places that were in-person throughout 2021, the average student lost ground, but there was no widening gap like Blacks and whites, Hispanics and whites, high poverty low poverty, schools all lost about the same amount.

The story was really different though in the places that remained remote during 2021, especially the ones that remained remote for a half a year or more. In those places, students lost—in high poverty schools—lost the equivalent of about 22 weeks’ worth of instruction and the students in the low poverty schools that were out for more than half a year also lost ground, but they lost something like the equivalent of 13 weeks. Now, as we think about what kind of response is going to be commensurate, it’s important to just pause and say, “22 weeks of instruction? That’s a lot of instruction missed.” And it should be obvious, once you think about that, that if the average kid missed the equivalent of 22 weeks in those schools, you’re not going to make up for that magnitude loss with a little summer school or, you know, by providing tutors to a small percentage and the research confirms that, you know. I was startled because, as an education researcher, I know that there have been very few interventions that have ever been shown to have an impact close to 22 weeks of instruction. The one type of intervention [that] has been shown to have an impact that size is high dosage tutoring, but districts are not prepared to implement high dosage tutoring in the scale that would be required. You’d have to provide a high dosage tutor to every single child in a high poverty school that was remote for half a year during 2021 in order to make up for that magnitude gain. You know providing tutors to 10% of those kids is not going to be nearly enough to do it.

Jim Cowen When I think about the actual funding allocations too, when districts were supposed to allocate 20%, I think, of relief dollars to learning loss.

Tom Kane Yeah.

Jim Cowen So, considering that and considering your comments, that means, well, they’re going be put every penny to learning loss and that just doesn’t feel feasible.

Tom Kane So, Jim, here’s the thing everybody’s got to remember: when the American Rescue Plan passed in March of ‘21, nobody knew how big the achievement losses were going to be.

Jim Cowen Right.

Tom Kane And there was a lot of anxiety at the time that state and local revenues were going to fall short. And so, you know, Congress gave school districts just a ton of leeway. They were just saying, “Look, we don’t know how bad it’s going to be.” So, the 20% minimum was an absolute minimum. It wasn’t a suggestion and yet I think as district plans have been coming out, it’s as if they’ve been taking it more as a suggestion, rather than as an absolute minimum. So, we’re seeing plans where just a little over 20%, they’re planning to spend a little over 20% on academic catch up.

But one way to see clearly that they’re going to need to spend more than that in some places or in a lot of places would be to just say, “OK, 22 weeks”—say if I’m at one of those high poverty school districts that lost the equivalent of 22 weeks—”well, that’s more than half a year.”

Jim Cowen Right.

Tom Kane So how much is it going to cost to make up for that? Well, not less than half a year’s worth of a budget, right? I mean, what’s it cost to run a school for a school district for half a year? And it’s not going to be cheaper to try to make up for that amount of lost learning with tutors, like that’s got to be a lower bound. So, what does it cost if you’re running with class sizes of, you know, 23, 25 kids in elementary school, or larger in middle school? So, half of a typical year’s budget seems like a lower bound to what it’s going to cost to catch up and yet, for a lot of districts, the amount of federal aid they received added up to about a half a year. So that’s saying, “OK, well, they should be expecting to spend—at least in the places that were remote for more than half the year in 2021—those places should be expecting to spend a large share, if not all of the dollars on academic catch up.”

Jim Cowen So, this all makes complete sense to me of what you’re saying with the problem with learning loss. But I’m trying to think of this from the feasibility standpoint of everything else that people are dealing with, right? With mental health issues, teacher shortages, others, like if high dosage tutoring is the best way to combat learning loss, being laser focused on that, do you think the country has the political will to go laser focused on learning loss while leaving the other issues of mental health and others?

Tom Kane First of all, I think that addressing mental health needs has to be part of the academic strategy. Like I don’t think it’s an either or there, like of course we’ve got to be thinking about what kind of supports would help students reengage with school and that we should be thinking that as part of the academic catch up strategy.

So, one constraint is teachers supply, tutors supply. But one key challenge that we need to also keep in mind is just the bandwidth of district and school leaders…

Jim Cowen Right.

Tom Kane …in planning a set of interventions over and above normal school. So just running a school, just one school district, you know in a normal year is challenging enough. If adding on top of it all these other things, like the tutors and the summer school and so forth, double dosing, it’s pushing the bandwidth of the leadership in these systems even more than it’s, you know, constraining their abilities to spend dollars.

That’s why I’m hoping that more districts will start to think about, “OK, shouldn’t we extend the school year in the next couple of school years?” It’s certainly not a politically attractive option. I’m not naive enough to think that it would be. But I think when school districts start to think about their ability to deliver tutoring on the scale that would be necessary, or the ability to hire enough math teachers to provide double dose math to everybody, that once they start to think about that, the idea of extending the school year will become a lot more attractive. And hopefully will be politically preferable to letting these gaps become permanent.

I think once people realize the challenge of trying to scale up interventions and the long-term consequences of letting these gaps become permanent, that the political feasibility of extending the school year will improve because it may be logistically, in some districts, the only way that they’re going to be able to provide interventions on the scale large enough to catch kids up.

Jim Cowen So, I’m actually trying to put myself in the shoes of a state superintendent or a district leader who’s staring at this information. Given, you know, what we’ve all seen to be the current situation for school board members, or any sort of public facing role, a leadership role right now it is tough. I mean, it’s a super divisive moment. Like I’ve never seen anything like it before. And unfortunately, you know, so many ideological views, political views are crossing over to the classroom and that’s, you know, creating a whole new dynamic that makes these sort of decisions tough. I’m just curious, what advice or recommendations you would give a public leader around education on how to process this information or act on it.

Tom Kane Yeah, so Jim, that is a really critical…

Jim Cowen It’s the million-dollar question, right? It’s a multimillion-dollar question.

Tom Kane You’ve absolutely put your finger on it. Here’s what I would suggest: district leaders need to be transparent with parents about what it’s going to take to catch up. And it would be helpful for them to start to do the math themselves and then share that math, like calculate what is the size of the loss that a specific district has, you know, suffered.

And one, for researchers like me, the challenge is to describe that loss in a way, in the same kind of units, that the effect sizes of the available interventions are measured in. Now earlier in this conversation, I was converting everything into weeks, but another way to do it would be to convert things into standard deviations.

Jim Cowen Oh, I was worried you were going to go into standard deviations.

Tom Kane But I’m doing that not just because I like that language but because, unfortunately, those are the units that most of the research is reported in. So, when people say like, “What’s the effect of high dosage tutoring,” if you go look up the research, it’s not going to say, “Oh, it’s about 19 weeks.” What the research is going to say is it’s about 0.38 standard deviation. And, you know, every district ought to work with their research team to calculate the magnitude of their losses. Either converted into weeks, using one of the conversions I was doing, or leave it in standard deviations, but whatever they do, put it in the same units as the effect sizes of the available interventions and then just do the math. So, if we’ve lost 22 weeks on average, if I’m in a district that’s lost 22 weeks and I say, “OK, here I’ve got this effect size for tutors, which is, you know, 19 weeks. And I can give that to 10% of my kids. Then that means 0.1 times 19, which is equal to 1.9,” which is less than one tenth. So how much would an average achievement rise if I provided tutors to 10% of my kids? Not 19 weeks, but 1.9 weeks and then do a similar calculation adding in what proportion of kids are we offering double dose math to? How many kids are going to summer school? And it’ll be by doing that adding up that a leader could help parents and teachers see, “Hey, look, we’re just not going to get there with these kinds of interventions.”

Here’s what I’m fearful of: of course, people are going to learn this next spring when the spring ‘23 test scores come out. And they’re going to say “Well, gosh, I guess we didn’t, you know, we’re still down, you know, X points in proficiency. Wow, so I guess we didn’t do enough in terms of interventions this year.”

But at that point there’s only one more year left to spend the federal dollars. And so somehow, especially late summer, like if people realize it in July or August ‘23, that they’ve not done enough, there’s not much time left to use the federal dollars [for] the amount of catch up. Even though I agree we haven’t quite gotten there yet, if there are any superintendents out there who understand this problem, who want to work with us to come up with the best language for describing it and doing this calculation we’ve just been talking about, they should reach out because that’s the trick is getting the sense of urgency while there’s still federal dollars to spend on these things.

Jim Cowen Yeah, yeah. I wanted to ask you about the research collaboration that you have happening right now with some of the districts. What sort of things are you finding, are you hoping to find by sort of gathering these different districts together and are you looking at individual interventions…

Tom Kane Yes.

Jim Cowen …and trying to see which ones are the biggest impact? Is that how that’s working?

Tom Kane Yeah, there’s a set of districts that we started working with last fall. And, you know, everybody says they want to be using this time to learn about what’s where it’s not, but people rarely take a step back and say, “OK, so what data do we need to be collecting so that in the future we can say something about whether this was working or not?” And in particular, we told these districts, “Hey, look, you’re going to have to track which kids are getting tutoring. Not just that you’re doing tutoring, but which kids are getting the tutors? Track how many sessions they go to.” We’re going to need that in order so that we can tell you later whether the kids have got tutors had any bigger gains. So, in August, we’re planning to provide a report on “here were the kids that got tutors and here were their gains during ‘21/ ‘22 versus the kids that didn’t.” We’re also looking at other things like double dose and after school programs. There were some districts in Texas that we’re working with that added some weeks of instruction in some of their schools during intercessions during the school year. So, we’ll get to see hopefully, you know, what the impact of that was.

Texas is interesting in that they have a House Bill 4545, which required schools to provide 30 hours of small group instructions to kids that score below proficiency on the state test. So hopefully, because we’ve got a couple of Texas districts in our group that we’re working with, hopefully we’ll be able to say something about like what were the effects of that kind of approach of prescribing some kind of minimum amount of extra help kids must receive based on their performance on the end of year test. I know that districts in Texas are struggling with figuring out how to implement this, but we’re going to be hopefully saying something about just how effective it is in terms of helping kids catch up.

Jim Cowen Yeah. Well, this is a point of our conversation where we take a question from a parent throughout the country. And I really like this one that this person has today, so we’re going to play that for you now.

Rachel Ramirez Hi Jim and Tom. My name is Rachel Ramirez and I am the mother of two boys transitioning from middle to high school this summer. Much of the research and news coverage I am seeing as a parent indicates that students are faring even worse academically than many of us expected. I am worried my kids are further behind than they should be, but they’ve been bringing home A’s and B’s for much of the last year. How can I really identify how my kids are doing and how can I push my school district to make sure my kids are getting the learning acceleration that they need?

Tom Kane Parents should be asking their school district two questions. Number one, what is the magnitude of our district’s losses? Don’t just tell me where my kid is right now but tell me where in pre-pandemic years, they should have been like at the end of seventh grade, or the end of eighth grade, or so forth. If a district is using the NWEA test or the iReady or any of these other interim assessments, they ought to be able to say, “Where is my kids in terms of norms relative to, you know, prior years? So, is my kid at the right, at the median of their class this year, but they’re at the 37th percentile of where they should have been in a pre-pandemic year. So, tell me on average and for my kid how much ground we’ve lost. And then don’t just tell me what the school district is doing, but provide for me a rationale for why what you’re doing you think is going to be enough” and that will require them to do this kind of math that I’ve been urging people to do.

The frustrating thing is that when the state reports results back to a school district, they just report declines in proficiency rates. They’re not reporting it in terms of weeks lost or in terms of standard deviation losses, because if they did, it would be easier for school districts to do that comparison to the effect sizes, and then say, “Holy cow, what we’re doing is not enough.” Until these things are reported in the same units, it’s really hard for anybody who’s not an education researcher to know that most districts’ plans are not nearly enough to make up for the losses their students have experienced.

Jim Cowen So, I have one additional question for you given this very bleak outlook on a current situation: where is there hope? What actually gives you some inspiration that we’re going to get through this and be able to deal with this?

Tom Kane So here’s what I’m hoping: I’m hoping that there will be some states and some school districts out there that are eager to implement a more aggressive package of interventions in the spring of ‘23. So, take the next few months to plan a major set of interventions in the spring of ‘23. We’d love to work with those places to evaluate the effects of those things so that we can then provide the rest of the country with evidence on, “OK, here’s a set of interventions that will be enough to close the gap.” I mean the problem is I couldn’t say do these four things and you’re going to make up for the gap. Until we have that, it’s going be very hard for advocates to hold local leaders accountable for whether or not they did those things. And what we need now is some folks to step forward, work with researchers [or] practitioners to just plan a set of interventions that on paper seem like the right magnitude that would help students catch up, pilot them in the spring of ’23—we’d work with our data to measure the effects—so that early in the summer of ‘23 we could start getting the word around about, “OK, here’s the kind of thing. Here’s a package of interventions that would be sufficient to close the gap.”

Failure’s not an option here. If we let these losses become permanent, this is like a historic widening of the Black white, Hispanic white, high poverty low poverty achievement gap. So, we’ve been working for 30 years to close the Black white achievement gap and Hispanic white achievement gap and we’ve made, you know, gradual progress over the last 30 years. And not a lot of people are aware of that, but actually even before the pandemic we were making progress, but if we let these achievement gaps become permanent, we will have given up like between a half and a third of that progress over the last 30 years.

And I think most of your listeners out there don’t want to stand by and let that happen. And so that’s what I guess that’s what gives me hope is I just know as people come to understand just the stakes here, that there are a lot of good people working on this. And, you know, we’ve got to get ourselves organized to deliver on it. This is going to be a state and local led effort that the federal government has provided the resources, but they’re just not in a position to provide leadership on kinds of interventions—that’s going to have to be found at the state and local level. I’m hoping we get enough state and local examples in the next year that would then inspire others to copy during the ‘23- ‘24 school year.

Jim Cowen I want to thank you for your time. This is such an important issue to talk about. We certainly look forward to staying in touch with you and your research collaboration as more information comes through. And we’re hoping that we can make a dent in this issue. Thank you very much for your time.

Tom Kane Great. Thanks Jim.

Jim Cowen This is Jim Cowen from The Collaborative for Student Success. Thanks for listening to today’s episode of Route K-12: Exploring Education Recovery. For each week, we’ll travel the country to showcase the ways federal recovery funds are reshaping schools. Along the way we’re talking to people doing the hard work to educate America’s kids. Got a question or insight you’d like to share about what’s going on in education? We’d love to hear it. Reach out to us at EduRecoveryHub.org/RouteK12 or follow us on Twitter at our handle @StudentSuccess.

About Jim Cowen

Jim is the the executive director of the Collaborative for Student Success, a non-profit organization that defends strong K-12 policies that benefit all students.