By Chad Aldeman and Dale Chu
In a report for FutureEd last month, Lynn Olson and Tom Toch advocate for a “matrix sampling” approach to federally required achievement tests. Basically, instead of testing all kids in grades 3-8, states could test just a sample of students and still get high-level accountability results.
As Dale noted last week, this approach seems to be gaining steam as a way to break the assessment stalemate. Indeed, it’s an approach championed by the National Center for Assessment last year and the Center for American Progress back in 2021, and the state of Washington has proposed a similar approach.
If states could get the same information with a lot less testing time for students, that sounds like a good deal, right? Well, there are three major downsides to the sampling approach:
- Reducing the federal testing footprint might not even work.
The goal behind these proposals is to reduce the amount of time that kids spend taking tests, potentially freeing up more time for classroom instruction.
But the federally required state tests are not the main problem in terms of testing time. In fact, these exams account for only a small fraction of the time typically devoted to assessments in a given year. No, the real culprits are the layers upon layers of other tests mandated by states and local districts. All of these additional tests won’t go away even if the feds dial back their testing stipulations.
- Parents would no longer receive objective information about their own child.
This may seem an obvious point, but the federal government currently requires schools to inform parents about their child’s performance against state standards in reading, math, and science. If states switched to a sampling approach, there’s no guarantee that parents would be able to access objective, comparable results on how their own child is doing.
Sure, parents might still be able to access general information about school and district performance, but they wouldn’t have an honest appraisal of whether their child was reading and doing math at grade level. This is one of the key reasons annual testing remains popular.
As parents, this point is a big deal to us. We don’t enjoy watching our kids spend time taking tests, but we recognize the valuable information those tests provide. A sampling approach would eliminate that possibility.
- Policymakers could no longer analyze subgroup performance at the school or district level.
Because of sampling size issues, it simply wouldn’t be possible to look at school-level results for different student subgroups. This might sound hyperbolic, but the sample sizes start to get real small real fast, which eliminates the possibility of looking at whether a given school did an effective job serving all of its students.
Here’s what one of us (Chad Aldeman) found when he ran the numbers on grade-span testing back in 2015:
To get a sense of how many students could become newly “invisible,” consider public elementary schools in Washington, D.C. Applying the same minimum group size currently used for entire schools to the fifth grade only, about half of the city’s 119 elementary schools with fifth graders taking math tests would not be held accountable for the progress of low-income or African-American students, because there aren’t enough of them in that grade to constitute a reliable sample size. For that same reason, less than 10 percent of schools would be responsible for Hispanic students or English language learners, and not a single elementary school would be accountable for the progress of students with disabilities.
The same math applies to school districts as well. Across the country, there were almost 9,000 school districts that served between 100 and 1,000 students last year. Those districts educate over 4 million students, but a shift to a sampling approach or grade-span testing wouldn’t tell us much about different subgroups of students.*
To be sure, sampling has a lot of technical and theoretical merits, but it would move us in the wrong direction, especially right now. In the wake of the pandemic, historically marginalized students fell even further behind. Policymakers at the state, district, and school level need honest, objective information about individual student performance in order to target recovery supports.
It’s also a bad time to backtrack on transparency. There’s a troubling “perception gap” where 90% of parents think their kid is at or above grade level in reading and math, even as objective data puts the number much lower. If anything, states need to do a better job of getting that data back to parents quickly so they have time to act on it—to say nothing of presenting it in a more compelling way
Both of us support additional innovation in assessments. But policymakers need to be clear-eyed on the policy tradeoffs in different approaches. In the case of sampling, the perceived benefits need to be weighed against the very real costs. Instead of backtracking, policymakers should be focused on how to improve the transparency and usability of state testing data, not trying to make excuses for gathering less of it.
*It’s technically possible to over-sample certain student populations to get enough data to draw meaningful conclusions—that’s what pollsters do when their sample is not representative or if they want to hone in on certain populations—but that would defeat the purpose of a sampling approach. If you think the stakes of state tests are high now, imagine being among the small group of students whose scores will determine the results of your entire school.
