Variability of the impact of COVID-19 on student achievement

Dan Goldhaber, Thomas Kane, Andrew McCheen, Emily Morton, Tyler Patterson, Douglas Steger 28 July 2022

The Covid-19 pandemic has led to a critical moment in the history of US education. There is now strong evidence that barriers dramatically affect student learning and well-being and exacerbate pre-existing educational disparities for historically marginalized students (Curriculum Associates 2021, Darling-Aduana et al. 2021, Dorn et al. 2020, Greunig 2020, Kogan and Lavertu 2021, Lewis and Kuhfeld 2021). District leaders, teachers and communities preparing for the upcoming school year need to know how far behind their students are in order to provide targeted academic support. But while we know that students have lost ground academically as a result of the pandemic, much of the existing research has not documented how or why these effects vary across districts, making it difficult to provide universal recommendations about the scale of additional support needs, or the subjects and students in any one district. aimed at recovery efforts.

To that end, we recently published two reports that provide a comprehensive examination of the academic impact of the epidemic, as well as differences in those impacts across districts (Goldhaber et al. 2022a and 2022b). We use NWEA MAP Growth Test data from more than two million students across 49 states to examine changes in achievement and growth from fall 2017 to fall 2021. To examine the effect of the pandemic on growth, we estimate the effect size of the difference between the fall 2019 to fall 2021 pandemic period and the most recent pre-pandemic period, the difference in student academic growth from fall 2017 to fall 2019.

Our reports make four primary contributions. First, we add to the growing body of evidence that the pandemic has been devastating to student achievement and growth. Relative to fall 2019, midterm test scores in fall 2021 decreased by about 0.20 standard deviations in math and 0.10 standard deviations in reading. In perspective, these drops are larger than those observed in Louisiana after Hurricanes Katrina and Rita (Sacerdot 2012) and larger than the COVID-19 drops predicted by researchers in the spring of 2020 (Burgess and Sivertsen 2020). We estimate a decline in two-year achievement growth during the pandemic period. We used the average NWEA test score gain per week over a school year (across grades 3–8) to translate our estimates into weeks of learning lost. Nationwide, students were on average three months behind their expected achievement in math and more than two months behind in reading.

Second, we show that the impact of epidemics on test scores was not the same across districts. Using data on school districts’ remote learning status from the American Enterprise Institute’s ‘Return to Learn Tracker,’ we find that for the 2020-21 school year, primarily remote school students, high-poverty school students,1 Elementary school students, and students of color, tend to be more negatively affected. Our analysis shows that the impact of distance schools on historically marginalized students was twofold. High-poverty schools, which had a higher percentage of students of color, were more likely to work remotely for longer years. And The negative effect of remoteness was larger for subgroups of students served by these schools. We estimate that students in high-poverty schools who were absent 50% or more in 2020-21 were 5.5 months — or more than half of a school year — behind in math in the fall of 2021.

Our analysis of the incidence and effects of distance schooling allows us to make a third contribution, documenting the role of distance instruction in widening the achievement gap by race/ethnicity and school poverty status. As shown in Figure 1, math test score declines were similar between high-poverty and low-poverty districts that did not operate remotely for the large majority of the 2020-21 school year. But, among remote districts for more than half of the year, math-score drops in high-poverty schools were about 1.7 times the size of low-poverty schools.

Figure 1 Epidemic achievement effects by distance schooling and school poverty (math).

Fourth, we show that despite broad trends showing patterns of districts, students, and subjects most affected by the epidemic, district population and the amount of time a district has spent remotely do not tell the whole story. The spread of points in Figure 2 shows that the impact of the epidemic on test scores varied widely across districts. Although about 90% of districts experienced below-expected achievement (the point below zero on the Y-axis), not all districts did. Districts serving low-achieving students, who would already be expected to have low achievement in the fall of 2021, tend to fall further behind (all points in the bottom-left quadrant of the figure). But in some cases, math results for fall 2021 were quite different (the same is true for reading tests), with similar pre-pandemic achievement, enrollment, student population, income levels, and distance instruction volume in 2020-21. District A and District B are similar on all these counts. But students in District A were about two weeks behind in math than would be expected from the pre-pandemic year, while students in District B were about 14 weeks behind. This suggests that districts wishing to accurately target students for epidemic-related academic recovery need to carefully evaluate local data rather than relying on national trends to estimate local recovery needs.

Figure 2 Intermediate Fall 2021 District Math Achievement Changes (Grades 3-5)

Taken together, our results support calls for the urgent implementation of additional support for students at a scale tailored to a district’s needs. Fortunately, districts have access to nearly $190 billion in federal aid through the Elementary and Secondary School Emergency Relief (ESSER) fund, which amounts to about $3,850 per student. To put this sum into perspective, according to the US Census, public school spending per student was about $13,500 in FY 2020 (US Census Bureau 2022). Thus, additional funding provided through ESSER (which may be mandated until September 2024) exceeds 20% of the average cost per student.2 Thus, districts have a tremendous opportunity to invest in academic recovery interventions.

Districts are implementing a variety of strategies, including but not limited to class sizes, tutoring programs, summer learning programs, Saturday academies, virtual learning programs, extended school days and years, double-dose math and reading blocks (FutureEd 2022). Unfortunately, we have limited evidence of the effectiveness of many of these strategies; Even those most promising for recovery, such as tutoring, may not have a large enough impact for full recovery in many districts (e.g. Lynch et al. 2022, Filges et al. 2018).

High-dose tutoring (HDT)—tutoring conducted by a qualified tutor in one-on-one or very small group settings for at least 30 minutes several times a week—stands out as an evidence-based strategy with great potential. Large effects of HDT on math scores were found in elementary school students (+0.44 standard deviation, or 16 weeks) and middle school students (+0.20 standard deviation, or 14 weeks) (Nickow et al. 2020). However, implementation challenges this year, such as finding available tutors, prevented or postponed HDT in many districts. Moreover, based on our calculations, for the most disadvantaged districts, even providing HDT to all students may not close the recovery gap.

Full academic recovery from the pandemic will almost certainly take multiple years and multiple strategies. Unfortunately, we know from a wealth of research that conceptually well-founded programs often fail to improve student outcomes (e.g. Heinrich et al. 2010). Timely monitoring and evaluation of districts’ recovery initiatives will be essential, so we can adapt our strategies over time and give our children the best chance at recovery.

reference

Burgess, S and HH Sievertsen (2020), “Schools, Skills and Learning: Impact of COVID-19 on Education”, VoxEU.org, 1 April.

Curriculum Associates (2021), “Academic Achievement at the End of the 2020-2021 School Year: Insights After More than a Year of Disruptive Teaching and Learning”, June.

Darling-Aduana, J, HT Woodyard, TR Sass and SS Barry (2022), “Learning mode choice, student engagement and achievement growth during the Covid-19 pandemic”, CALDER Working Paper 260-0122.

Dorn, E, B Hancock and J Sarakatsanis (2021), “COVID-19 and education: The lingering effects of unfinished learning”, McKinsey & Company, 27 July.

Filges, T, CS Sonne-Schmidt and BCV Nielsen (2018), “Small Class Sizes for Improving Student Achievement in Elementary and Secondary Schools: A Systematic Review”, Campbell Systematic Review 14(1): 1-107.

FutureEd (2022), “How local educators plan to spend billions in federal Covid aid”, 7 June.

Goldhaber, D, TJ Kane, A McEachin, E Morton, T Patterson and DO Staiger (2022), “Consequences of Remote and Hybrid Instruction During Pandemics”, NBER Working Paper 30010.

Goldhaber, D, TJ Kane, A McEachin and E Morton (2022), “A Comprehensive Picture of Achievement Across the COVID-19 Pandemic Years: Examining Variation in Test Levels and Growth Across Districts, Schools, Grades, and Students”, CALDER Working Paper 266- 0522.

Grewenig, E, P Lergetporer, K Werner, L Woessmann and L Zierow (2020), “Covid-19 school closures hit low-achieving students especially hard”, VoxEU.org, 15 November.

Heinrich, CJ, RH Meyer and G Whitten (2010), “Supplemental education services under No Child Left Behind: Who signs up and what do they gain?”Educational evaluation and policy analysis 32(2): 273–298.

Kogan, V and S Lavertu (2021), “The COVID-19 pandemic and student achievement on Ohio’s third-grade English language arts assessment”, 27 January.

Lewis, K and M Kuhfeld (2021), “Learning during COVID-19: An update on student achievement and growth at the start of the 2021-22 school year”, Northwest Evaluation Association (NWEA), December.

Lynch, K, L An and Z Mancenido (2022), “The Impact of Summer Programs on Student Mathematics Achievement: A Meta-Analysis”, Edworkingpaper 21-379.

Nickow, AJ, P Oreopoulos and V Quan (2020), “The Affective Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Empirical Evidence”, EdWorkingPaper 20-267.

Sacerdote, B (2012), “When the Saints March Out: Long-Term Consequences of Student Evacuation from Hurricanes Katrina and Rita”, American Economic Journal: Applied Economics 4(1): 109–135.

US Census Bureau (2022), “Per-pupil spending continues to rise in 2020”, May.

Endnote

1 We defined high-poverty schools as schools with more than 75% of students eligible for free or reduced-price lunch and low-poverty schools as schools with less than 25% of students eligible for free or reduced-price lunch.

Districts were allocated 2 ESSER funds based on the Title 1 funding formula, so the amount of funding received by each district varied by an average of $3,850 per student.

Leave a Reply

Your email address will not be published.