The effect of state-level stay-at-home orders on COVID-19 infection rates

Background

The emergence of the COVID-19 virus and the lack of therapeutic responses have led to widespread implementation of nonpharmaceutical interventions (NPIs) across the United States. Substantial evidence exists for the effectiveness of NPIs in preventing the spread of infection.^{1 , 2} However, significant controversy still exists about the value and cost-benefit of these interventions in response to the current pandemic. Specifically, it is unclear what social isolation actions must be mandated by government policy to reach desired mitigation outcomes.³

In response to the rising death toll associated with COVID-19, 42 states and the District of Columbia have implemented some variation of a state-wide stay-at-home order. In each case, there was controversy regarding the appropriateness of the order, and it is widely understood that these decisions incur human and economic costs.⁴ In the current analysis, we present data from the states with stay-at-home orders and examine the effect of these policies on the rate of increase in COVID-19 diagnoses.

Methods

Results

Table 1 displays the dates stay-at-home orders went into effect as well as logged, preorder and postorder slopes and 95% confidence intervals for all 42 states and the District of Columbia. R2 fit statistics are also shown. In general, the fit of the slopes to the logged data is excellent, as would be expected from an exponentially growing pandemic in a naive population. The average rate of increase preorder was 0.113 (95% C.I.: 0.110, 0.115) per day and postorder was 0.047 (95% C.I.: 0.045, 0.048) per day

Fig 1 presents meta analyses results using combined data across states. Two meta analyses were conducted using different sets of weights: number of days during the period and final number of cases during the period. Weighing by number of days yielded a pooled standardized mean difference of 3.847 (95% C.I.: 3.653, 4.041; P < .0001). Weighing by final number of cases yielded a similar result, with a pooled standardized mean difference of 6.847 (95% C.I.: 6.831, 6.863; P < .0001). Overall, these slope changes translate to a reduction from about 12% more cases per day (and thus a 5 to 6-day doubling rate) to 5% more cases per day (and thus a 14-day doubling rate).

The same analyses were repeated with postorder datasets starting 1 week after stay-at-home orders went into effect. In this sensitivity analysis, the pooled standardized mean difference was 4.229 (95% C.I.: 4.011, 4.448; P < .0001) after weighing by number of days and was 6.811 (95% C.I.: 6.798, 6.824; P < .0001) after weighing by final number of cases.

Discussion

These data suggest a remarkably consistent and important effect associated with the issuing of stay-at-home orders and are generally supportive of such measures. In combination with other social isolation approaches and NPIs, these orders may play a significant role in “flattening the curve.” However, this study has a number of important limitations. First, it would be impossible to isolate the effect of these orders against the background of numerous other local, state, and federal interventions occurring at the same time. Second, the expected COVID-19 expansion curve in the absence of interventions to reduce the transmission is unknown. As such, it is possible that the observed slope changes are driven by the natural history of the pandemic as well as the specific policies in question. Two sensitivity analyses were conducted to examine whether the number of days in the pre- and postperiods were drivers of either the slopes or the change in the slopes between periods. In both cases, we found no relationship. Third, states implemented stay-at-home orders in response to the pandemic and these observations are thus profoundly threatened by selection and indication biases. Fourth, even with the use of trend data in this analysis, there is a threat of regression to the mean if stay-at-home orders were consistently placed at the peak of epidemic growth.

Finally, and perhaps most importantly, a major limitation of this work is the endogenous relationship between case counts and both the availability and use of testing. The availability of testing was a significant barrier to COVID-19 diagnoses in the early stages of the pandemic in the United States and improved gradually over the month of March. Tests per day increased roughly 10-fold between March 15th and March 31st and have increased at a more gradual rate of 4-fold over the last 6 weeks.⁶ We conducted an additional sensitivity analysis to examine whether the decrease in case incidence between pre- and postorder periods was different if the order occurred before or after April 1st and found the change in slopes to be nearly identical. As mentioned earlier, we also conducted a sensitivity analysis to account for the effect of time on these slopes and found no effect. Overall, the results must be interpreted in the context of this threat. Not only was the availability of tests evolving over this time period, but there is a direct relationship between the number of infections and public awareness of the pandemic and the number of tests performed. As such, it would be difficult to begin to parse out the effect of testing on the rate of incidence of COVID-19. That said, it is likely that the availability of tests may have an unmeasured impact on these results, but we cannot know if the impact was to increase or decrease the incidence rates. Despite these limitations, the consistency and strength of the results is notable. The results presented here will be updated daily and available for public download at www.hpmcovidpolicy.org.

Conclusions

Stay-at-home orders are generally supported by our results. This data is publicly available and provides daily updates on national infection rates. By tracking stay-at-home orders and the resulting infection rates, information is gathered on the effectiveness of such orders in mitigating the spread of COVID-19. The data may also inform policy and mitigation efforts for future infection outbreaks. Further research is needed to examine the effect of these policies in the longer term and in combination with other interventions.