Health Affairs article by Rochelle Walensky
Vaccine: Implementation Over Efficacy
PUBLISHED:NOVEMBER 19, 2020Free Accesshttps://doi.org/10.1377/hlthaff.2020.020
ABSTRACT
The global effort to develop a coronavirus disease 2019 (COVID-19) vaccine is likely to soon produce one or more authorized vaccines. We examine how different definitions and thresholds of vaccine efficacy, coupled with different levels of implementation effectiveness and background epidemic severity, translate into outcomes including cumulative infections, hospitalizations, and deaths. Using a mathematical simulation of vaccination, we find that factors related to implementation will contribute more to the success of vaccination programs than a vaccine’s efficacy as determined in clinical trials. The benefits of a vaccine will decline substantially in the event of manufacturing or deployment delays, significant vaccine hesitancy, or greater epidemic severity. Our findings demonstrate the urgent need for health officials to invest greater financial resources and attention to vaccine production and distribution programs, to redouble efforts to promote public confidence in COVID-19 vaccines, and to encourage continued adherence to other mitigation approaches, even after a vaccine becomes available. [Editor’s Note: This Fast Track Ahead Of Print article is the accepted version of the peer-reviewed manuscript. The final edited version will appear in an upcoming issue of Health Affairs.
From the earliest stages of the coronavirus disease 2019 (COVID-19) pandemic, the development of safe and effective vaccines against severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), the viral cause of COVID-19, has been widely considered an essential component of any strategy to control the virus, the disease, and its effects. Since the publication of the SARS-CoV-2 viral sequence on January 10, 2020, an unprecedented global collaboration among governments, vaccine manufacturers, and researchers has been mounted to develop COVID-19 vaccines.1
In the United States, this work is supported through billions of dollars in public investment and new entities such as Operation Warp Speed and the Accelerating COVID-19 Therapeutics and Vaccines public-private partnership.2 Global coordination of vaccine research and development is provided by the Coalition for Epidemic Preparedness and Innovation (CEPI), Gavi, and the World Health Organization.3 According to CEPI, 321 COVID-19 vaccine candidates were in development worldwide as of September 2020.4 Of those, as of November 2020, over 50 have progressed to clinical testing in humans, 11 of which were in Phase 3 clinical trials—the large-scale population-based testing capable of producing the safety and efficacy evidence required for regulatory approval.5 As of early-November 2020, four Phase 3 COVID-19 vaccine clinical trials are underway in the United States, with preliminary results likely to be made available in the coming months and more complete results thereafter.6
Vaccine efficacy is a particularly critical outcome to be measured in these trials and subsequently evaluated by regulatory bodies such as the U.S. Food and Drug Administration (FDA) and its international counterparts. In a June 2020 guidance document to vaccine manufacturers, the FDA adopted a broad definition of vaccine efficacy that encompasses both transmission effects (i.e., the ability of the vaccine to prevent the spread of SARS-CoV-2 from an infected person to a susceptible person) and disease-modifying effects (i.e., the ability of the vaccine—among those vaccinated but who nonetheless become infected—to slow or prevent progression of illness, to speed recovery, to decrease utilization of critical-care resources, and/or to reduce mortality).7
Recognizing that vaccines can provide both direct protection (reducing susceptibility among the uninfected) and indirect protection (reducing viral spread in those who are infected), the FDA guidance recommended both a transmission endpoint—confirmed SARS-CoV-2 infection with one or more COVID-19 symptoms—and a disease-modification endpoint—evaluating whether a COVID-19 vaccine prevents severe disease among persons who become infected.7,8 Regardless of how a manufacturer defined their efficacy endpoint, the FDA also established a minimum efficacy threshold, specifying a primary efficacy endpoint point estimate of at least 50% to ensure—in FDA’s view—that a widely deployed COVID-19 vaccine is effective.7
These definitions and thresholds are highly consequential, yet the FDA guidance document provides no justifications for either. The 50% efficacy threshold most closely resembles the typical effectiveness of vaccines against influenza, a less transmissible, morbid, and lethal disease than COVID-19.9,10 It is also a considerably lower efficacy standard than those of virtually all other approved and widely used vaccines.11 But in the context of a global pandemic with ruinous economic and public health consequences, the FDA’s 50% threshold raises the question: Might we settle for a vaccine with more modest effects and, if so, how modest? Conversely, what infection and mortality benefits could we anticipate if recent, preliminary reports of a vaccine with 90% efficacy are confirmed?12 Would a vaccine that has a limited impact on transmission but significantly reduces progression from infection to severe disease be acceptable, even preferable? How might we compare such a vaccine to one that lowers susceptibility to infection but has no impact on disease progression?
Recent work demonstrates that dramatically different epidemic trajectories result from changing assumptions about the strength and duration of adaptive immune response to SARS-CoV-2 and its interaction with vaccines and non-pharmaceutical interventions of varying efficacy.13 Yet, these biological factors—including vaccine efficacy as demonstrated through clinical trials—are only some of the many influences whose complex interaction will determine the real-world effectiveness of COVID-19 vaccination and its ability to alter the trajectory of the pandemic. How well a vaccine program “works” will also depend on how quickly it can be manufactured, how efficiently it can be distributed to locations in greatest need, how persuasive health messaging can be in promoting public acceptance, and how consistently the public can adhere to the many complementary prevention strategies (e.g., masks, hand-washing, distancing) to limit the spread of the virus.
We sought to understand the interplay between these parallel considerations related to COVID-19 vaccines and vaccination—vaccine efficacy as determined through clinical testing and the design and execution of vaccination programs that follow. Specifically, we asked how vaccine-related changes in susceptibility to infection, progression of disease, and severity of illness might translate into population outcomes of interest, such as cumulative infections, hospitalizations, and deaths. We explored how those downstream outcomes might vary in the face of alternative operational assumptions (e.g., the pace of scale-up and the degree of public acceptance) and changes in the epidemiological context. We consider the implications of our results for ongoing efforts to hasten the development and deployment of COVID-19 vaccines in the months ahead.
STUDY DATA AND METHODS
STUDY DESIGN
We used a simple mathematical model to estimate the population benefits of a vaccine against COVID-19. We considered vaccines with varying degrees of preventive benefit (transmission effect) and disease-modifying benefit (progression and mortality effect). We considered different assumptions regarding the speed of manufacturing/distribution (pace) and extent of vaccine delivery (coverage), two implementation parameters that are independent of vaccine clinical trial results. We also considered different background epidemic severities, as measured by the reproduction number (Rt). Outcomes of interest—including total infections, deaths, and peak hospital/ICU utilization—were reported on both an absolute basis and as a percent reduction from a “No Vaccination” scenario, over a 6-month planning horizon. We initialized the simulation with a population size of 100,000, of whom 100 (0.1%) were exposed and 9,000 (9%) were recovered cases.14 The model was implemented as a spreadsheet and parameterized and validated using population-average data inputs (online appendix exhibit 1).15
COMPARTMENTAL MODEL
The SEIR (Susceptible-Exposed-Infectious-Recovered) model is one of the simplest deterministic, mathematical frameworks for portraying the trajectory of an infectious disease through an at-risk population. Briefly stated, the SEIR framework treats the process of viral transmission and disease progression as a sequence of transitions among a finite number of health states (or “compartments”). Transitions are governed by mathematical equations that capture both the transmission dynamics of the virus and what is known about the natural history of disease.
We adapted the classic SEIR framework in two important ways (appendix exhibit 4).15 First, we divided the “Infected” compartment into four distinct sub-compartments, to capture the increasing severity and resource use associated with more advanced COVID-19 disease: “Asymptomatic,” “Mild” (outpatient), “Severe” (hospitalized), and “Critical” (hospitalized in an ICU). Second, we introduced the possibility of vaccination by creating a parallel set of compartments to the ones described above. Individuals receiving the vaccine moved from the “Susceptible Unvaccinated” state to the “Susceptible Vaccinated” state. From there, their progress to Exposure, Infection, Recovery, and Death was adjusted to reflect the transmission and disease-modifying benefits of the vaccine. This modeling device also permitted us to adjust the infectiousness of persons who received an imperfect vaccine but who nevertheless became infected (i.e., “breakthrough infections”).
VACCINE EFFICACY
To capture the broad definition of “vaccine efficacy” in the FDA’s June 2020 guidance, we considered three different vaccine types (appendix exhibit 2):15 First, a preventive vaccine that decreases susceptibility to infection in uninfected persons. Second, a disease-modifying vaccine that improves the course of disease in infected persons, slowing progression, speeding recovery, reducing mortality, and decreasing infectiousness. Finally, a composite vaccine that combines the attributes of both the preventive and disease-modifying vaccines. We set the efficacy for each of these attributes at 50% in the base case and examined ranges of 25% to 75% in sensitivity analysis. (For the recovery rate increase, the base case value was 100% (i.e., cutting recovery time in half) with range 75–150%.) We considered lag times between vaccine administration and when effects take hold ranging from 14 days (representing a fast-acting, single-dose vaccine) to 30 days in the base case (representing a two-dose vaccine with administration 30 days apart and partial efficacy after the first dose) to 42 days (representing a two-dose vaccine with no efficacy after the first dose).16,17
IMPLEMENTATION EFFECTIVENESS
The challenges of vaccine development do not end once an effective vaccine is identified. The model includes two implementation measures: pace and coverage. Pace, the percent of the population that could be vaccinated on a given day, is a measure of manufacturing and logistical preparedness. We assumed a base case value of 0.5% for the pace parameter to approximate the daily rate of influenza vaccination in the US during the peak period of vaccination efforts each fall.18 This reflects our assumption that while a COVID-19 vaccine may need to be administered in two doses, the urgency of the pandemic may prompt sponsors to bring production and distribution to scale at twice the rate of the influenza vaccine. Given the uncertainty surrounding these assumptions, we considered alternative values ranging from 0.1% to 2% in sensitivity analysis. We defined coverage as the percent of the population ultimately vaccinated, a measure of public acceptance and the success of public health efforts to make vaccines available to all who desire them. We used a base case value of 50% (range 25% to 75%) reflecting recent US polling data on vaccine acceptability.19 At a daily pace of 0.5%, it would take .5/.005 = 100 days to achieve a 50% coverage goal.
EPIDEMIOLOGY AND NATURAL HISTORY
We defined three epidemic severity scenarios: a base case with reproduction number (Rt) of 1.8; a best case (Rt = 1.5) representing strict adherence to social distancing and other preventive best practices; and a worst case (Rt = 2.1) reflecting the higher risks associated with winter weather and greater indoor activity. We also report results for Rt = 1.2 in the appendix.15
Input data on the development and natural history of COVID-19 (including incubation times, likelihood of symptoms, rates of progression, recovery, and fatality rates) were obtained primarily from modeling guidance issued by the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR), supplemented by published literature.20 We attempted to use the most current input data available. However, as clinical care and outcomes improve, as testing services magnify, and as the pandemic expands its demographic reach, our analysis will require adjustment and updating. Specifically, hospitalization and mortality rates are improving as the COVID pandemic is controlled among the elderly and extends its reach to younger populations. Recognizing how quickly these statistics are evolving, we deliberately focused our attention in this analysis on infections, not deaths.
Appendix exhibit 1 documents all inputs and sources.15
LIMITATIONS
Like any model-based analysis, our evaluation has important methodological limitations. First, we assume a model of homogenous mixing. While this simplifies the underlying mathematics, recent evidence suggests that spikes in local positivity—and resultant protective immunity—may be attributable to spatially-correlated, small group gatherings. Vaccine hesitancy may also vary by setting and other demographics. Future refinements might consider more complex, geospatial or age-based mixing assumptions.
Second, we have not stratified vaccine deployment or coverage scenarios across different at-risk and vulnerable populations as suggested recently by the National Academies Framework for Equitable Allocation.21 To some extent, sensitivity analyses on Rt might serve as surrogates for alternative communities of variable epidemic control. Furthermore, our framework does not allow for differential prioritization or uptake among groups at higher risk for hospitalization and death. Published data used to populate the model are necessarily taken from early in the epidemic course. Additional evidence (e.g., age-adjusted outcomes, new strategies for COVID-19 clinical care, geographic case clustering, and patterns of vaccine hesitancy and acceptance among the public) may permit the model to be stratified by age or other dimensions and updated for risk of complications and death at the individual level.22
Third, we have assumed constant rates of transition from one model compartment to the next. This produces exponentially distributed residence times—time spent in a given state can be quite long, even if the mean duration is short—and could bias the analysis against prevention and in favor of rapid implementation. As better data on the natural history of disease emerge, it may be possible to address the problem using multiple sequential compartments.
Finally, our base case analysis restricts attention to a 6-month horizon. While we also report projections over 12 months, this should be interpreted with caution, as waning immunity after disease and vaccine durability remain ongoing concerns.23
A comprehensive description of the model, its parameters, governing equations, and input data values is provided in the appendix.15
STUDY RESULTS
BASE CASE
In a population of 100,000 and at a baseline Rt of 1.8, the model projects 61,112 infections and 2,725 cumulative deaths over the course of 6 months without a vaccine. Introducing preventive, disease-modifying and composite vaccines at baseline efficacy levels would result in 42,583, 39,767 and 31,625 cumulative infections and 1,896, 1,318 and 1,199 cumulative deaths, respectively (supplemental exhibit 1).15 Across all values of Rt, a 50%-effective disease-modifying vaccine would have a greater impact on mortality and peak hospitalizations than a 50%-effective preventive vaccine. The impact of both vaccines on total infections would be similar in a high-severity epidemic (Rt = 2.1); but the disease-modifying vaccine would have a more pronounced impact on total infections in a lower-severity epidemic (Rt = 1.8 and 1.5). The 50%-effective composite vaccine, which combines the attributes of both the preventive and disease-modifying vaccines, would have the best overall performance. However, its impact would be much less than the sum of the impacts of the other two vaccine types combined.
SENSITIVITY TO VACCINE EFFICACY
To explore different possible clinical trial outcomes, we set vaccine efficacy variables to 25%, 50%, and 75% while holding all program implementation parameters constant (supplemental exhibit 2).15 We considered two implementation scenarios: first, a base case (left panel) with pace = 0.5% and coverage = 50%; and a more aggressive implementation (right panel) with pace = 1% and coverage = 90%.
Greater vaccine efficacy always produced more favorable outcomes. In the case of preventive vaccines, the returns to increased efficacy were close to constant. For example, under base case implementation assumptions (supplemental exhibit 2, left panel)15 and Rt = 1.8, the incremental contribution to infections and deaths averted from a preventive vaccine with efficacy 25%/50%/75% were 14%/17%/17% and 14%/17%/17%, respectively (see appendix exhibit 5 for results on deaths averted).15 Results with vaccine efficacy as high as 90% follow similar trends (see appendix exhibit 6).15 By contrast, there were markedly diminishing marginal returns to increased efficacy using disease-modifying and composite vaccines; these vaccines attained much of their full potential effect on outcomes at efficacy level 25%. For example, under the aggressive implementation scenario (supplemental exhibit 2, right panel)15 with Rt = 1.8, the incremental infections averted from a disease-modifying vaccine with efficacy 25%/50%/75% were 40%/22%/9%; incremental contribution to deaths averted were 62%/15%/5% (appendix exhibit 5 and appendix exhibit 6).15
Supplemental exhibit 215 illustrates that potential benefits of even the most optimistically effective vaccine are diminished if it is introduced into a more severe epidemic. For all three vaccine types, a 75%-effective vaccine implemented in a population where Rt = 2.1 averted a smaller proportion of infections and deaths than a 25%-effective vaccine implemented under less severe pandemic conditions (Rt = 1.5). The figure also illustrates that vaccination programs that confer higher levels of protection—even if for a smaller fraction of the target population—generally outperform strategies that confer lower protection on a broader population. For example, a 75% effective vaccine administered to 50% of the population bettered a 25% effective vaccine given to 90%. The findings presented here persisted for Rt = 1.2, for time horizons extending to 12 months, and for efficacy delays ranging from 14 to 42 days (see appendix exhibits 7A, 7B, 8A, and 8B).15
Supplemental exhibit 215 also illustrates the modest superiority of the composite vaccine. While it achieved the greatest reduction in infections for any combination of pace and coverage, its impact was much less than the sum of the infections averted by the preventive and disease-modifying vaccines.
SENSITIVITY TO IMPLEMENTATION EFFECTIVENESS
To understand how imperfect implementation might affect vaccination program success, we held all vaccine efficacy parameters at their base values (i.e. efficacy 50% separately for Rt 1.8, 1.5, and 2.1) and simultaneously varied the two program uptake parameters: pace and coverage (supplemental exhibits 3–5).15 With Rt = 1.8 (supplemental exhibit 3),15 a disease-modifying vaccine that attained even 90% coverage only averted 6% of infections at a pace of 0.1%; that same vaccine averted only 11% of infections at coverage 10%, even when it attained a pace of 2.0%. Bringing both coverage and pace up to their base case levels (50% and 0.5%) averted 35% of infections. (See black highlighted cells in the figure.) The pattern observed is one that persisted for Rt = 1.5 (supplemental exhibit 4) and 2.1 (supplemental exhibit 5).15 It was also observed across all vaccine types (appendix exhibit 9A),15 and all lag-time assumptions (appendix exhibits 9B and 9C):15 sufficient pace and coverage function as complements, not substitutes, and both are necessary for a vaccination program to produce large reductions in infections. High performance on one implementation measure cannot fully compensate for low performance on the other.
The impact of a vaccine dissipates dramatically as the severity of the epidemic (i.e., Rt) increases. For example, a disease-modifying vaccine with 50% coverage and 1.0% pace averted 82%/58%/35% of infections when Rt = 1.5/1.8/2.1. (See white highlighted cells in supplemental exhibits 3, 4, and 5.)15 All other things being held equal, the proportional power of any vaccine to reduce infections, deaths, and peak hospitalization was greatest at lower values of Rt.
While shorter efficacy lag times invariably resulted in more favorable vaccine outcomes, the qualitative findings highlighted here for the 30-day efficacy lag were similar to those for vaccines with efficacy lags of 14 and 42 days (appendix exhibits 9B and 9C).15
The results of additional sensitivity analysis are reported in the appendix.15
DISCUSSION
Our results demonstrate that the benefits of any COVID-19 vaccine—whether highly, moderately, or modestly efficacious by any trial-defined outcome—will depend at least as much on how swiftly and broadly it is implemented and the epidemiological environment into which it is introduced as it will on the vaccine’s physiological properties as shown through clinical trials. While these latter vaccine-specific characteristics are fixed, the medical, public health, and government communities can productively intervene with respect to the contextual considerations that would increase the benefits of a vaccine upon its introduction.
First, the effects of any COVID-19 vaccine are highly dependent on the effective reproductive number of the virus (Rt) at the time a vaccine is deployed. In our model, Rt functions in part as a proxy for the success of efforts to promote widespread, sustained adherence to risk mitigation strategies such as masking, physical distancing, and limitations on large gatherings.24 When Rt is comparatively low (1.5)—indicating that viral circulation is being controlled through these non-pharmaceutical measures—vaccines with low efficacy (25%) are capable of producing larger reductions in the fraction of infections and deaths than vaccines with much higher efficacy (75%) introduced at times when Rt is significantly higher (2.1). Furthermore, the additional benefit of a vaccine with 25% vs. 75% efficacy very much depends on the background Rt; in cases of outbreak control (Rt ≤1.5), a vaccine with 25% efficacy might well have a substantive impact. Even the effects of a vaccine with 90% efficacy—as Pfizer has characterized the performance of its vaccine in preliminary press statements—relies heavily on the background Rt at the time of its introduction.12
The complexity of infectious disease transmission dynamics accounts for what may be this counterintuitive result, one that goes against the usual finding in clinical needs assessment that efforts should be targeted where severity is greatest. For a highly infectious disease, even a vaccine with seemingly adequate efficacy, pace, and coverage may be insufficient to alter the fundamental population dynamics that produce high disease prevalence. Mathematical modeling has shown that differences in steady-state prevalence and the marginal steady-state impact of vaccine effectiveness are typically inversely proportional to the reproduction number.25 The same is true in prevention interventions such as syringe exchange, which is more effective against HIV than against hepatitis C given differences in Rt.26
Managing and reducing Rt requires a sustained commitment to the public health practices and tools known to reduce the spread of COVID-19. Investment in these activities remains imperative not simply until the arrival of a vaccine but throughout the likely prolonged period during which a vaccine is being deployed.
Second, our results show that the effectiveness of a COVID-19 vaccine will be shaped by the success or failure of efforts to deliver a trusted vaccine quickly to the public. The pace of vaccination—how quickly the vaccine is introduced—will be determined by a combination of manufacturing capacity, the development of distribution systems and infrastructure, the creation of mass vaccination clinics in diverse locations, and related logistical considerations. The vaccine benefit also depends on how many doses are required. Most vaccines currently in large-scale clinical trials are two-dose series, including those from Pfizer and Moderna most likely to be authorized first.5 A two-dose vaccine that take 28–42 days to achieve efficacy—where maximum efficacy may be reached during the coming winter months with a higher Rt—should be expected to have diminished impact when compared to a one-dose vaccine with only a 14-day delay to efficacy.
Vaccination coverage—the percentage of the population that ultimately receives a vaccine—is dependent on efforts that foster widespread public enthusiasm for vaccination and address sources of hesitancy for vaccines, in general, and COVID-19 vaccines, in particular.27,28 It also requires efforts to ensure that vaccines are accessible to all communities, particularly underserved groups for which longstanding disparities in vaccination coverage have been observed.29,30 This includes racial and ethnic minority groups among whom the effects of COVID-19 have been disproportionately felt.31,32 Delivering the vaccine to as many people as possible as quickly as possible can result in large reductions in infections and death, even at higher Rt. Conversely, a slow pace of vaccination or low vaccination coverage dramatically reduces the benefits of vaccines even with moderate or high efficacy.
Some of the activities associated with accelerating vaccine production, distribution, deployment—such as the advance manufacturing of vaccine doses while clinical trials remain underway and planning for robust post-authorization vaccine safety monitoring—have received considerable attention and investment from Operation Warp Speed and other federal agencies. But many other components of vaccination planning are at much earlier stages of development, even as vaccines rapidly approach potential FDA authorization and as mass vaccination efforts are expected to begin virtually immediately thereafter. Among the issues for which considerable work remains are the detailed design of what will be exceedingly complex and unprecedented vaccine supply chain strengthening and distribution activities in communities (e.g., medical records and lot documentation to track dual-dose vaccine administrations and investments in ultra-low temperature cold-chain capacity) and companion, culturally-sensitive and evidence-based communication to promote vaccine acceptance and convey the continued need for other prevention practices even after a vaccine becomes available.
State and local health officials expected to design and carry out much of the on-the-ground work related to COVID-19 vaccination have stated since spring that they lack sufficient funds to do so successfully.33,34 In September 2020, the CDC director concurred, telling Congress that an additional $6 billion in federal funding to states was required for their role in vaccine distribution and related public outreach.35 The need for those funds—an amount representing only half of the approximately $12 billion committed to COVID-19 vaccine development to date—was disputed by other federal health officials, and Congress has not acted on this request as of early November 2020.36
Overall, our results suggest that the significant public optimism regarding the potential value of vaccines in reducing the burden of COVID-19 is warranted, even if vaccines in development are shown to be only moderately efficacious. Not surprisingly, vaccine-associated benefits increase with greater levels of efficacy against infection, infectiousness, disease progression, and mortality. But even vaccines below the 50% efficacy threshold established in the June 2020 FDA guidance document could make valuable contributions to COVID-19 prevention and response.
But the efficacy of the COVID-19 vaccines currently being studied in phase 3 trials and soon to be reviewed by the FDA, while important, will be only one contributor to the overall effectiveness of the vaccination programs of which they may eventually be part. The ultimate success of COVID-19 vaccination efforts will depend on embracing a wide range of vaccine efficacy profiles at the time of authorization or approval, managing expectations regarding how a vaccine will contribute to public health responses in tandem with the continued use of non-pharmaceutical interventions, and investing substantially and rapidly in efforts to rapidly deliver vaccines to as large a portion of the population as possible and to quickly identify and respond to sources of vaccine hesitancy. Such a strategy would maximize the individual and population benefits of any authorized or approved COVID-19 vaccines and increase the likelihood that they approach the very high expectations placed upon them.
