A Review and Autopsy of Two COVID Immunity Studies

By MARTIN KULLDORFF

How effective is immunity after Covid recovery relative to vaccination? An Israeli study by Gazit et al. found that the vaccinated have a 27 times higher risk of symptomatic infection than the Covid recovered. At the same time, the vaccinated were nine times more likely to be hospitalized for Covid. In contrast, a CDC study by Bozio et al. claims that the Covid recovered are five times more likely to be hospitalized for Covid than the vaccinated. Both studies cannot be right.

 
I have worked on vaccine epidemiology since I joined the Harvard faculty almost two decades ago as a biostatistician. I have never before seen such a large discrepancy between studies that are supposed to answer the same question. In this article, I carefully dissect both studies, describe how the analyses differ, and explain why the Israeli study is more reliable.
 

The Israeli Study

 
In the Israeli study, the researchers tracked 673,676 vaccinated people who they knew not to have had Covid and 62,833 unvaccinated Covid-recovered individuals. A simple comparison of the rates of subsequent Covid in these two groups would be misleading. The vaccinated are likely older and, hence, more prone to have symptomatic disease, giving the Covid recovered group an unfair advantage. At the same time, the typical vaccinated patient received the vaccine long after the typical Covid-recovered patient got sick. Most Covid recovered patients got the infection before the vaccine was even available. Because immunity wanes over time, this fact would give an unfair advantage to the vaccinated group.
 
To make a fair and unbiased comparison, researchers must match patients from the two groups on age and time since vaccination/disease. That is precisely what the study authors did, matching also on gender and geographical location.
 
For the primary analysis, the study authors identified a cohort with 16,215 individuals who had recovered from Covid and 16,215 matched individuals who were vaccinated. The authors followed these cohorts over time to determine how many had a subsequent symptomatic Covid disease diagnosis.
 

 
Ultimately, 191 patients in the vaccinated group and 8 in the Covid recovered group got symptomatic Covid disease. These numbers mean that the vaccinated were 191/8=23 times more likely to have subsequent symptomatic disease than the Covid recovered. After adjusting the statistical analysis for comorbidities in a logistic regression analysis, the authors measured a relative risk of 27 with a 95% confidence interval between 13 and 57 times more likely for the vaccinated.
 
The study also looked at Covid hospitalizations; eight were in the vaccinated group, and one among the Covid recovered. These numbers imply a relative risk of 8 (95% CI: 1-65). There were no deaths in either group, showing that both the vaccine and natural immunity provide excellent protection against mortality.
 
This is a straightforward and well-conducted epidemiological cohort study that is easy to understand and interpret. The authors addressed the major source of bias through matching. One potential bias they did not address (as it is challenging to do) is that those with prior Covid may have been more likely to be exposed in the past through work or other activities. Since they were more likely to be exposed in the past, they may also have been more likely exposed during the follow-up period. That would lead to an underestimate of the relative risks in favor of vaccination. There may also be misclassification if some of the vaccinated unknowingly had Covid. That would also lead to an underestimate.
 

The CDC Study

 
The CDC study did not create a cohort of people to follow over time. Instead, they identified people hospitalized with Covid-like symptoms, and then they evaluated how many of them tested positive versus negative for Covid. Among the vaccinated, 5% tested positive, while it was 9% among the Covid recovered. What does this mean?
 

Though the authors do not mention it, they adopt a de facto case-control design. While not as strong as a cohort study, this is a well-established epidemiological design. The first study to show that smoking increases the risk of lung cancer used a case-control design. They compared hospitalized patients with lung cancer and found more smokers in that group compared to non-cancer patients, who served as controls. Note that if they had restricted the control group to people with (say) heart attacks, they would have answered a different question: whether smoking is a larger risk factor for lung cancer than it is for heart attacks. Since smoking is a risk factor for both diseases, such a risk estimate would differ from the one they found.

In the CDC study on Covid immunity, the cases are those patients hospitalized for Covid disease, having both Covid-like symptoms and a positive test. That is appropriate. The controls should constitute a representative sample from the population from which the Covid patients came. Unfortunately, that is not the case since Covid-negative people with Covid-like symptoms, such as pneumonia, tend to be older and frailer with comorbidities. They are also more likely to be vaccinated.
 
Suppose we wanted to know whether the vaccine rollout successfully reached not only the old but also frail people with comorbidities. In that case, we could conduct an age-adjusted cohort study to determine if the vaccinated were more likely to be hospitalized for non-Covid respiratory problems such as pneumonia. That would be an interesting study to do.
 
[the_ad id=”126906″]
 
The problem is that the CDC study answers neither the direct question of whether vaccination or Covid recovery is better at decreasing the risk of subsequent Covid disease, nor whether the vaccine rollout successfully reached the frail. Instead, it asks which of these two has the greater effect size. It answers whether vaccination or Covid recovery is more related to Covid hospitalization or if it is more related to other respiratory type hospitalizations.
 
Let’s look at the numbers. Of the 413 cases (i.e., Covid positive patients), 324 were vaccinated, while 89 were Covid recovered. That does not mean that the vaccinated are at higher risk since there may be more of them. To put these numbers in context, we need to know how many in the background population were vaccinated versus Covid recovered. The study does not provide or utilize those numbers, although they are available from at least some of the data partners, including HealthPartners and Kaiser Permanente. Instead, they use Covid-negative patients with Covid like symptoms as their control group, of which there were 6004 vaccinated, and 931 Covid recovered. With these numbers in hand, we can calculate an unadjusted odds ratio of 1.77 (not reported in the paper). After covariate adjustments, the odds ratio becomes 5.49 (95% CI: 2.75-10.99).
 
Ignoring covariates for the moment, we will look at the unadjusted numbers in more detail for illustrative purposes. The paper does not report how many vaccinated and Covid recovered people there are in the population at risk for hospitalization with Covid-like symptoms. If there were 931,000 Covid recovered and 6,004,000 vaccinated (87%), then the proportions are the same as among the controls, and the results are valid. If, instead, there were (say) 931,000 Covid recovered and 3,003,000 vaccinated (76%), then the odds ratio would be 0.89 instead of 1.77. There is no way to know the truth without those baseline population numbers unless one is willing to assume that those hospitalized for Covid-like symptoms without having Covid are representative of the background population, which they are unlikely to be.
 
With a background population to define a cohort, one must still adjust for age and other covariates as in the Israeli study. Some may argue that the Covid negative hospitalized patients with Covid-like symptoms are a suitable control group because they provide a more representative sample of the population at risk of Covid hospitalization. That may be partially true compared to an unadjusted analysis, but the argument is incorrect as it does not address the key issue of the relevant medical question being asked. There is both a relationship between being vaccinated/recovered and Covid hospitalization and a relationship between being vaccinated/recovered and non-Covid hospitalization. Rather than evaluate the first one, which is of intense interest for health policy, the CDC study evaluates the contrast between the two, which is not particularly interesting.
 
[the_ad id=”155165″]
 
The CDC study adjusts for covariates such as age, but the procedure does not resolve this fundamental statistical issue and may even exacerbate it. Frail people are more likely to be vaccinated, while active people are more likely to have been Covid recovered, and neither of those are properly adjusted for. With the contrast analysis, there is also more confounding that must be adjusted for: both the confounding related to the exposures and Covid hospitalizations and the confounding related to the exposures and non-Covid hospitalizations. This increases the potential for biased results.
 
While not the main problem, there is one other curious fact about the paper.

Read more