Vaccine Safety Research: Too Much Confounding and Spin.

In a systematic review, we found that authors of vaccine safety observational studies conceded residual confounding in about 75% of the studies. Moreover, spin was noted in about half of them.

Historically…

Historically, postlicensure safety monitoring has relied on passive surveillance and ad hoc epidemiological studies. Both are useful due to their relatively low operational costs.

However, observational studies based on passive surveillance data suffer from several methodological limitations, which may hinder the ability to draw reliable conclusions. Notably, passive reporting systems do not allow for comparisons between vaccinated and unvaccinated persons.

More recently…

More recently, large-linked databases, such as the Vaccine Safety Datalink (VSD) in the United States and the Vaccine Adverse Event Surveillance and Communication in Europe, have gained popularity. However, vaccination coverages are very high in these databases and few unvaccinated control subjects are left available for comparative analyses.

While postlicensure observational studies of vaccine safety are of critical importance to public health by generating evidence on incidence, prevalence, associations, and prognosis of certain conditions, their nonexperimental nature renders them particularly susceptible to unmeasured confounding and different kinds of bias.

Specifically, differential health-seeking behaviours between vaccinated and unvaccinated individuals often give rise to the healthy vaccinee effect – a situation when patients, who are in better health conditions, are more likely to adhere to the vaccination.

Our study

In our work out in the American Journal of Medicine (also fully available here), we systematically reviewed studies published in 6 top medical journals from their inception through March 2024 and found several interesting findings: 37 studies were examined.

Here are our main findings:

• Authors often acknowledge residual confounding – residual confounding was conceded in 54% of studies, and an additional 24% did so implicitly.

• Authors rarely use negative control outcomes (NCO) – NCO were used in a single study.

• Spin is common – spin practices were noted in 48.6% of the studies.

All the 37 included studies focused on vaccines for the prevention of infectious diseases and concerned a variety of products for children and adults. Unsurprisingly, a large number of studies (30%) focused on COVID-19 vaccines. Most studies (41%) were conducted in the United States.

Despite the intrinsic limitation of observational study designs in eliminating residual confounding, some methods, such as negative control outcomes (NCO), can improve the methodological robustness of such studies by detecting unmeasured confounding. We found that only one study used NCO. NCO are also called falsification testing because they can identify spurious correlations in observational datasets.

Notably, nearly a third of the studies acknowledged the healthy vaccinee effect.

Despite these limitations, spin was found in over 50% of the studies, most commonly of the type “over-interpretation or inappropriate extrapolation of results”. Mind that we exclusively assessed studies published in high-impact medical journals, where, theoretically, the highest standards of quality and scrutiny are upheld.

Our study is not free from limitations. Only one reviewer handled the data extraction and analysis, which may introduce subjectivity. Also, top-tier journals might encourage authors to over-interpret their findings, though these journals may also scrutinize observational studies more carefully during peer review.

To conclude

Our results suggest that observational studies could improve their use of proper methods for detecting confounding factors and, importantly, that editors and reviewers of high-impact journals should make sure that the language used in these studies clearly reflects both the findings and their limitations.