Review: A Scoping Review of Real-World Study in Vaccine Evaluation

The initial search found 792 articles, with 243 duplicates removed. After excluding 271 articles based on title and abstract review, 278 articles underwent full-text screening. Additionally, 12 articles were identified through reference list review and manual search. Ultimately, 154 articles were included in the synthesis (Figure 1).

Figure 1. 

Flowchart illustrating the process of identification, selection, eligibility, and inclusion of studies for analysis.

Abbreviation: CNKI=China National Knowledge Infrastructure.

Note: Wrong publication type: reviews, perspectives, letters to the editor, news articles, and comments. Irrelevant studies: studies on animals, databases, medicine, and treatment.

The figure in Figure 2A illustrates the publication trend of studies that mentioned keywords related to RWD/RWE/RWS from December 2016 to July 2023. The number of publications notably increased annually, peaking at 75 studies in 2022. Among the 154 articles, 111 were from high-income countries or regions, according to the World Bank Development Indicators (https://www.worldbank.org). The most researched vaccine, with 111 studies, was the coronavirus disease 2019 (COVID-19) vaccine, followed by the HPV vaccine, the influenza vaccine, and the pneumonia vaccine, each with 9 studies (Figure 2B). Figure 2C shows that the studies covered 34 countries, with China leading in the number of vaccine-related real-world studies (33 studies), followed by the United States (28 studies). The primary uses of real-world studies were to assess effectiveness (81 studies), safety (40 studies), and immunogenicity (22 studies). Additionally, two studies on the EV71 vaccine were exclusively conducted in China. Real-world studies were also applied to evaluate vaccine coverage rates, health economics, impact, vaccine hesitancy, willingness to pay, and awareness of vaccine evaluations.

Figure 2. 

Trends and applications of real-world study in vaccine evaluation.

Abbreviation: VCR=vaccination coverage rates.

The most common study designs in the literature reviewed were cohort studies (86 studies) and case-control studies (28 studies). Cohort studies were predominantly utilized for assessing effectiveness (45 studies) than safety (15 studies) or immunogenicity (15 studies), while case-control studies, particularly the test-negative design (16 out of 26 studies), were commonly employed for evaluating vaccine effectiveness. Cross-sectional studies were frequently utilized for assessing vaccine safety (21 studies). Additionally, other study designs such as target trial emulation studies, screening methods, ecological studies, and pragmatic randomized clinical trial (PCT) were employed for vaccine effectiveness evaluation. Furthermore, two studies utilized modeling for economic evaluations of vaccines (Table 1).

Table 1 demonstrates that administrative databases were predominantly utilized in 49 studies for assessing vaccine effectiveness in real-world scenarios. For evaluating immunogenicity, survey data from real-world settings were commonly employed in 20 studies. In terms of vaccine safety assessment, researchers frequently examined large-scale datasets from administrative databases (20 studies) and survey databases using RWD tailored for specific research goals (16 studies). Most parameters for economic evaluations of vaccines were sourced from administrative databases, medical claims databases with cost records, and electronic medical records. Among the reviewed studies, only three used administrative databases for the economic assessment of vaccines. In the evaluation of HPV vaccine effectiveness in real-world studies, registry databases were the primary information source for HPV diseases in 4 out of 9 studies. Utilization of claims databases in real-world vaccine evaluations was infrequent, constituting only 4.9% of the total, whereas EMR/EHR accounted for 10.4%.

Our analysis revealed a substantial rise in RWSs on vaccines post the FDA’s issuance of a regulatory framework in 2018, indicating wider adoption of RWS. The surge may also be linked to the integration of big data in the context of COVID-19, which since 2020 has further stimulated researcher interest and regulatory bodies’ willingness to embrace RWSs. These studies primarily originated from affluent regions with possibly better access to reliable data, enabling more robust real-world research. The HPV vaccine received considerable attention in this review due to challenges in endpoint event identification in clinical trials caused by low cervical cancer incidence and long latency periods. Therefore, RWSs assessing the HPV vaccine are valuable. The relationship between serological markers and the protective efficacy of the hepatitis B vaccine is well-documented (6). Consequently, the vaccine’s effectiveness can be determined via serological assessments, reducing the reliance on real-world studies for efficacy evaluation. RWSs regarding influenza and pneumonia vaccines in the elderly mainly concentrate on health economics, driven by the significant economic impact of respiratory infectious diseases in this age group. Studies on childhood immunization program vaccines like diphtheria, tetanus, pertussis, hepatitis B, and polio predominantly focus on vaccine coverage and adherence. Research on novel vaccines like DTaP2-IPV-HB-Hib and PHiD-CV centers on real-world safety evaluation.

Our review of the literature revealed that RWSs commonly employ observational designs, in line with the inherent nature of such studies. Cohort and case-control studies are predominant, along with the increasing popularity of the TND case-control design and screening method design (7). The stepped wedge design involves the gradual implementation of a vaccination program to participants over several time periods, enabling a sequential assessment of its effects. Marshall HS utilized this approach to evaluate the safety of the 4CMenB vaccine in adolescents (8). A framework termed “target trial emulation study” has been developed to assist in the design and analysis of observational studies using RWD (9). Additionally, the target trial simulation design demonstrates promising potential in assessing the real-world effectiveness of the COVID-19 vaccine (10).

The assessment of vaccine effectiveness relies on two essential factors: the exposure and outcome variables. Therefore, the choice of a data source is crucial for real-world vaccine research. The electronic health database managed by the Abu Dhabi Health Services Company (SEHA) facilitates the provision of COVID-19 vaccinations to all residents of the United Arab Emirates, oversees the management of vaccine-related complications, and administers all COVID-19 designated hospitals within Abu Dhabi (11). This comprehensive database contains essential variables and a range of covariates crucial for the assessment of vaccine effectiveness. Additionally, Clalit Health Services (CHS) represents the foremost integrated payer-provider healthcare organization in Israel (12). The data repositories contain an extensive array of information such as demographic details, diagnostic results, pharmacological data, laboratory test outcomes, procedural records, imaging studies, and hospital admission information. These countries, with their advanced systems, can link exposure and outcome variables to swiftly and effectively analyze vaccine efficacy in real-world scenarios.

RWS is a broad term that covers various epidemiological designs, primarily observational and non-interventional studies. Although not included in the FDA guidelines, the concept originates from the National Medical Products Administration (NMPA) guidelines (13). The FDA offers detailed procedures for submitting real-world data to inform regulatory decisions on drugs and biological products (14). Recommendations for RWS include engaging with the FDA beforehand, assessing data sources, and providing study outcomes. For future real-world vaccine research, it is vital to evaluate the suitability of data sources with extensive exposure variables, outcome variables, and covariates. Reliable data sources and proper analysis are essential for generating high-quality evidence. Encouraging low-income and lower-middle-income countries to conduct vaccine RWS can be cost-effective. Additionally, enhancing basic databases and training scientific researchers in these regions is crucial.

This study is subject to some limitations. First, our research scope was constrained due to the 21st Century Cures Act implemented by the FDA in December 2016. Second, we might have overlooked certain real-world studies that did not explicitly refer to the keywords RWS/RWD/RWE in their text, although we did include numerous nationwide vaccination studies through manual searches. Third, we did not evaluate the quality of the included literature due to the broad definition and absence of specific criteria for RWS. As the body of literature on vaccine real-world studies expands and the standards for RWS become more consistent, we will update our review accordingly.

RWS have gained significant popularity in vaccine evaluation since 2016 and are particularly prominent in high-income and upper-middle-income countries due to the increased availability and accessibility of RWD. These studies typically rely on observational study designs such as cohort, case-control, and cross-sectional studies, utilizing data from sources like administrative databases, EMR/EHR, and claims databases. The field of vaccine RWS is continuously evolving, and this review aims to offer guidance for researchers interested in conducting such studies.

No conflicts of interest.

Dr. Lance Rodewald from the Chinese Center for Disease Control and Prevention for his valuable comments and assistance with editing the English language.