Advanced Search

Review: A Scoping Review of Real-World Study in Vaccine Evaluation

View author affiliations
  • loading...
  • Funding: Beijing Natural Science Foundation (L212058)
  • [1] Kaplan NM, Sproul LE, Mulcahy WS. Large prospective study of ramipril in patients with hypertension. CARE Investigators. Clin Ther 1993;15(5):810-8. https://pubmed.ncbi.nlm.nih.gov/8269447.
    [2] United States of America Congress. 21st Century cures act. 2016. https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf. [2023-6-11].
    [3] U.S. Food and Drug Administration. Framework for FDA’s real-world evidence program. 2018. https://www.fda.gov/media/120060/download?attachment. [2023-2-23].
    [4] European Medicines Agency. ICH Reflection paper on proposed international harmonisation of real-world evidence terminology and convergence of general principles regarding planning and reporting of studies using real-world data, with a focus on effectiveness of medicines. 2023. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-reflection-paper-proposed-international-harmonisation-real-world-evidence-terminology_en.pdf. [2023-8-19].
    [5] Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018;169(7):467 − 73. https://doi.org/10.7326/M18-0850CrossRef
    [6] Klinger G, Chodick G, Levy I. Long-term immunity to hepatitis B following vaccination in infancy: real-world data analysis. Vaccine 2018;36(17):2288 − 92. https://doi.org/10.1016/j.vaccine.2018.03.028CrossRef
    [7] Wang XL, An ZJ, Huo D, Jia L, Li J, Yang Y, et al. Enterovirus A71 vaccine effectiveness in preventing enterovirus A71 infection among medically-attended hand, foot, and mouth disease cases, Beijing, China. Hum Vaccin Immunother 2019;15(5):1183 − 90. https://doi.org/10.1080/21645515.2019.1581539CrossRef
    [8] Marshall HS, Koehler AP, Wang B, A’Houre M, Gold M, Quinn H, et al. Safety of meningococcal B vaccine (4CMenB) in adolescents in Australia. Vaccine 2020;38(37):5914 − 22. https://doi.org/10.1016/j.vaccine.2020.07.009CrossRef
    [9] Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol 2016;183(8):758 − 64. https://doi.org/10.1093/aje/kwv254CrossRef
    [10] Dickerman BA, Gerlovin H, Madenci AL, Kurgansky KE, Ferolito BR, Figueroa Muñiz MJ, et al. Comparative effectiveness of BNT162b2 and mRNA-1273 vaccines in U. S. veterans. N Engl J Med 2022;386(2):105 − 15. https://doi.org/10.1056/NEJMoa2115463CrossRef
    [11] Al Kaabi N, Oulhaj A, Ganesan S, Al Hosani FI, Najim O, Ibrahim H, et al. Effectiveness of BBIBP-CorV vaccine against severe outcomes of COVID-19 in Abu Dhabi, United Arab Emirates. Nat Commun 2022;13(1):3215. https://doi.org/10.1038/s41467-022-30835-1CrossRef
    [12] Magen O, Waxman JG, Makov-Assif M, Vered R, Dicker D, Hernán MA, et al. Fourth dose of BNT162b2 mRNA Covid-19 vaccine in a nationwide setting. N Engl J Med 2022;386(17):1603 − 14. https://doi.org/10.1056/NEJMoa2201688CrossRef
    [13] National Medical Products Administration of China. Real-world data used to generate real-world evidences (Interim). 2021. https://www.cde.org.cn/main/news/viewInfoCommon/2a1c437ed54e7b838a7e86f4ac21c539. [2023-5-28]. (In Chinese). 
    [14] U.S. Food and Drug Administration. Considerations for the use of real-world data and real-world evidence to support regulatory decision-making for drug and biological products. 2023. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug. [2023-8-31].
  • FIGURE 1.  Flowchart illustrating the process of identification, selection, eligibility, and inclusion of studies for analysis.

    Abbreviation: CNKI=China National Knowledge Infrastructure.

    Note: Wrong publication type: reviews, perspectives, letters to the editor, news articles, and comments. Irrelevant studies: studies on animals, databases, medicine, and treatment.

    FIGURE 2.  Trends and applications of real-world study in vaccine evaluation.

    Abbreviation: VCR=vaccination coverage rates.

    TABLE 1.  Characteristics of real-world studies on vaccines.

    Categories Effectiveness Safety Immunogenicity Impact Economics VCR Hesitancy Other* Total
    Study design 161
    Cohort 45 15 15 3 3 5 86
    Case-control 26 2 28
    Cross-sectional 21 7 1 2 2 33
    Target trial emulation 4 1 5
    Screening method 2 2
    Ecological 2 1 3
    Model 2 2
    PCT 1 1
    CRCT 1 1
    Data source 163
    Administrative database 49 20 2 3 3 2 79
    EMR/EHR 11 4 1 1 17
    Claims database 4 1 3 8
    Registry 5 5
    Survey 13 16 20 1 2 2 54
    Abbreviation: VCR=vaccination coverage rates; PCT=pragmatic randomized clinical trial; CRCT=cluster randomized controlled trial; EMR/EHR=electronic medical records/electronic health records.
    * Other: Willingness to pay, Awareness of vaccine.
    Certain study designs or data sources involve multiple applications.
    Download: CSV

Citation:

通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索
Turn off MathJax
Article Contents

Article Metrics

Article views(737) PDF downloads(7) Cited by()

Share

Related

A Scoping Review of Real-World Study in Vaccine Evaluation

View author affiliations
  • 1. National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases (NITFID), Beijing, China
  • 2. National Immunization Program, Chinese Center for Disease Control and Prevention, Beijing, China.
  • Corresponding author:

    Huaqing Wang, wanghq@chinacdc.cn

  • Funding: Beijing Natural Science Foundation (L212058)
  • Online Date: April 19 2024
    Issue Date: April 19 2024
    doi: 10.46234/ccdcw2024.067
  • Real-world study (RWS) gained prominence starting with a significant investigation on ramipril’s impact on hypertension in 1993 (1). A key advancement for the Food and Drug Administration (FDA) occurred with the enactment of the 21st Century Cures Act in 2016, emphasizing RWS (2). The FDA’s Real-world Evidence Program establishes real-world data (RWD) as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources”, while real-world evidence (RWE) is “clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD” (3). The proliferation of big data, electronic medical records (EMR), electronic health records (EHR), and medical claims data offers vast information resources facilitating RWS. Enhanced access to RWD allows for worldwide monitoring of the impacts of various public health initiatives like vaccination programs. While randomized clinical trials (RCTs) traditionally gauge vaccine efficacy and short-term safety, RWS can assess vaccine performance and safety across larger, more diverse populations and are adept at identifying rare events not easily discernible in RCTs due to their infrequency.

    The FDA established definitions for RWD and RWE in 2018, yet there is a lack of globally standardized definitions for RWD and RWE (4). Given the array of vaccine-preventable conditions and the variety of implementation methods, real-world studies (RWS) on vaccine use draw from a broad range of data sources and employ numerous study designs. Although previous systematic reviews have concentrated on assessing the real-world effectiveness and safety of individual vaccines, some investigations meet the requirements for RWD but do not explicitly incorporate RWD or RWE concepts. Furthermore, there is an absence of an established, overarching definition for RWS, as well as a standardized framework for evaluating vaccines. Consequently, the aims of this study were to: 1) examine the trends and applications of RWS in vaccine evaluation, and 2) describe the study designs and data sources used in RWS concerning vaccine evaluation.

  • We utilized the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines to conduct our scoping review (5), following this method to obtain thorough and relevant results.

  • We conducted a literature search on PubMed, Web of Science, CNKI, and Wanfang Database from December 2016 to July 2023. The search included studies in English or Chinese using the keywords “real-world study”, “real-world research”, “real-world data”, “real-world evidence”, “vaccine”, and “vaccination”.

  • The study encompassed human vaccination research on effectiveness, safety, immunogenicity, impact, health economics, vaccination coverage rates (VCR), vaccine hesitancy, and related factors. Excluded were non-human studies, research on methodology, databases, medicine, and treatment, as well as case reports, reviews, perspectives, letters, news articles, and comments.

  • Two researchers (WL and YH) reviewed titles, abstracts, and full texts of the articles. Discrepancies were resolved by consulting with the principal investigator (WH). Data were gathered utilizing a standardized Excel sheet, then summarized based on publication date, country, authorship, vaccine type, study purpose, design, population demographics, sample size, and data origin. Findings were delineated and juxtaposed by vaccine types, study methodologies, and data resources.

    • The initial search found 792 articles, with 243 duplicates removed. After excluding 271 articles based on title and abstract review, 278 articles underwent full-text screening. Additionally, 12 articles were identified through reference list review and manual search. Ultimately, 154 articles were included in the synthesis (Figure 1).

      Figure 1. 

      Flowchart illustrating the process of identification, selection, eligibility, and inclusion of studies for analysis.

      Abbreviation: CNKI=China National Knowledge Infrastructure.

      Note: Wrong publication type: reviews, perspectives, letters to the editor, news articles, and comments. Irrelevant studies: studies on animals, databases, medicine, and treatment.

    • The figure in Figure 2A illustrates the publication trend of studies that mentioned keywords related to RWD/RWE/RWS from December 2016 to July 2023. The number of publications notably increased annually, peaking at 75 studies in 2022. Among the 154 articles, 111 were from high-income countries or regions, according to the World Bank Development Indicators (https://www.worldbank.org). The most researched vaccine, with 111 studies, was the coronavirus disease 2019 (COVID-19) vaccine, followed by the HPV vaccine, the influenza vaccine, and the pneumonia vaccine, each with 9 studies (Figure 2B). Figure 2C shows that the studies covered 34 countries, with China leading in the number of vaccine-related real-world studies (33 studies), followed by the United States (28 studies). The primary uses of real-world studies were to assess effectiveness (81 studies), safety (40 studies), and immunogenicity (22 studies). Additionally, two studies on the EV71 vaccine were exclusively conducted in China. Real-world studies were also applied to evaluate vaccine coverage rates, health economics, impact, vaccine hesitancy, willingness to pay, and awareness of vaccine evaluations.

      Figure 2. 

      Trends and applications of real-world study in vaccine evaluation.

      Abbreviation: VCR=vaccination coverage rates.
    • The most common study designs in the literature reviewed were cohort studies (86 studies) and case-control studies (28 studies). Cohort studies were predominantly utilized for assessing effectiveness (45 studies) than safety (15 studies) or immunogenicity (15 studies), while case-control studies, particularly the test-negative design (16 out of 26 studies), were commonly employed for evaluating vaccine effectiveness. Cross-sectional studies were frequently utilized for assessing vaccine safety (21 studies). Additionally, other study designs such as target trial emulation studies, screening methods, ecological studies, and pragmatic randomized clinical trial (PCT) were employed for vaccine effectiveness evaluation. Furthermore, two studies utilized modeling for economic evaluations of vaccines (Table 1).

      Categories Effectiveness Safety Immunogenicity Impact Economics VCR Hesitancy Other* Total
      Study design 161
      Cohort 45 15 15 3 3 5 86
      Case-control 26 2 28
      Cross-sectional 21 7 1 2 2 33
      Target trial emulation 4 1 5
      Screening method 2 2
      Ecological 2 1 3
      Model 2 2
      PCT 1 1
      CRCT 1 1
      Data source 163
      Administrative database 49 20 2 3 3 2 79
      EMR/EHR 11 4 1 1 17
      Claims database 4 1 3 8
      Registry 5 5
      Survey 13 16 20 1 2 2 54
      Abbreviation: VCR=vaccination coverage rates; PCT=pragmatic randomized clinical trial; CRCT=cluster randomized controlled trial; EMR/EHR=electronic medical records/electronic health records.
      * Other: Willingness to pay, Awareness of vaccine.
      Certain study designs or data sources involve multiple applications.

      Table 1.  Characteristics of real-world studies on vaccines.

    • Table 1 demonstrates that administrative databases were predominantly utilized in 49 studies for assessing vaccine effectiveness in real-world scenarios. For evaluating immunogenicity, survey data from real-world settings were commonly employed in 20 studies. In terms of vaccine safety assessment, researchers frequently examined large-scale datasets from administrative databases (20 studies) and survey databases using RWD tailored for specific research goals (16 studies). Most parameters for economic evaluations of vaccines were sourced from administrative databases, medical claims databases with cost records, and electronic medical records. Among the reviewed studies, only three used administrative databases for the economic assessment of vaccines. In the evaluation of HPV vaccine effectiveness in real-world studies, registry databases were the primary information source for HPV diseases in 4 out of 9 studies. Utilization of claims databases in real-world vaccine evaluations was infrequent, constituting only 4.9% of the total, whereas EMR/EHR accounted for 10.4%.

    • Our analysis revealed a substantial rise in RWSs on vaccines post the FDA’s issuance of a regulatory framework in 2018, indicating wider adoption of RWS. The surge may also be linked to the integration of big data in the context of COVID-19, which since 2020 has further stimulated researcher interest and regulatory bodies’ willingness to embrace RWSs. These studies primarily originated from affluent regions with possibly better access to reliable data, enabling more robust real-world research. The HPV vaccine received considerable attention in this review due to challenges in endpoint event identification in clinical trials caused by low cervical cancer incidence and long latency periods. Therefore, RWSs assessing the HPV vaccine are valuable. The relationship between serological markers and the protective efficacy of the hepatitis B vaccine is well-documented (6). Consequently, the vaccine’s effectiveness can be determined via serological assessments, reducing the reliance on real-world studies for efficacy evaluation. RWSs regarding influenza and pneumonia vaccines in the elderly mainly concentrate on health economics, driven by the significant economic impact of respiratory infectious diseases in this age group. Studies on childhood immunization program vaccines like diphtheria, tetanus, pertussis, hepatitis B, and polio predominantly focus on vaccine coverage and adherence. Research on novel vaccines like DTaP2-IPV-HB-Hib and PHiD-CV centers on real-world safety evaluation.

      Our review of the literature revealed that RWSs commonly employ observational designs, in line with the inherent nature of such studies. Cohort and case-control studies are predominant, along with the increasing popularity of the TND case-control design and screening method design (7). The stepped wedge design involves the gradual implementation of a vaccination program to participants over several time periods, enabling a sequential assessment of its effects. Marshall HS utilized this approach to evaluate the safety of the 4CMenB vaccine in adolescents (8). A framework termed “target trial emulation study” has been developed to assist in the design and analysis of observational studies using RWD (9). Additionally, the target trial simulation design demonstrates promising potential in assessing the real-world effectiveness of the COVID-19 vaccine (10).

      The assessment of vaccine effectiveness relies on two essential factors: the exposure and outcome variables. Therefore, the choice of a data source is crucial for real-world vaccine research. The electronic health database managed by the Abu Dhabi Health Services Company (SEHA) facilitates the provision of COVID-19 vaccinations to all residents of the United Arab Emirates, oversees the management of vaccine-related complications, and administers all COVID-19 designated hospitals within Abu Dhabi (11). This comprehensive database contains essential variables and a range of covariates crucial for the assessment of vaccine effectiveness. Additionally, Clalit Health Services (CHS) represents the foremost integrated payer-provider healthcare organization in Israel (12). The data repositories contain an extensive array of information such as demographic details, diagnostic results, pharmacological data, laboratory test outcomes, procedural records, imaging studies, and hospital admission information. These countries, with their advanced systems, can link exposure and outcome variables to swiftly and effectively analyze vaccine efficacy in real-world scenarios.

      RWS is a broad term that covers various epidemiological designs, primarily observational and non-interventional studies. Although not included in the FDA guidelines, the concept originates from the National Medical Products Administration (NMPA) guidelines (13). The FDA offers detailed procedures for submitting real-world data to inform regulatory decisions on drugs and biological products (14). Recommendations for RWS include engaging with the FDA beforehand, assessing data sources, and providing study outcomes. For future real-world vaccine research, it is vital to evaluate the suitability of data sources with extensive exposure variables, outcome variables, and covariates. Reliable data sources and proper analysis are essential for generating high-quality evidence. Encouraging low-income and lower-middle-income countries to conduct vaccine RWS can be cost-effective. Additionally, enhancing basic databases and training scientific researchers in these regions is crucial.

      This study is subject to some limitations. First, our research scope was constrained due to the 21st Century Cures Act implemented by the FDA in December 2016. Second, we might have overlooked certain real-world studies that did not explicitly refer to the keywords RWS/RWD/RWE in their text, although we did include numerous nationwide vaccination studies through manual searches. Third, we did not evaluate the quality of the included literature due to the broad definition and absence of specific criteria for RWS. As the body of literature on vaccine real-world studies expands and the standards for RWS become more consistent, we will update our review accordingly.

    • RWS have gained significant popularity in vaccine evaluation since 2016 and are particularly prominent in high-income and upper-middle-income countries due to the increased availability and accessibility of RWD. These studies typically rely on observational study designs such as cohort, case-control, and cross-sectional studies, utilizing data from sources like administrative databases, EMR/EHR, and claims databases. The field of vaccine RWS is continuously evolving, and this review aims to offer guidance for researchers interested in conducting such studies.

    • No conflicts of interest.

    • Dr. Lance Rodewald from the Chinese Center for Disease Control and Prevention for his valuable comments and assistance with editing the English language.

Reference (14)

Citation:

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return