Methods and Applications: Comparison of Statistical Signal Detection Methods in Adverse Events Following Immunization — China, 2011–2015

The study utilized spontaneous AEFI reports from the national AEFI information system (8) from 2011 to 2015. Data preparation involved several steps, including the removal of confidential information (9) and duplicate reports, as well as reports without valid AEFI clinical diagnosis that would not contribute to the vaccine-AEFI pair. Additionally, reports with multiple AEFI clinical diagnoses or suspected vaccines were separated into multiple individual reports, each with a unique AEFI clinical diagnosis and suspected vaccine (10). The AEFI clinical diagnoses were coded using the Medical Dictionary for Regulatory Activities (MedDRA) dictionary, version 24.0 (11). All vaccines and AEFI clinical diagnoses were included in the analysis.

Statistical signal detection was performed by analyzing the reporting rates of specific adverse events associated with specific vaccines. Four commonly used methods for analyzing disproportionality were applied: proportional reporting ratio (PRR) (4), reporting odds ratio (ROR) (2–3), Bayesian confidence propagation neural network (BCPNN) (1,3), and multi-item gamma Poisson shrinker (MGPS) (5). PRR and ROR are frequentist methods, while BCPNN and MGPS are Bayesian methods. The computation techniques for each method can be found in the supplementary table S1 and relevant publications (1–5).

The disproportionality analysis is based on a 2×2 table (Table 1). In this table, cell a represents the number of reports containing both the vaccine of interest and the AEFI of interest. Cell b represents the number of reports containing the vaccine of interest with AEFIs other than the AEFI of interest. Cell c represents the number of reports containing the AEFI of interest with vaccines other than the vaccine of interest. Cell d represents the number of reports containing AEFIs and vaccines other than the ones of interest.

Table 2 presents the signal detection methods and the threshold values to be assessed. Each signal detection method was evaluated using up to three signal threshold values. Vaccine-AEFI combinations with statistical values exceeding the threshold values were deemed as positive signals.

We calculated the operating characteristics (sensitivity and specificity) of each signal detection algorithm to classify each vaccine-AEFI combination as either a signal or a non-signal. We used published reference standards as our gold standards (14).

Two sets of reference standards based on the global manual on the surveillance of adverse events following immunization by the World Health Organization (WHO) (15) and safety signals from previous studies (16–20) were created (Table 3). Sensitivity and specificity were determined and presented in Table 4.

The baseline characteristics of AEFI data were analyzed to assess disproportionality. We examined the number of signals detected by each signal detection algorithm and calculated the cumulative distribution of signals for each algorithm. The distribution of signals was determined by dividing the number of signals with a specific number of reports by the total number of signals detected by each algorithm. Sensitivity and specificity for each detection method were also determined as described previously. Analyses were conducted using R software (version 4.3.1, The R Foundation for Statistical Computing, Lucent Technologies, Auckland, New Zealand) and the PhViD package (version 1.0.8, The R Foundation for Statistical Computing, Lucent Technologies, Auckland, New Zealand).

The original dataset consisted of 587,149 reports documenting AEFI. After removing 762 duplicate records and 15,844 records without a valid AEFI clinical diagnosis, these unique records were further separated into individual reports, resulting in 871,647 records that contained both an AEFI and a suspected vaccine. After removing 485 records without a valid vaccine name and 132 records with duplicate AEFI-vaccine pairings, the final analyzable data set consisted of 871,030 records. This data set included 41 different vaccines, 771 specific AEFI events, and 3,893 unique combinations of vaccines and AEFI events.

Table 5 presents the number of signals detected by each signal detection algorithm. PRR1 and ROR1 identified the highest number of signals, while MPGS1 identified the lowest number of signals. MPGS methods detected fewer signals compared to PRR, ROR, and BCPNN.

PRR3, PRR1, ROR1, and MGPS3, exhibited the highest number of signals detected when the number of reports was five or fewer. Conversely, MGPS1 did not detect any signals when the number of reports was 5 or fewer. Among the signals detected by PRR3, 38.9% had a number of reports equal to or less than 5, while for MGPS3 this percentage was 33.7%. On the other hand, PRR1, PRR2, ROR1, ROR2, and BCPNN identified the greatest number of signals when the number of reports exceeded five.

Table 6 presents the sensitivity and specificity values for each signal detection algorithm using two reference standards: reference standard one for common adverse events and reference standard two for rare adverse events. Based on reference standard one, the algorithms PRR1 and PRR2 demonstrated the highest sensitivity at 38.2%, closely followed by ROR1 and ROR2 at 37.3%. MGPS1 exhibited the lowest sensitivity at 0.9%. On the other hand, MGPS1 exhibited the highest specificity at 96.7%, followed by MGPS2 at 96.2%. MGPS sensitivity was significantly lower than that of PRR, ROR, and BCPNN, while its specificity was higher than that of PRR, ROR, and BCPNN.

Based on reference standard 2, the diagnostic tests PRR1, PRR2, ROR1, ROR2, and BCPNN exhibited the highest sensitivity (73.3%), while PRR3, MGPS1, MGPS2, and MGPS3 showed a lower sensitivity (53.3%). Among the tests, MGPS1 demonstrated the highest specificity (96.9%). Although MGPS had lower sensitivity compared to PRR, ROR, and BCPNN, its specificity was higher than those three tests.

Our study aimed to assess the main features of commonly employed algorithms for detecting signals in spontaneous reporting datasets. Specifically, we examined the performance of four signal detection methods in identifying vaccine safety signals within AEFI data collected in China from 2011 to 2015. To do this, we analyzed the data using different thresholds of signal positivity. In order to evaluate the accuracy of the algorithms, we compared their results to reference standards from published scientific analyses, which were considered as the gold standard. From these comparisons, we calculated the sensitivities and specificities of each algorithm.

The number of signals detected varied significantly among the algorithms, which aligns with the findings of Kubota and colleagues (21). The PRR and ROR methods identified the highest number of safety signals, while MGPS method identified the fewest signals. Specifically, PRR1 found 475 more signals than MGPS1. The distribution of signals differed significantly among algorithms when the number of reports was five or fewer, but not when the number exceeded five. PRR1 and ROR1 demonstrated similar performance in signal identification, as did PRR2 and ROR2. The variation in the number of signals identified by PRR1, ROR1, and PRR3 was related to the variability in signals for more commonly reported events (i.e., those with more than five reports). On the other hand, the variability in PRR1 (ROR1) compared to PRR2, ROR2, BCPNN, and MGPS was due to differences in signal identification when the number of reports was fewer than five.

The signal-finding algorithms showed considerable variation in sensitivity and specificity. PRR1 and PRR2 demonstrated the highest sensitivity, followed by ROR1, ROR2, and BCPNN, which were also sensitive but to a lesser extent. However, MGPS1 exhibited the highest specificity, but had the lowest sensitivity. Further research is needed to investigate the reasons behind this finding. When using the reference standard for rare side effects, PRR1, PRR2, ROR1, ROR2, and BCPNN were more sensitive than PRR3, MGPS1, MGPS2, and MGPS3. MGPS1 was found to be the most specific. In summary, our study indicates that PRR, ROR, and BCPNN are more sensitive than MGPS for detecting safety signals, while MGPS is more specific. These findings align with previous studies (12,21–22).

The initial analysis of our data highlighted the significance of data preparation. The standardized processing of data is crucial for ensuring consistent signal detection analyses (9). In order to perform signal detection analyses, it is necessary to preprocess spontaneous AEFI reports. This involves eliminating duplicate and invalid records, as well as separating AEFI-vaccine pairs in reports that contain multiple pairs.

Variations in the number of reports, as well as sensitivity and specificity, can be attributed to several factors. First, computation methods differ when dealing with a small number of reports (21–22). Bayesian shrinkage calculations used by BCPNN and MGPS result in more stable but conservative results compared to PRR and ROR. Second, variations arise from the different thresholds selected (21). Future research should systematically evaluate the impact of threshold values on sensitivity and specificity. Therefore, the variations observed in the number of reports and sensitivity and specificity highlight the importance of selecting appropriate signal detection methods and threshold values based on specific use case scenarios.

To the best of our knowledge, this is the first study to investigate the reference standard for performance evaluation. We systematically evaluated the variation in the number of reports, as well as the sensitivity and specificity of the signal detection method, using the AEFI database in China. Our findings can offer valuable insights for the selection of signal detection methods and corresponding threshold values for the routine signal detection system in China's AEFI data. It is important to strike a balance between sensitivity and specificity when choosing signal detection methods and threshold values, while considering factors such as the ability to investigate detected signals (12), the severity of the AEFI under investigation, and the potential impact on public health if a true safety signal is missed.

Based on our study and an extensive review of relevant scientific literature, we propose different approaches for the detection of AEFI, depending on the severity and prevalence of the events, as well as the type of vaccine. For common or mild AEFIs, we recommend utilizing more specific signal detection methods such as the BCPNN or the MGPS, along with more stringent thresholds such as PRR2 or ROR2. These methods and thresholds can effectively reduce the number of false positives. In contrast, for rare or severe AEFIs, or for new licensed vaccines, we advise using more sensitive signal detection methods like the PRR or the ROR, along with less stringent thresholds. These approaches are designed to minimize the risk of missing true signals.

This study has some limitations. First, there is no universally accepted gold standard for evaluating the performance of signal detection (12). In this study, we used reference standards based on the World Health Organization’s global manual on surveillance of adverse events following immunization (15) and safety signals identified in previous studies (16–20) as the gold standards. Second, AEFI data are collected through a passive surveillance system, and the quality of the reports may affect the detection of signals. Additionally, AEFI data is subject to known limitations, such as under-reporting, selective reporting, or over-reporting (23). Therefore, safety signals identified solely based on AEFI data in this study cannot determine causality and should be interpreted cautiously.

In our study, we conducted a comprehensive analysis of the number of signals detected and the performance of various methods for vaccine safety signal detection. The analysis was based on data from a passive, spontaneously reported database of AEFI. We recommend further research to evaluate the specific characteristics of the identified signals and assess the impact of different thresholds on signal detection accuracy. This additional research will provide valuable insights for enhancing the accuracy of vaccine safety signal detection in the context of vaccines used in China.

All authors declare no competing interests.

Dr. Lance Rodewald for his review and valuable feedback.