Preplanned Studies: Comparison of Three Influenza Surveillance Data Sources for Timely Detection of Epidemic Onset — Chengdu City, Sichuan Province and Beijing Municipality, China, 2017–2023

The influenza-like illness (ILI) surveillance system in China plays a crucial role in the syndromic surveillance infrastructure, serving as an essential tool for the early warning of influenza epidemics (1–2). Despite its historical significance, the effectiveness of this system is often compromised during periods when multiple respiratory pathogens are present, which may obscure the specific activities of the influenza virus (3–4).

In addition to the ILI surveillance system, China employs two other primary systems for influenza monitoring: the nationwide Notifiable Infectious Diseases Reporting Information System (NIDRIS) and the Virological Surveillance System (5). A notable challenge within these systems is the potential for undiagnosed or misdiagnosed influenza cases, particularly during non-epidemic periods and in the absence of laboratory confirmation. Nevertheless, NIDRIS requires the nationwide reporting of all confirmed influenza cases, providing a comprehensive dataset for analysis. On the other hand, the Virological Surveillance System, while offering highly accurate tracking of viral activities, is hampered by its time-intensive processes and lack of immediate data availability.

This study aimed to compare the effectiveness of these three surveillance datasets in detecting the timely onset of influenza epidemics. We employed the exponentially weighted moving average (EWMA) and a modified version of the Cumulative Sum (modified-CUSUM) to assess the performance of each dataset.

The findings from our evaluation suggested that the data derived from NIDRIS demonstrated superior performance in the early detection of influenza epidemics, particularly in scenarios involving the co-circulation of various respiratory pathogens. Based on these results, we recommend that public health practitioners in China prioritize the use of NIDRIS-reported case data to enhance the accuracy and timeliness of influenza epidemic detection. This approach could significantly improve the responsiveness and effectiveness of public health interventions against influenza outbreaks.

We analyzed three datasets covering the period from 2017 to 2023 to facilitate early detection of influenza epidemics in Chengdu City, Sichuan Province and Beijing Municipality, cities emblematic of the southern and northern China, respectively. The first dataset comprised the daily count of reported influenza cases from the NIDRIS. The second dataset included daily reports of ILI cases from an ILI surveillance network, drawing data from 144 sentinel hospitals in Beijing and 11 in Chengdu. Finally, the third dataset was constructed by multiplying weekly ILI reports by the influenza positivity rate (PR) (ILIs*PR), as determined from the Virological Surveillance System active in Beijing and Chengdu.

We utilized two widely accepted outbreak detection algorithms, the EWMA and the modified-CUSUM, to facilitate early warnings of influenza epidemics (Supplementary Material). The EWMA algorithm is designed for analyzing time series data by applying exponentially decreasing weights to older observations. An alert is triggered when the moving average surpasses a predefined threshold. Similarly, the modified-CUSUM algorithm monitors the cumulative difference between observed and expected values, issuing an alert when this deviation exceeds its respective threshold (6). The benchmark for these analyses was the influenza PR, which we deemed the gold standard. We defined the influenza season as spanning from the 27th week of one year to the 26th week of the following year. The threshold for an alert was set when the PR exceeded 40% of its peak value (7).

To assess model performance, we employed evaluation criteria including sensitivity, specificity, and the Youden Index to optimize parameters. Sensitivity measures the model’s ability to correctly identify true positive instances when an alert is warranted, calculated as the ratio of true positives to all instances that should trigger an alarm. Specificity, in contrast, evaluates the model’s accuracy in identifying true negative instances where no alert is needed, computed as the ratio of true negatives to all instances without an alarm (8). The Youden Index is derived by summing the sensitivity and specificity and then subtracting one, providing a single statistic that balances both metrics.

Data for each season were partitioned into training and test sets to facilitate early warnings. For the 2021–2022 season, for instance, all preceding data was utilized as the training set, while data from the 2021–2022 season comprised the test set. This approach was uniformly adopted for subsequent seasons. We optimized parameter values using the training sets, employing evaluation metrics such as sensitivity, specificity, and the Youden Index to determine effectiveness. Subsequently, the model demonstrating the highest performance with these optimized parameters was selected for early signal detection in the test sets.

In the case of the ILIs*PR dataset, the PR for the most recent week was only available on the final day of that week. Consequently, we employed a Long Short-Term Memory (LSTM) model (9) to forecast the latest PR in the test sets. The LSTM network, a variant of recurrent neural network, is particularly adept at longitudinal data prediction. It effectively retains critical data over time, which is essential for capturing long-term dependencies that enhance the accuracy of both current and future predictions.

Prediction accuracy was quantified using the mean squared error (MSE). All analyses were conducted using R (Version 4.3.1, R Foundation for Statistical Computing, Vienna, Austria), except for the LSTM model which was implemented using Python (Version 3.8.13, Python Software Foundation, Beaverton, USA).

Surveillance data from Beijing revealed a consistent annual peak in ILIs each January up until 2020. However, the emergence of the coronavirus disease 2019 (COVID-19) pandemic and the simultaneous circulation of multiple respiratory pathogens disrupted this established seasonal trend. Notably, an atypical peak occurred in 2023, occurring despite limited influenza virus activity (Figure 1). In contrast, the combination of reported ILI cases and virus PRs provided a more accurate reflection of influenza virus activity when the virus was actively circulating.

Figure 1. 

The 7-day moving average of the daily number of new cases in the three datasets and the weekly influenza positivity rate, 2013–2023. (A) In Beijing Municipality; (B) In Chengdu City, Sichuan Province.

Note: This figure illustrates the trends in the 7-day moving average of reported influenza cases, ILI cases, ILI cases adjusted for influenza positivity rate, and the weekly influenza positivity rate. The light blue shaded area represents the trend in ILI cases, while the dark blue shaded area indicates the trend in ILI cases adjusted by the influenza positivity rate. The pink shaded area depicts the trend in reported cases derived from the nationwide NIDRIS. The dotted line displays the weekly influenza positivity rate. Reported cases indicate the number of reported cases from the nationwide NIDRIS; ILIs*PR indicates the number of cases of influenza-like illness multiplied by the positivity rate of the influenza virus.

Abbreviation: PR=positivity rate; NIDRIS=Notifiable Infectious Diseases Reporting Information System; ILI=influenza-like illness.

During periods of concurrent circulation of multiple respiratory pathogens, both reported cases and ILIs*PR demonstrated superior performance compared to ILIs alone. Specifically, across the last three seasons in Beijing, reported cases and ILIs*PR consistently exhibited good sensitivity, with reported cases also displaying strong specificity. The composite Youden Index for these three seasons indicated that reported cases achieved the best overall performance (0.905), followed by ILIs*PR (0.844), and ILIs (0.566). In Chengdu, reported cases consistently exhibited better sensitivity and specificity. Conversely, the performance of ILIs and ILIs*PR was found to be suboptimal, achieving overall Youden Index scores of only 0.49 and 0.74, respectively, as shown in Table 1.

The somewhat reduced specificity observed in reported cases and ILIs*PR can be primarily attributed to the prompt issuance of early warnings, which, in turn, led to a rise in the number of false positives. In scenarios where the prioritization of early warnings is critical, such performance might be deemed beneficial. Notably, an increase in false positives for ILIs was particularly evident during periods of low influenza activity, underscoring that in the presence of co-circulating pathogens, this dataset might not represent the optimal choice (Figure 2).

Figure 2. 

Warning results of the optimal model across three datasets for the early warning of influenza epidemic in Beijing Municipality and Chengdu City, Sichuan Province, 2021–2024. (A) Reported cases in Beijing; (B) Reported cases in Chengdu; (C) ILIs in Beijing; (D) ILIs in Chengdu; (E) ILIs*PR in Beijing; (F) ILIs*PR in Chengdu.

Note: This figure shows the performance of the three datasets for the early warning of influenza epidemics. For season 2021–2022, all data before 2021–2022 served as the training set, with data for 2021–2022 as the test set. This approach was consistently followed for subsequent seasons. The gray line represents the 7-day moving average number of corresponding cases. The red dots indicate TP, where the model accurately predicted an impending influenza epidemic; the orange dots represent FP, where the model incorrectly predicted an epidemic; the green dots signify TN, where the model correctly predicted the absence of an epidemic; and the purple dots represent FN, where the model failed to predict an actual influenza epidemic. Reported cases indicate the number of reported cases from the nationwide NIDRIS; ILIs*PR indicates the number of cases of influenza-like illness multiplied by the positivity rate of the influenza virus.

Abbreviation: PR=positivity rate; NIDRIS=Notifiable Infectious Diseases Reporting Information System; ILI=influenza-like illness; TP=true positive; FP=false positive; TN=true negative; FN=false negative.