Methods and Applications: Comparison Between Threshold Method and Artificial Intelligence Approaches for Early Warning of Respiratory Infectious Diseases — Weifang City, Shandong Province, China, 2020–2023

In this study, “infectious disease early warning” refers to identifying outbreak signs before or during its initial phases through the analysis of infectious disease data from various surveillance sources. Data from January 2020 to May 2023, including influenza-like illness (ILI) statistics, the Baidu index, and clinical data, were analyzed. All methodologies used in this study relied on a uniform and collective data origin.

ILI data from the National Influenza Surveillance Network in China were segmented into China Northern ILI%, Shandong Province ILI%, and Weifang City ILI%. The ILI definition matched the criteria established by the Department of Disease Control and Prevention of the National Health Commission of China, identifying ILI as fever (body temperature ≥38 °C) with cough or sore throat, as referenced (8). ILI% represented the ratio of cases among individuals seeking medical care.

The Baidu index, sourced from the publicly accessible Baidu index website, represents the aggregate search frequency of specified keywords on Baidu web pages, with each keyword assigned a particular weight. In the context of “treatment,” the index takes into account search terms including “Flu Treatment,” “Cold Medicine,” “Antipyretic,” “Lianhuaqingwen,” “What is the most effective flu medicine,” “Liuganwan,” “Ganmaoqingre,” “Banlangen,” “Baijiahei,” “Oseltamivir,” and “Tamiflu.” Conversely, the Baidu index for “non-fever symptoms” comprises phrases such as “Fever,” “Cough,” “Pharyngalgia,” “Sore throat,” “Runny nose,” “Pneumonia,” “Chest tightness,” “Symptoms of influenza,” “Sneezing,” “Lacking in strength,” and “Muscle soreness” (9).

Clinical data from primary and tertiary medical institutions in Weifang City included 21,584,148 chief complaints, 23,128,256 initial diagnoses, 39,486,100 pharmaceutical sales, and 426,171 instances of emergency call data (120). This study focused on respiratory symptoms data, incorporating chief complaints, diagnoses, pharmaceutical sales, and emergency call data (120) with proportional representation.

This study performed a comparative analysis of three early warning methods used in Weifang City, China. The first method improves upon the conventional threshold approach by autonomously determining an optimal threshold, enhancing its practical usability. The second method utilizes supervised machine learning models, whereas the third employs unsupervised machine learning models. Specific details of these models are provided below.

The ADTM method integrates automatic adjustments into conventional fixed-threshold methods to improve sensitivity and specificity. It consists of five comprehensive phases.

Phase 1. Modeling and parameter setting: Establish models for three distinct scenarios: the beginning of an epidemic season, sudden increases in case numbers, and outliers surpassing historical levels. Each scenario had specific thresholds set through various techniques (Table 1). A total of 1,620 thresholds were determined based on the three warning signal scenarios and different criteria (Figure 1). This process aimed to ensure the model’s accuracy in accommodating the dynamic and changing patterns in epidemiological data.

Figure 1. 

Mapping of scenarios and criteria for infectious disease early warning thresholds.

Figure 1 illustrates the criteria for activating alerts in various scenarios. In the epidemic season, an alert is triggered by either “Abrupt Growth” or “Outliers exceeding historical levels.” Conversely, during the off-epidemic season, an alert necessitates the concurrent presence of both “Abrupt Growth” and “Outliers exceeding historical levels.”

Phase 2. Threshold determination using SoftMax function: This method employs a Softmax function to determine the warning thresholds. The aim is to optimize the balance between timeliness, sensitivity, and specificity for threshold determination, which is crucial for accurate and timely epidemic detection.

Phase 3. Optimal warning strategy for single-source data: Optimally calibrated warning thresholds are applied to single-source data indicators, such as ILI. Warning signals are generated whenever such data surpasses the established threshold, signaling potential health risks that warrant immediate attention.

Phase 4. Integration of multi-source warning signals: Warning signals are synthesized from multi-source warning signals. A comprehensive assessment of warning probability is achieved by calculating a weighted ensemble probability, where each data source is assigned a specific weight. This integration enhances the reliability and accuracy of the warning systems.

Phase 5. Threshold setting for warning probability: A definitive threshold for the warning probability is established to evaluate integrated warning signals. Exceeding this threshold prompts the issuance of an alert, signaling the potential emergence of a public health threat or the initiation of an epidemic.

During these specified phases, the ADTM provides a comprehensive strategy for epidemic surveillance. It incorporates single-source and multi-source data while adjusting thresholds dynamically based on critical epidemiological parameters.

Comparative study of early warning methods: The timeliness of an early warning method was determined by the date of the first warning signal, positioned within the timeline of an outbreak period. The volume of warnings is reflected in the count of days with issued warning signals, as dictated by the warning rules. Consistency was assessed using the Kappa coefficient, which accounts for the probability of random agreement. Statistical significance was attributed to findings with a P<0.05.

This approach employs fully supervised learning to reframe the warning issue as a classification task. It accomplishes the categorization of warning levels through the acquisition of multi-source time-series characteristics. The efficacy of early warning for the target metric (Weifang ILI%) is attained by constructing a dataset suitable for supervised learning and utilizing the eXtreme Gradient Boosting (XGBoost) machine learning model (10). The XGBoost model, which leverages decision trees and gradient boosting, serves as the underlying framework, which we detail in the Supplementary Materials. Initially, aligning the multi-source time series with the warning labels of the target metric establishes a correspondence between features and labels. Subsequently, these features and labels are fed into the XGBoost model for training, effectively addressing the supervised learning issue as illustrated in Figure 2.

Figure 2. 

Schematic diagram of the MLSM and MLUM models

Abbreviation: MLSM=machine learning supervised method; MLUM=machine learning unsupervised method.

In the MLSM study, we utilized a training set spanning from January 1, 2020 to November 30, 2022, comprising 1065 days. The test set ranged from December 1, 2022 to May 31, 2023, totaling 182 days. The training set to test set ratio is approximately 6∶1 requirements for dataset partitioning.

This approach reconceptualizes the challenge of early warning into a task of anomaly detection. Utilizing unsupervised learning, the model analyzes characteristics of multi-source time series data to identify atypical signals indicative of early warnings. We employ the Isolation Forest algorithm, a machine learning model notably used for its efficacy in anomaly detection (11), to thoroughly examine the intrinsic properties of the provided multi-source time series data. The fundamental principle of the Isolation Forest method is that normal and anomalous data points manifest distinct traits; by evaluating and segregating the outliers, the model successfully pinpoints potential anomalies (Supplementary Figure S1). An advantage of this technique over fully supervised learning is that it eschews the necessity for data labeling, thereby simplifying the implementation of the early warning system (Figure 2). Both the training and test sets in the MLUM were identical to those in the MLSM.

The traditional adaptive dynamic threshold method’s effectiveness was rigorously compared with two other methods by assessing their sensitivity and specificity. To establish a reliable benchmark for this evaluation, we used an expert-based consensus. Professionals from the Weifang CDC and senior medical experts reviewed case timelines, labeling moments requiring early warning with a “1” and all other instances with a “0.” The application of our technique and the subsequent analyses were performed using Python (version 3.6.13; Python Software Foundation, Fredericksburg, VA, US), aided by the scikit-learn library (version 0.24.2), and the R (version 4.3.1; The R Foundation for Statistical Computing, Vienna, Austria). For Python analyses, the utilized packages included Pandas (1.2.0), Numpy (1.19.5), Xgboost (2.0.3), and Scikit-learn (1.0). For R, the employed packages were ggplot2 (3.4.4), patchwork (1.1.3), scales (1.2.1), dplyr (1.1.4), tidyverse (2.0.0), and readxl (1.4.3).

This study evaluated the performance of the traditional ADTM in comparison with two machine-learning-based methods, MLSM and MLUM, over a period of 182 days from December 1, 2022 to May 31, 2023. ADTM issued 37 warnings with a sensitivity of 71% and a specificity of 85%. MLSM generated 35 warnings with a sensitivity of 82% and a specificity of 87%, while MLUM produced 63 warnings with a sensitivity of 100% and a specificity of 80%. ADTM and MLUM issued initial warnings on December 11, with MLSM following on December 16. Pairwise Kappa coefficient analysis indicated significant consistency among these methods (P<0.05) (Figure 3, Table 2).

Figure 3. 

Comparative early warning models using three different approaches. (A) ADTM; (B) MLSM; (C) MLUM.

Abbreviation: ADTM=adaptive dynamic threshold method; MLSM=machine learning supervised method; MLUM=machine learning unsupervised method.

Panel A illustrates the warning signals derived from the ADTM method for the Weifang ILI% data. Panel B shows the results using the MLSM method, and Panel C depicts the outcomes from the MLUM method. The blue line represents the ILI percentage curve, while the red points indicate the warning signals.

In addition, in this study, we utilized three evaluation metrics, precision, recall, and F1-score, to evaluate the warning results of the test set, as shown in Supplementary Table S1.

Early warning systems have advanced by utilizing diverse data sources, incorporating big data and machine learning to improve surveillance. This research compares the ADTM approach with MLSM and MLUM, assessing their effectiveness in early warning scenarios. Through establishing ADTM as the reference point, we evaluate the consistency of results from MLSM and MLUM, emphasizing the impact of machine learning on enhancing public health readiness.

This study introduces an enhanced method for infectious disease surveillance, integrating an automatic threshold selection system to enhance adaptability and scalability across various regions. The China Infectious Diseases Automated-Alert and Response System (CIDARS) implements a spatiotemporal early warning model for Type 1 diseases, covering nine infectious diseases, and Type 2 diseases involving 19 infectious diseases, utilizing Fixed-threshold, Temporal, and Spatial detection methods (10). However, challenges arose in determining precise thresholds for different regions, times, populations, policies (11), and behaviors, hindering rapid adjustments to these factors. To tackle this issue, a dynamic threshold selection function has been developed in this study, enabling real-time adaptation of thresholds, thereby increasing the method’s versatility and facilitating its application across diverse geographical areas.

The three methodologies, ADTM, MLSM, and MLUM, exhibit varying effectiveness and suitability in early warning systems. MLSM and MLUM represent the fundamental paradigms of machine learning, each offering unique approaches to problem solving. ADTM and MLUM are particularly relevant for timely anomaly detection crucial for outbreak response. ADTM excels in specificity, reducing false alarms and saving resources, but may lack sensitivity in detecting certain anomalies. MLSM strikes a balance between sensitivity and specificity, albeit with less straightforward interpretability. MLUM stands out for its high sensitivity, benefiting disease detection at the expense of specificity, making it valuable for conditions with significant clinical impacts. Validating the reliability of machine learning methods in infectious disease early warning, the study uses ADTM as a benchmark. Machine learning’s computational strength, combined with independence from traditional benchmarks, bears promise for future applications. However, caution is advised as predictive models, with moderate Kappa coefficient agreement, are not infallible and should not be the sole determinants of public health decisions. A more robust approach involves integrating diverse data sources and surveillance methods with predictive models to enhance early warning system reliability and effectiveness, mitigating the limitations of single-model predictions and fortifying public health strategies.

The methodology of the study has limitations, particularly in finding a dependable benchmark for early warning models, notably with machine learning. The study aimed to compare models under similar conditions without in-depth exploration of their intricacies. The MLUM model prioritized timeliness and sensitivity, albeit with a trade-off in specificity due to its parameter settings. Future research may consider more sophisticated models to enhance accuracy. Furthermore, the study was confined to the Weifang area, suggesting the necessity for broader validation in other regions in subsequent work.

No conflicts of interest.