Preplanned Studies: Developing Machine Learning Models Based on Clinical Manifestations to Predict Influenza — Chongqing Municipality, China, 2022–2023

Introduction: Clinical manifestations are essential for early diagnosis of influenza-like illness (ILI). Machine learning models for influenza prediction were developed and a new ILI definition was introduced.

Methods: A retrospective cohort study was conducted at three hospitals in southwest China during June 2022 and May 2023. Artificial intelligence was used to extract variables from medical records and XGBOOST algorithm was used to develop prediction models for the total population and three age subgroups. A new ILI definition was introduced based on the optimal model and its performance was compared with WHO, China CDC, and USA CDC definitions.

Results: Totally 200,135 patients were included. 4,249 (36.2%) were confirmed influenza. The predictors of the optimal model included epidemiological characteristics, important symptoms and signs, and age for the total population [Area under curve (AUC) 0.734 (0.710–0.750), accuracy 0.689 (0.669–0.772)]. The new ILI definition was fever (≥37.9 °C) with cough or rhinorrhea, and its AUC, sensitivity, and specificity for diagnosing influenza were 0.618 (0.598–0.639), 0.665 and 0.572, outperformed the WHO, China CDC, and USA CDC definitions (P<0.05).

Conclusions: Fever, cough, and rhinorrhea maybe the most important indicators for influenza surveillance.

Influenza poses a significant public health threat. Early identification of influenza based on clinical manifestations is crucial for optimal treatment outcomes and prognosis (1). Influenza surveillance serves as a critical component for outbreak early warning systems and timely implementation of preventive and control measures (2). The World Health Organization (WHO) established a global symptom surveillance network for influenza in 1952, known as influenza-like illness (ILI) surveillance (3). However, limited research has evaluated the performance of ILI definition in influenza surveillance using large-scale data from Chinese populations. To address this gap, a retrospective cohort study was conducted at three tertiary comprehensive and influenza sentinel hospitals in Chongqing Municipality, China, between June 2022 and May 2023 (Supplementary Material). Our findings demonstrate that body temperature, cough, and rhinorrhea may be the most important clinical indicators for influenza diagnosis and ILI surveillance.

The study cohort comprised all patients who visited the emergency departments or fever clinics of the three participating hospitals during the study period. Exclusion criteria included: 1) patients who returned to either department for respiratory illness within one month, and 2) patients lacking diagnosis, chief complaint, or present illness history documentation. Laboratory confirmation of influenza infection followed established diagnostic criteria (4). A total of 27 symptom and sign variables were extracted (Supplementary Table S1) from the electronic medical record (EMR) information systems of the outpatient and emergency departments. CongRong (Supplementary Material), an artificial intelligence (AI) assistant and pre-trained large language model, was utilized to extract symptom variables for database construction.

The model development and validation dataset comprised cases with confirmed influenza laboratory testing results. Important variables were identified using the boruta algorithm, and machine learning models were developed for three age subgroups (0–14 years, 15–64 years, and ≥65 years) and the total population using eXtreme Gradient Boosting (XGBOOST) algorithm (5). For each age group, four models based on different combinations of epidemiological and other variables were constructed , following the process outlined in Supplementary Figure S1. The resulting 16 candidate models were evaluated using the testing dataset (Table 1). The model with the highest area under curve (AUC) of the receiver operating characteristic (ROC) curve was designated as optimal. To interpret the machine learning models, SHapley Additive exPlanations (SHAP) values were employed to quantify the direction and magnitude (mean SHAP value) of important variables in the optimal model (5).

Based on the most significant symptoms and signs positively associated with influenza (indicated by higher importance value with a positive SHAP value) in the optimal model for the total population, a new ILI definition was developed. The diagnostic performance of the new definition alongside existing WHO, China CDC, and USA CDC ILI definitions was evaluated using the testing dataset. Additionally, cross-correlation analysis of time series between ILI cases under these definitions and confirmed influenza cases was conducted using the cross-correlation function from the Stats package.

All statistical analyses were performed using R software version 4.3.2 (R foundation, Vienna, Austria). Continuous variables were compared using t-tests or Kruskal-Wallis tests as appropriate. Categorical variables were analyzed using chi-squared tests or Fisher’s exact tests. The pROC package was employed to determine the optimal body temperature cut-off value (maximum Youden index) and compare model AUC values using the DeLong method.

After data extraction and processing, we established a comprehensive database comprising 200,135 cases. The CongRong model demonstrated exceptional performance in symptom variable extraction, achieving an accuracy of 0.997, sensitivity of 0.991, and specificity of 0.998 in the testing dataset (Supplementary Table S2). From the influenza sub-dataset used for developing and validating infection prediction models (n=11,753; Supplementary Figure S1), we identified 4,249 (36.2%) influenza-positive cases in the total population, with positivity rates of 41.6%, 34.9%, and 41.5% in the 0–14 years, 15–64 years, and ≥65 years age groups, respectively (Supplementary Table S1).

The Boruta algorithm identified distinct sets of important candidate variables for modeling: 18 for the total population, and 7, 16, and 8 variables for the 0–14 years, 15–64 years, and ≥65 years age groups, respectively (Figure 1). The predictive performance metrics of all 16 machine learning models are presented in Table 1. For the total population, model_1 emerged as the optimal prediction model, achieving an accuracy of 0.689 (0.669, 0.772) and an AUC of 0.734 (0.710, 0.750). The most influential predictors in this model included body temperature, age, visiting during the epidemic season, visiting during a certain week of epidemic season, cough, and rhinorrhea, with all factors except age showing strong positive associations with influenza (Figure 1). In the 0–14 years age group, model_1 performed optimally, with body temperature, visiting during a certain week of epidemic season, rhinorrhea, visiting during epidemic season, and cough emerge as the most significant predictors, all demonstrating strong positive correlations with influenza (Figure 1). For the 15–64 years age group, model_2 proved most effective, with body temperature, visiting during a certain week of epidemic season, age, cough, and rhinorrhea identified as key predictor, all except age showing strong positive associations with influenza (Figure 1). In the ≥65 years age group, model_1 demonstrated optimal performance, with visiting during epidemic season, visiting during a certain week of epidemic season, body temperature, rhinorrhea, and cough emerging as the most important predictors, all showing strong positive correlations with influenza (Figure 1). The complete performance metrics for both testing and training datasets are detailed in Table 1 and Supplementary Table S3, respectively.

Figure 1. 

SHAP summary plot illustrating variable importance and directional relationships obtained from the optimal model for influenza prediction across the (A) total population, (B) 0–14 years group, (C) 15–64 years group, and (D) ≥65 years group.

Note: Variables with higher importance values (yellow) and positive SHAP values (right side) demonstrate positive associations, while those with higher importance values (yellow) and negative SHAP values (left side) indicate negative associations.

Abbreviation: SHAP=SHapley Additive exPlanations.

Based on the most important symptoms and signs positively associated with influenza in the optimal model for the total population — body temperature, cough, and rhinorrhea — and using the identified cut-off value for body temperature of 37.9 °C, a new ILI definition was established: fever (≥37.9 °C) with either cough or rhinorrhea. This new definition significantly outperformed (P<0.001) the existing WHO, China CDC, and USA CDC definitions in diagnosing influenza, achieving an AUC of 0.618 (0.598, 0.639), accuracy of 0.605 (0.585, 0.625), sensitivity of 0.665, and specificity of 0.572 (Table 2). Time series analyses of ILI cases under the new, WHO, China CDC, and USA CDC definitions alongside confirmed influenza cases during the study period, revealed that the daily trend cross-correlation coefficients between ILI cases under the new, WHO, China CDC, and USA CDC definitions and influenza cases were 0.701 (P<0.05), 0.685 (P<0.05), 0.648 (P<0.05), and 0.653 (P<0.05) respectively, with peak correlations occurring simultaneously.

This study developed machine learning models for influenza prediction using data from three large sentinel hospitals in Chongqing, China, and identified an optimal model with the highest AUC. Based on the model’s top three predictive symptoms (fever, cough, and rhinorrhea), a new ILI definition was proposed that demonstrated superior performance compared to existing WHO, China CDC, and USA CDC definitions. Our findings suggest that body temperature, cough, and rhinorrhea serve as crucial indicators for both early clinical diagnosis of influenza and ILI surveillance.

Our study employed XGBOOST, an advanced machine learning algorithm that has demonstrated superior performance in clinical and epidemiological studies compared to other approaches such as Ranger, Random Forest, Cforest, SVM, Artificial Neural Network, and Deep Learning (5–6). The SHAP value analysis of our optimal models revealed consistent important variables across all age subgroups and the total population, including epidemic season timing, body temperature, cough, and rhinorrhea, aligning with previous research findings (5,7-9). These results underscore two critical points: first, the importance of timely reporting of influenza epidemic trends by national and regional CDC authorities based on surveillance data; and second, the primary clinical indicators — body temperature, cough, and rhinorrhea — that clinicians should prioritize when diagnosing influenza during epidemic seasons.

While ILI definitions vary across countries and WHO revised its definition in 2011 (9), our study aimed to establish a more accurate definition. Based on our optimal prediction model’s identification of body temperature, cough, and rhinorrhea as the most significant positive predictors of influenza, with a body temperature threshold of 37.9 °C, we proposed defining ILI as fever (≥37.9 °C) with either cough or rhinorrhea. This new definition not only outperformed WHO, China CDC, and USA CDC definitions in our study but also showed the highest daily trend cross-correlation coefficient with confirmed influenza cases. Our temperature threshold of 37.9 °C closely approximates the 38.0 °C specified by WHO and China CDC, and the 37.8 °C by USA CDC. While our definition shares the core elements of fever and cough with existing definitions, it notably substitutes rhinorrhea for sore throat as an alternative criterion. This modification is supported by previous studies identifying rhinorrhea as a common influenza symptom occurring alongside cough (10-11), providing evidence-based justification for optimizing the ILI definition.

Our study has two primary limitations. First, its retrospective design may introduce inherent biases. Second, despite including multiple centers, the data remains geographically confined to one region. Future research should incorporate data from diverse regions and countries to validate these findings.

Cloudwalk Technology for providing artificial intelligence data processing support.

Received approval from the Research Ethics Board of The First Affiliated Hospital of Chongqing Medical University (approval number: K2024-171-01), with informed consent obtained from all participants.