Preplanned Studies: Construction of a Competency Evaluation Indicator System for Emergency Response Staff in Disease Control and Prevention Institutions — China, 2023

With increasing complexity in natural and social environments, the frequency of emerging and re-emerging infectious diseases and public health emergencies have risen. Recent outbreaks of H1N1 influenza, H7N9 influenza, coronavirus disease 2019 (COVID-19), and monkeypox (Mpox) exemplify this trend, underscoring the need for rapid, high-quality emergency responses (1). This situation necessitates enhancing emergency response capacity and improving disease control and prevention systems to ensure high-quality development. During major public health crises like pandemics, challenges often include inefficient organization of health emergency personnel, mismatches between job roles and professional skills sets, and suboptimal emergency response outcomes (2-3). Effective health emergency response management thus requires a comprehensive understanding of emergency staff capabilities to maximize individual and team effectiveness. Currently, a comprehensive, systematic, and scientific set of evaluation criteria or indicators for assessing the competency of health emergency staff, particularly those in disease control and prevention during significant infectious disease outbreaks, is lacking. This paper proposes using the Delphi method, informed by past experiences with infectious disease epidemics and public health emergencies, to develop a competency evaluation system for emergency response personnel in disease control and prevention institutions. This system aims to enhance health emergency management and contribute to developing a skilled workforce within the disease control and prevention system.

This study employed a two-round Delphi method to solicit expert consultations. Experts were eligible if they possessed at least an associate senior title or higher, a master’s degree or higher, and a minimum of 5 years of experience in fields such as epidemic prevention, emergency response, health policy, health management, and infectious disease epidemiology theory and practice. Following established Delphi method guidelines and computational approaches (4-5), the study aimed to recruit 50 experts. Qualitative methods, including literature reviews and interviews, informed the development of an initial competency indicator system and a subsequent expert consultation questionnaire. The questionnaire encompassed aspects such as the theoretical foundation, framework description of the indicator system, assessment of each indicator, experts’ basic information, and evaluations of familiarity with and authority over the questionnaire. Indicator screening in each consultation phase employed the boundary value method (6). Indicators not meeting the boundary criteria were reviewed by the research group, and decisions regarding inclusion were made based on expert feedback and research group deliberations. Valid expert recommendations were incorporated, resulting in indicator modifications (Supplementary Figure S1).

To determine whether the system could serve as a competency evaluation tool, an empirical study was conducted using a questionnaire survey scale. Cronbach’s α coefficient reliability testing, confirmatory factor analysis (CFA), and convergent and discriminant validity testing were performed.

Data entry and cleaning were performed using Excel (Microsoft Office Home and Student Edition 2016, Microsoft Corporation, Redmond, USA). The expert authority coefficient, degree of concentration, and coefficient of variation (CV) for expert opinions were calculated, along with indicator screening in the Delphi method. Discriminant and convergent validity were assessed through empirical research. SPSS software (version 26.0, IBM, Armonk, NY, USA) was used to analyze the expert coordination coefficient and consultation reliability within the Delphi method, followed by calculating Cronbach’s α coefficient for the empirical research. CFA was conducted using R software (version 4.2.3, R Core Team, Vienna, Austria). The significance level was set at α≤0.05.

This study developed an indicator pool to evaluate the competency of emergency response staff in disease control and prevention institutions. This pool was established by synthesizing indicators from relevant literature and conducting qualitative interviews with professionals. The organizational framework of the pool was structured based on epidemiological investigations and emergency response processes in China (7) and the McClellan competency dictionary. This process created a preliminary competency evaluation indicator system, including 4 first-level indicators, 10 second-level indicators, and 47 third-level indicators (Table 1). These indicators generated the first-round questionnaire for the Delphi consultation.

In the initial consultation round, 48 of 50 distributed questionnaires were effectively completed, representing a 96% response rate. The participating experts were highly qualified, possessing extensive experience in disease prevention and control, emergency response, and epidemiological investigations. Of these experts, 97.92% had experience managing significant infectious disease outbreaks. The distribution of experts was relatively even across regions: 24.32% from eastern, 29.73% from central, and 32.43% from western provincial-level administrative divisions (PLADs). Additionally, 64.58% of experts held a master’s degree or higher, 77.08% held at least an associate senior title, and 93.75% had over 10 years of professional experience (Supplementary Table S1). In the subsequent round, the same 48 experts were consulted, yielding 45 effective responses.

Statistical analysis showed that the CVs for the importance scores in the first and second consultation rounds were 0.123 and 0.109, respectively, with the lower CV in the second round indicating increased consensus among experts regarding the indicators. The reliability of the expert consultations across both rounds was high, with Cronbach’s α values of 0.957 and 0.948, respectively, both exceeding the threshold of 0.7, suggesting good internal consistency among expert opinions. During initial consultation, the coordination coefficients for the first, second, and third-level indicators were 0.354, 0.400, and 0.201, respectively; in the subsequent round, these values were 0.394, 0.353, and 0.160, respectively. All coefficients reached statistical significance at P<0.001, indicating consistent expert opinions across the two rounds and effective coordination of opinions. However, a decrease in the coordination coefficients for the second- and third-level indicators in the second round prompted further investigation. Analysis revealed that 16 experts assigned scores of 3 or lower to eight indicators, including variables like Environmental and food hygiene, Other basic medical knowledge, Recent research updates, humanistic adaptability, Good at innovation, Self-directed learning, Job recognition, and motivation for achievement. The coordination coefficient for these scores was 0.258 with a notable coordination point (4). Consequently, follow-up telephone interviews with these 16 experts revealed that their assessments reflected the current situation and showed a minimal requirement for capability enhancement. In contrast, the remaining 29 experts displayed greater anticipation for future skill development. The research team decided to adopt the perspective of the latter group.

The initial consultation phase yielded average authority coefficients of 0.929, 0.927, 0.875, and 0.848 for the 4 primary indicators: Knowledge base, Professional skills, Personal qualities, and Personality and Motivation, respectively. These values shifted slightly to 0.908, 0.908, 0.878, and 0.850, respectively, in the subsequent consultation round. All experts’ authority coefficients were above 0.7, indicating a high level of expert authority. Analysis of indicator importance scores prompted the identification and modification of several indicators, as detailed in Supplementary Table S2. These modifications included adding indicators such as Environmental and food hygiene, Place investigation, and Report writing. Conversely, indicators like risk judgment, risk identification ability, analysis chart-making, and related expressions were simplified or removed, resulting in a refined set of indicators for the second round. This updated indicator system provided a more comprehensive and clearer definition and positioning of each element, demonstrating a more structured workflow progression and alignment with the competency onion model. Further consultation and analysis led to enhancements in the definitions of each third-level indicator, ultimately establishing a framework of 4 first-level indicators, 10 second-level indicators, and 46 third-level indicators (Table 1). This structure aligns with the competency evaluation indicator system for emergency response personnel, as illustrated in Figure 1, based on the competency onion model theory.

Figure 1. 

Structure of the competency evaluation indicator system for emergency response personnel in disease prevention and control institutions.

In the empirical study, self-assessment data from 383 from national, provincial, municipal and county level CDCs individuals were analyzed. The constructed competency self-assessment scale demonstrated a Cronbach’s α coefficient greater than 0.8, indicating good reliability (Supplementary Table S3). CFA was performed using a second-order factor model, yielding the following fit indices: χ²/df=2.675, CFI=0.901, IFI=0.902, and RMSEA=0.066, suggesting an acceptable overall model fit. Convergent validity was confirmed with all CRs (Composite Reliability) greater than 0.7, AVEs (Average Variance Extracted) for first-level indicators all exceeding 0.5, and 90% of the second-level indicators with AVEs also above 0.5, indicating that the convergent validity of the model was acceptable (8-9). For discriminant validity, the model comparison approach (10) was employed, comparing the original four-factor model against various reduced-factor models. The four-factor model showed superior fit compared to all three-factor, two-factor, and one-factor models, thus confirming the discriminant validity of the first-level indicators designed by the indicator system (Supplementary Table S4).