Methods and Applications: Establishment and Validation of a Competency-Based Evaluation Indicator System for On-Site Disinfection Service Providers — Shanghai Municipality, China, 2023

With the increasing complexity of natural and social environments, infectious disease pathogens often undergo adaptive evolution, leading to enhanced survival ability and infectivity. Severe Acute Respiratory Syndrome (SARS) coronavirus, Middle East Respiratory Syndrome (MERS) coronavirus, and endemic Human Coronaviruses (HCoV) have been shown to persist on inanimate surfaces such as metal, glass or plastic for up to 9 days (1). Comprehensive reviews have indicated that contact with contaminated environments is the primary transmission pathway for certain respiratory viruses, including influenza (2). Therefore, effective disinfection is a key measure to interrupt the transmission routes of infectious diseases.

The efficacy of disinfection is significantly influenced by multiple factors, including temperature, disinfectant concentration, and exposure duration (3). Consequently, improper disinfection practices often result in disinfection failure may inadvertently facilitate disease transmission. Empirical evidence has underscored these challenges. A study evaluating disinfection efficacy in epidemic foci found that approximately 10% of object surfaces failed to meet the disinfection standards, and the factors influencing, this failure included the type of disinfectant, the operation by disinfection personnel, on-site operation related to the use of disinfection equipment, and other issues (4). There have been reports of cases of operational errors during disinfection, resulting in occupational exposure among personnel (5). Current evaluation of disinfection efficacy mainly relies on the inactivation rate of natural flora or indicator microorganisms post-disinfection as primary evaluation indicator (6).

However, the evaluation results of disinfection efficacy can only be determined after disinfection is completed, making pre-intervention impossible. Therefore, it is important to establish a pre-disinfection indicator system to evaluate the capability of on-site disinfection service providers (ODSP) before intervention. This system aims to minimize disinfection failures and prevent subsequent disease transmission.

This study investigated disinfection service providers registered in Shanghai until June 2024. Eighty-eight providers participated in the study. Data collection included questionnaire surveys and on-site investigations conducted by providers. The workflow for data collection, indicators establishment, and validation is shown in Supplementary Figure S1.

Competency evaluation indicators for ODSP were formulated in accordance with legal standards such as the Measures for the Administration of Disinfection, General Principles for Disinfection of Epidemic Foci (GB 19193-2015), and Standards for On-site Disinfection Evaluation (WS/T797-2022). The framework of the indicator system and the indicator weights were determined based on two rounds of expert interviews. Fifteen experts were recruited for this study, and only those with professional titles of associate senior or above and at least five years of experience in fields such as disinfection, epidemic prevention, emergency response, health policy, and health supervision theory and practice were eligible. The experts assigned the importance of indicators to the framework and used the analytic hierarchy process (AHP) to determine the weight of each indicator. The consistency ratio (CR) values of the judgment matrices for all indicators obtained from the consistency test ranged from 0.015 to 0.069, indicating good consistency (CR<0.1), and the weights of each indicator were thereby determined.

The competency evaluation indicators for ODSP were primarily based on the regulatory guidelines, such as on-site disinfection effect evaluation standards and expert interviews. This study systematically constructed a multidimensional capability evaluation indicator system with 4 primary and 13 secondary indicators (Table 1).

The above-mentioned indicators were used to evaluate and analyze 88 ODSP. Among all providers, 52 had carried out infectious disease disinfection and conducted on-site disinfection effect evaluation, whereas the other 36 mainly carried out preventive disinfection work. To further explore the factors influencing, infectious disease disinfection failure in ODSP, the providers were grouped according to whether they had a history of disinfection failure, and the differences among different indicators were compared. To evaluate the agreement between the newly established competency thresholds and historical efficacy outcomes, taking the results of historical effect evaluation as the gold standard, the sensitivity and specificity corresponding to different capability scores were calculated, and the receiver operating characteristic (ROC) curve was plotted. The area under the curve (AUC) was calculated to quantify the evaluation accuracy, where AUC=1 indicated perfect discrimination and AUC=0.5 suggested random classification. Subsequently, Kappa consistency analysis was performed. Data entry and cleaning were performed using Excel (Microsoft Office Home and Student Edition 2021, Microsoft Corporation, Redmond, USA). Statistical analyses were conducted using R Statistical Software (version 4.3.3, R Development Core Team, Vienna, Austria). The significance level was set at α=0.05.

Analysis of the capability scoring results of the 88 ODSP showed that they achieved relatively high scoring rates in terms of management systems (93.75%) and material reserves (90.45%). However, their scoring rates in personnel capability (75.08%) and implementation of quality control (51.97%) were relatively low (Supplementary Table S1). The total scores of providers with experience in infectious disease disinfection were significantly higher than those of providers without experience. A further analysis of the 52 providers with experience showed that the scores of the providers that failed the on-site disinfection effect evaluation were significantly lower in terms of personnel capability and quality control than those with all passing results (P<0.01) (Table 2).

The AUC of the ROC curve is 0.954 [95% (CI): 0.898–1] (Figure 1). The sensitivity and specificity corresponding to different scores are shown in Supplementary Table S2. Based on the Youden index for each score, when the total score was less than 70, the on-site disinfection service institution did not have sufficient capability to carry out disinfection for infectious disease. A consistency test was conducted using 70 points as the evaluation criterion. These results were in good consistency with the evaluation results of the on-site disinfection, with a kappa value of 0.822 (Table 3).

Figure 1. 

ROC curve for predicting results of on-site disinfection effect evaluation by the assessment indicators of on-site disinfection service providers.

Abbreviation: ROC=receiver operating characteristic; AUC=area under curve.

Currently, the Measures for the Administration of Disinfection does not contain relevant requirements for ODSP. In the practices of various provinces and cities, management is primarily based on local regulations and industry association standards. The existing inspection frameworks cover aspects such as basic information, facilities and equipment, personnel qualifications, and management systems. This study found that ODSP scored relatively high in the above-mentioned aspects, but some still had deficiencies in terms of personnel capability and disinfection quality control. Moreover, issues related to disinfection quality control may cause on-site disinfection failures. Studies have shown that effective on-site disinfection can be achieved through training and re-education of disinfection staff, adding more cleaning staff or supervisors, and/or the use of implementation or quality checklists (7).

When on-site disinfection at infectious disease loci fails to meet efficacy standards, re-disinfection becomes necessary, leading to substantial economic and temporal costs. This study found that there are significant differences in the capabilities of different ODSP. Although many of these providers, have long been engaged in disinfection, they lack the capability to conduct on-site disinfection for infectious diseases. During a pandemic, the demand for on-site disinfection increases significantly. Disinfection service providers are unsure whether their own capabilities are sufficient to carry out infectious disease disinfection, and managers lack the tools to identify qualified providers. Therefore, for infectious disease disinfection, especially during major epidemics, it is necessary to establish a mechanism for hierarchical certification and dynamic supervision to ensure the competence of providers engaged in disinfection, thereby guaranteeing the effectiveness of on-site disinfection.

The evaluation indicator system developed in this study addresses the challenges associated with major infectious disease outbreaks. During emergencies, the system enables managers to quickly and accurately identify the capability level of ODSP, thereby facilitating timely initial selection and task allocation. For example, institutions can be classified into different grades based on their evaluation scores, such as those qualified to conduct preventive disinfection, on-site disinfection for common infectious diseases, and disinfection for Class A and B infectious diseases. Conversely, in non-emergency situations, it can promote the capability improvement of ODSP, allowing them to identify their own deficiencies and carry out targeted upgrades in both infrastructure and personnel training.

This study had several limitations. First, this was a retrospective study. ODSP capabilities were investigated after the evaluation of the providers’ capabilities, which mainly relied on historical records. This may have affected the reliability of the research results owing to inaccurate data. Future prospective studies are required to optimize this evaluation system. Second, as this study was conducted in Shanghai, the results may not be generalizable to other regions or populations. External validation through multicenter or national studies is essential for future research.

We thank all study participants who contributed to the study by collecting and sharing data.