| Abstract|| |
Background: The lack of objective methodologies and open datasets for the evaluation of the algorithms complicates the objective evaluation by specialists and hinders the widespread use of this technology in health care. The purpose of this study was to estimate the accuracy of Skinive's algorithm 2020 version, then, after an algorithm improvement in 2020–2021, to show a statistically significant decrease in neural network errors in the risk assessment of skin pathologies in 2021. Methods: The Skinive neural network uses a machine-learning algorithm to calculate the risk rating of skin pathologies. For this study, we used Skinive's algorithm 2020 and 2021 versions trained on 64,000 and 115,000 images, respectively. Three validation datasets were used to assess the sensitivity of the algorithm: precancer + cancer, viral skin pathology, acne, containing 285 images in each set. The specificity has been calculated on a separate validation set containing 6,000 benign neoplasm cases. Results: The sensitivity of the Skinive neural network in detecting malignant neoplasms was 89.1% and 95.4% in 2020 and 2021, respectively. The specificity of Skinive's neural network in determining benign neoplasms was 95.3% in 2020 and 97.9% in 2021. For all skin neoplasms, in 2020, the sensitivity was 95.3%, and specificity was 93.5%; in 2021, these were 97.9% and 97.1%, respectively. Conclusions: The results of sensitivity and specificity of the Skinive neural network indicate that the algorithm is highly accurate in detecting various neoplasms and skin diseases. After improving the algorithm, we showed a statistically significant decrease in the number of neural network errors in determining the risks of skin pathologies.
Keywords: Artificial intelligence, machine learning, neural network, skin detection, skin diseases
|How to cite this article:|
Sokolov K, Shpudeiko V. Dynamics of the neural network accuracy in the context of modernization of the algorithms of skin pathology recognition. Indian J Dermatol 2022;67:312
|How to cite this URL:|
Sokolov K, Shpudeiko V. Dynamics of the neural network accuracy in the context of modernization of the algorithms of skin pathology recognition. Indian J Dermatol [serial online] 2022 [cited 2022 Sep 29];67:312. Available from: https://www.e-ijd.org/text.asp?2022/67/3/312/356724
| Background|| |
As a result of the more frequent appearance of benign neoplasms, pre-cancerous conditions, an increase in the number of malignant neoplasms, and neoplasm lesions, the demand for visits to general practitioners and specialists increased accordingly. Seeking treatment in later cases can lead to negative consequences in the form of progression of the tumor process, lower effectiveness of treatment, and poor prognosis.
At the same time, measures aimed at early detection of skin pathologies have several problems: 1) excessive vigilance; 2) prolonged waiting for consultations; 3) unjustified increase of the burden on the doctor, etc.
Statistics show a serious shortage of dermatologists—in the EU, there are only 5 dermatologists per 100,000 people. As a result, doctors cannot provide high-quality medical care for the population.
The solution to these problems has been made possible by the introduction of machine learning technologies into medical practice. Of particular importance in the development of such automation is the development of mobile applications.
The potential of mobile applications using machine learning algorithms to detect skin diseases becomes particularly relevant in times of adverse epidemiological conditions (for example, the situation with COVID-19, as reflected in recent publications) when telemedicine becomes particularly relevant.
The possibilities for in-depth machine training in the differential diagnosis of skin diseases are inspiring and demonstrate its potential to assist clinicians in their routine practice. For example, such solutions can help prioritize clinical care or help non-dermatologists to initiate dermatological care more accurately and potentially improve access to medical care.
The accuracy of diagnosis of skin diseases among general practitioners (GP) and dermatologists is significantly different. For example, the study proved that the accuracy of melanoma recognition among GPs is only 0.49–0.80, whereas among dermatologists this indicator reaches 0.85–0.89.
In another study, the authors attempted to compare the accuracy of skin pathology recognition between a machine learning algorithm and medical specialists. The results showed that the neural network reaches diagnostic accuracy of 90% (accuracy for the first three results given by the neural network).
These studies suggest that the accuracy of machine learning algorithms may be comparable to that of dermatologists, or even much higher., Also, the accuracy of such algorithms is significantly higher than that of general practitioners.
Despite these advantages, the application of software products to such responsible tasks as early detection of skin cancer has been criticized in various ways. The main argument of the critics is the dubious sensitivity and specificity, as well as the lack of a uniform approach to testing required for objective evaluation.
The integration of such solutions into clinical practice is possible only with the presence of a high-quality, well-trained neural network. This involves a range of activities, including the neural network software enhancements, training and validation datasets creation and improvement, regular re-education of the neural network, improvement of the external envelope for convenient use of the mobile application by end-user, and others.
Skinive neural networks can be an example of works carried out for network improvement.
| Materials and Methods|| |
Ethics approval and consent to participate
All images of patients' skin in our manuscript with the histological report are provided to the authors in accordance with the Skinive MD application approbation agreement (the agreement includes consent to the publication of photos in an impersonal form). The approbation was carried out in the Republic of Belarus by dermatologists and oncologists under the leadership of the Ministry of Health of the Republic of Belarus (more information: https://skinive.com/skinive-belarus-aprobation/). The approbation agreement can be provided upon request. The images were provided to the authors in an impersonal form, it is impossible to identify the patient from these images; therefore, additional consent for the publication of these photos from the patients is not required.
Characteristics of the Skinive neural network
A skinive algorithm is a pre-screened, full-fledged dermatological neural network and a neural network environment application located in a protected cloud and integrated with applications (web, mobile, and other) via an API service.
From the moment of its creation until May 2020, such layers as drop out layer and local response normalization layer have been implemented in the architecture of the Skinive neural network to increase the accuracy of recognition of skin pathologies.
Optimization of the Skinive neural network in 2020-2021
From May 2020 to August 2021, the Skinive neural network has undergone numerous technical and clinical improvements.
Technical improvements for the period 2020–2021 include the introduction of new recognition layers, the use of an improved optimization algorithm, the task of switching to a higher resolution was completed, and the introduction of an augmentation function (improvement of recognition by generating new data based on existing data).
Clinical improvements to the medical imaging database (training dataset) included: 1) quantitative improvement; 2) quality improvement; and 3) unique methods of medical image processing before sending them to train the neural network.
Quantitative improvement of the dataset means an increase in the number of recognized pathologies and the total number of images for training the neural network. As of August 2021, the number of recognized pathologies increased to 45 (in 2020: 31 pathologies).
The number of images for the neural network training (the volume of the training dataset) in 2021 increased to 115,000 images (in 2020: 64,000 images). In total, by August 2021, the Skinive neural network has analyzed more than 250,000 images (as of May 2020, 150,000 images were analyzed).
The Skinive team paid particular attention to the quality of the training datasets. The main principle of the work in 2020–2021 was not a desire to increase the number of marked-up images, but a qualitative improvement of data.
All data used in the neural network training underwent a multilevel check for compliance with the image quality and unambiguous interpretation by medical experts.
The unique method of processing medical images before sending them to train the neural network included 1) preliminary machine distribution of the data array by the previously trained neural network; 2) manual correction of the erroneous distribution of data by a medical expert; 3) engaging third-party medical specialists from our cohort of trusted doctors to verify complex cases of skin pathology in the photograph; and 4) software solutions for data labeling with multiple pathologies in one image or with complex anatomical localization.
The retraining of the Skinive neural network was initiated in July 2021 and completed in August 2021.
Immediately after that, in August 2021, a large-scale work was carried out, which made it possible to obtain new data on the results of the accuracy of the skin pathology recognition by the Skinive 2021 neural network.
Formation of the validation dataset
Validation datasets were drafted in 2020. These datasets were used in accuracy analysis in both 2020 and 2021, allowing the evolution of sensitivity and specificity.
To determine the algorithm sensitivity, validation datasets are formed from the Skinive user database, consisting of 285 images in each of the classes: 1) acne (including acne vulgaris, acne pustular, acne cystic, comedone closed, comedone open, milium, rosacea); 2) viral diseases (including viral papilloma, wart vulgaris, wart plane, wart plantar, molluscus contagiosum); 3) precancer + cancer (including actinic keratosis, bowen, basal cell carcinoma, squamous cell carcinoma, melanoma).
To determine the specificity of the algorithm, a separate set of validation data was formed, consisting of 6,000 images with benign skin tumors (including benign nevus, papilloma nevus, hemangioma, dermatofibroma, halo nevus, spitz nevus, pyogenic granuloma).
All formed validation datasets were sent in turn for analysis using the Skinive neural network. The analysis of each image included the stage of filtering foreign objects and noise, segmenting the lesion, followed by determining signs of similarity with a specific skin pathology. After that, for each image, the neural network formed the risk level: low risk or high risk. The resulting risk values from the neural network were compared with the reference values for the risk of skin pathology.
Sensitivity was defined as the ratio of the number of high-risk cases correctly identified by the algorithm (skin pathology present) to the number of all clinically confirmed cases (risk present, proportion of true positive cases). Specificity was defined as the ratio of the number of low-risk cases correctly identified by the algorithm (no pathology, benign skin neoplasm) to the number of all clinically confirmed benign neoplasms (no risk, proportion of true negative cases).
| Results|| |
After sending the formed validation datasets for analysis to the Skinive algorithm, the sensitivity and specificity values for each of the analyzed classes were obtained, as shown in [Table 1].
To understand the errors of the algorithm and detect the tendency of incorrect recognition of the neural network, the total number of false recognitions (Miss Rate) was calculated [Table 2].
Overall, Skinive's neural network sensitivity in 2020 was 95.3% (confidence interval [CI]: 95% 93.7–96.6%) with a specificity of 93.5% (CI 95% 93.2–93.6%). In 2021, 97.9% (CI 95% 97.2–98.5%), and 97.1% (CI 95% 96.9–97.2%) respectively.
| Discussion|| |
Thanks to a whole set of works carried out on the Skinive neural network, it was possible to obtain an increase in the indices of sensitivity and specificity for almost all classes of neoplasms and skin diseases studied.
The analysis of confidence intervals allows us to establish that the sensitivity of the Skinive algorithm in recognition of precancer states and malignant neoplasms has a reliable positive growth for the period 2020–2021. The specificity of recognition of benign neoplasms has also increased statistically.
The results make it possible to assert that the set of measures aimed at improving the Skinive neural network has been especially reflected in the improvement of the accuracy of the algorithm working with skin cancer pathology. This pathology is most relevant in terms of saving lives and the health of the population.
Comparing the total number of false recognitions for 2020 and 2021 shows a global tendency toward a decrease in the number of misinterpretations in the recognition of skin pathologies. In addition, this comparison allows us to note the dynamics and correctness of the vector of ongoing work on improving the neural network.
The most positive changes occurred in the precancer + cancer class. The total number of errors in 2021 was 4.6%, which was statistically significantly less than the same value obtained in 2020. The decrease in the number of errors in the recognition of benign neoplasms was no less significant. Therefore, in 2020, the level of erroneous recognition in this class was 13.7%, in 2021, 8.8%, which is also significantly lower than the previous year.
When analyzing the general indicators of sensitivity and specificity of the Skinive algorithm, the general trend of a statistically significant increase in indicators remains. Given the relevance of timely and reliable recognition of oncological risks, this trend indicates a real possibility of using the Skinive neural network in clinical practice.
| Conclusions|| |
The sensitivity and specificity of the Skinive neural network indicate the high accuracy of the algorithm in identifying various neoplasms and skin diseases and amounted to more than 90% for almost all analyzed classes of skin pathology.
The key points in improving neural networks designed to classify dermatological and oncological skin diseases include constant work on the qualitative composition of training datasets, the quantitative parameters of datasets, and software improvement of artificial intelligence algorithms.
The effect of the modernization of Skinive neural networks is confirmed by a significant positive difference between the values of sensitivity and specificity for determining skin pathology obtained in 2020 and 2021.
The high-precision results of the neural network and the presence of a certified mobile application “Skinive MD” (ISO 13485, CE-mark) intended for medical professionals indicate the possibility of introducing the Skinive project into medical practice.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Chatterjee P, Nagi N, Agarwal A, Das B, Banerjee S, Sarkar S, et al
. The 2019 novel coronavirus disease (COVID-19) pandemic: A review of the current evidence. Indian J Med Res 2020;151:147-59.
] [Full text]
Du-Harpur X, Watt FM, Luscombe NM, Lynch MD. What is AI? Applications of artificial intelligence to dermatology. Br J Dermatol 2020;183:423-30.
Michael DC, Judy W. Agreement between dermatologists and primary care practitioners in the diagnosis of malignant melanoma: Review of the literature. J Cutan Med Surg 2012;16:306-10.
Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P, et al
. A deep learning system for differential diagnosis of skin diseases. Nat Med 2020;26:900–8.
Brinker TJ, Hekler A, Enk AH, Berking C, Haferkamp S, Hauschild A, et al
. Deep neural networks are superior to dermatologists in melanoma image classification. Eur J Cancer 2019;119:11-7.
Udrea A, Mitra GD, Costea D, Noels EC, Wakkee M, Siegel DM, et al
. Accuracy of a smartphone application for triage of skin lesions based on machine learning algorithms. J Eur Acad Dermatol Venereol 2020;34:648-55.
[Table 1], [Table 2]