COVID-19 Prognosis from Chest X-ray Images by using Deep Learning Approaches: A Next Generation Diagnostic Tool

Global public health is overwhelmed due to the ongoing Corona Virus Disease (COVID-19). As of October 2022, the causative virus SARS-CoV-2 and its multiple variants have infected more than 600 million confirmed cases and nearly 6.5 million fatalities globally. The main objective of this reported study is to understand the COVID-19 infection better from the chest X-ray (CXR) image database of COVID-19 cases from the dataset of CXR of normal, pneumonia and COVID-19 patients. Deep learning approaches like VGG-16 and LSTM models were used to classify images as normal, pneumonia and COVID-19 impacted by extracting the features. It has been observed during the COVID-19 pandemic peaks that large number of patients could not avail medical beds and were seen stranded outdoors. To address such health emergency situations with limited available bed and scarcity of expert physicians, computer-aided analysis could save precious lives through early screening and appropriate care. Such computer-based deep-learning strategy could help during future pandemics, especially when the available health resources and the need for preventive measures to take do not match the burden of a disease.


INTRODUCTION
The continuously emerging and reemerging SARS-CoV-2 variants have led to a huge health crisis worldwide.2][3][4] This pandemic highly transmissible virus in its multiple mutated variant forms has killed millions globally, mainly due to human-human transmission. 5Initially, the Delta variant was most deadly and later the Omicron variant and its sub-lineages prevailed as the most infective ones.Emerging sub-lineages like BA.1, BA.2, BA.3, BA.4/5, BA.2.75, BA.4.6 tremendously affected healthcare infrastructure at global-scale.The virus reportedly gradually infected the upper and lower respiratory system, heart, kidney, liver, gut and the nervous system, leading to multiorgan damage. 6,72][13][14] To contain the virus, numerous measures were implemented.However, its continuously emerging mutating multiple virus variants have tremendously affected global healthcare infrastructure.Another major challenge that surfaced is the shortage of test kits and similar medical gadgets.Also, the seemingly low safety and efficacy of the available COVID-19 vaccines also play a crucial role in the emergence of virus variants. 15,16The pandemic continues to threaten the global population as a result.In such a scenario, early and automatic screening of samples may improve healthcare facility worldwide to handle the extraordinary health crisis.Various AI-based (especially ML and DL) techniques have been tried previously, for automatic screening of COVID-19 cases. 17hang et al.

Methodology
The COVID-19 CXR image dataset was used to perform COVID-19 prognosis.Dataset used for the work was sourced from https:// github.com/ieee8023/covid-chestxray-dataset. 24t contained 5863 images collected from hospitals, public sources and directly from physicians.It had 94 normal CXR images, 506 of COVID-19 and 46 of pneumonia.Two deep learning models, VGG16 and LSTM, were implemented to study the normal, COVID and pneumonia cases from the images.The study plan is schematically presented in Figure 1.The COVID-19 CXR image database was considered first.Then the features from the images were extracted.Then, deep learning models with 200 epochs were implemented on the database.80% data were considered for training and 20% data for the run.The performance accuracy, the ROC (receiver operating characteristics curve), precision, recall and F1 score of each model were ascertained.SoftMax classifier was used to classify images into normal, COVID-19 and pneumonia.

Dataset
As indicated, the dataset for the study was accessed from the publicly available resources (at https://github.com/ieee8023/covid-chestxraydataset).Dataset was processed in Python open-source software.TensorFlow environment was used to process.Keras, Pandas, NumPy, and matplotlib packages were used.The data were contained in two folders, one to train and the other for test, and both train and test folders contained three subfolders (normal, COVID-19 and pneumonia; In Figure 2, the normal CXR image depicted clear lungs without any abnormal opacity, COVID-19 image exhibited a focal lobar consolidation, and pneumonia image had a more diffuse interstitial pattern in both the lungs.The dataset contained 6432 X-ray images in all and 20% of the images were considered as test data.All CXR images were curated for consistency and quality by removing all low quality or unreadable  Each model was trained for 200 epochs.The batch size was 32, the learning rate was set to 0.0001, and ADAM was used as the loss function.It was decided to train a cross-entropy loss function on all models.

VGG16
VGG16 (Figure 3) is an advanced version of CNN architecture with distinct feature.The architecture consists of convolutional layer, pooling layer and two fully connected layer.The last layer of the architecture is soft max classifier.

Long short-term memory (LSTM)
Long short-term memory (LSTM) is a DL (RNN) architecture developed by Hochreiter and Schmidhuber. 26LSTM has the ability to handle total data points instead of single data points.It contains memory elements and consists of input gate, forget gate and output gate which control the flow of information through LSTM architecture.LSTMs is a type of RNN used for eliminating the long term dependency problem and can remember information for longer duration due to its feedback connection. 27Sigmoid gate is known as forget gate which helps in selection of data passing though LSTM architecture.The sigmoid function forms the input layer which takes the output of the last LSTM unit (ht1) at time t=1 and the current input (x t ) at time t, which determines the process of detecting and excluding data.The sigmoid function also determines which aspect of the earlier output could be removed.Tanh layer is an added new vector (ĉ) to update state value.h t-1 and x t form the forget gate (Figure 4) and returns 0 and 1 as the value, 1 representing storing the information and 0 representing removing the information from the cell.The output of forget gate from Figure 2 is:

RESULTS AND DISCUSSION
CXR image database accessed from the GitHub site containing three groups of RGB images, normal CXR, COVID CXR and Pneumonia CXR was used for the study.LSTM and VGG16 models were implemented to classify the images.The performance of each model in terms of confusion matrix, receiver operating characteristics curve (ROC), training and validation accuracy loss curve are explained below.

Performance measure in terms of confusion matrix
The confusion matrix of each model was assessed (Table 1 and 2).It is a matrix that measures the ability of a deep learning model to classify images (as normal, COVID-19, and pneumonia in this case).Being a three-class problem a 3*3 confusion matrix where the first cell represented the true positive samples was implemented.The confusion matrix contained four parameters as True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN) as detailed below:

FP
The false positive value was the sum of the values of the corresponding column except the TP value.It accounted for all COVID-19 and pneumonia negative cases misdiagnosed as positive cases.

TN
The true negative value was the sum of all columns and rows excluding the values of the class for which the values were being calculated.It indicated the number of healthy patients that were actually not suffering from either COVID-19 or pneumonia.
The confusion matrices of each model are presented in Figure 5 and 6.

Model performance in terms of ROC curve
The graph between true positive (TP) and false positive (FP) rates showed the ability of deep learning models to classify.More the area covered by the curve more shall be the accuracy of the prognosis by a model.ROC curves of each model are shown in Figure 7 and 8. VGG16 model performed better than LSTM in prognosing COVID-19 and pneumonia patients with a ROC value of 100% and 99%, respectively (Figure 8).LSTM model prognosed COVID-19, pneumonia and normal patients with 97%, 98% and 96% ROC values, respectively.The performance metrics formulae of DL models are as bellow:

Performance of models in terms of precision, recall and F1 score
Performance parameters of the models for the image database are provided in Table 3.The precision, recall and F1 score for VGG16 and LSTM models were obtained from Table 3. VGG16 returned reliable precision (0.95), recall (0.95) and F1 score (0.95) values for COVID-19 prognosis.For normal classification, the values obtained were precision: 0.91, recall: 0.95 and F1 score: 0.93.For pneumonia prognosis, the values obtained were precision: 0.95, recall: 0.90 and F1 score: 0.93.Compared to VGG16 the performance of LSTM model on precision (0.86), recall (0.82) and F1 score (0.84) in COVID-19 prognosis was relatively low, and the model returned precision, recall and F1 score values of 0.95, 0.86 and 0.90 respectively for normal classification.The performance of LSTM model for pneumonia classification was precision: 0.79, recall: 0.90 and F1 score: 0.84 values, respectively.High recall value indicated lower false negative prognosis values.False negative prognosis may misdiagnose a patient.Among the two test models, VGG16 was suitable for correct COVID and pneumonia prognosis as it exhibited higher recall value.

Model performance in terms of training and validation curve
LSTM model provided 92% validation accuracy and 88% training accuracy after 200 epochs (Figure 9).Validation loss of 0.01 and training loss of 0.001 were obtained at 11 epochs.After 11 epochs validation accuracy was superior to the training accuracy.Validation accuracy was at its maximum value at 35 epochs.At 200 epochs, 0.2 validation loss and 0.1 training loss were obtained.
VGG16 model returned best performance compared to LSTM model (Figure 10).The training accuracy showed better performance than validation accuracy initially for VGG16 model.

Comparative analysis
The performance and reliability of the proposed systems were compared with the stateof-art COVID-19 probing systems.Outcome of the first version of the proposed system is presented and compared with conventional methods in Table 4.The proposed system demonstrated remarkably accurate results compared to conventional methods.Also, the proposed improved Resnet-50 system is lightweight as compared to models like VGG16 or DenseNet.The proposed system outperformed the existing methods on precision, accuracy, sensitivity and F1 score.Table 4 lists the findings of the autoprognosis of COVID-19 using CXR images which was contrasted with the suggested model.Narin et al. 22 used 26 million parameters for COVID-19 prognosis, the two improved models outperformed other models in terms of accuracy (Table 4).Proposed VGG16 model used roughly five times as many (134 million parameters).This model costs more while is marginally higher on accuracy (97.5%) compared to other models.

CONCLUSION
This paper demonstrates application of VGG16 and LSTM deep learning techniques to CXR images for COVID-19 prognosis, using public repository dataset for multi-class and binary classification tasks.The LSTM model provided 98% validation accuracy and 92% training accuracy while the respective figures were 100% and 98.94% for VGG16.The validation loss of 0.01 and training loss of 0.001 were observed at 11 epochs in LSTM while the respective figures were 0.05 after 11 epochs and 0.12 after 145 epochs for VGG16.Proposed models exhibited higher degree of accuracy compared to previously reported works.A significant finding of the work was that, data fusion models can further increase diagnosis and prediction performances and build an effective model with larger database.The models would assist the clinicians in prognosis of COVID-19 critical patients effectively in few minutes that will be very helpful to empower the healthcare infrastructure especially in overwhelming healthcare emergency situation as the ongoing COVID pandemic.Early diagnosis could crucially curb viral transmission and facilitate in containing the infection.The authors plan to work on multi-criteria classification as an extended research work to decipher images with mixed lung infections due to several simultaneous infections like tuberculosis, AIDS, COVID-19, etc.

Figure 9 .Figure 10 .
Figure 9. Accuracy and loss curve of LSTM model 18developed CNN architecture for early automatic diagnosis of COVID-19 cases.They proposed the CNN-based approach for prognosis by utilising three different views of the CXR image (left lung, right lung and overall view) to extract individual features.The proposed model is effective for binary classification of healthy and COVID-19 cases.Haque and Abdelgawad 19 reported a custom convolutional neural network model to detect COVID-19 with 98.3% accuracy.A major issue of this model is, it cannot differentiate COVID-19 cases from other non-COVID cases (like pneumonia).Ghoshal and Tucker 20 applied a Bayesian convolutional neural network (BCNN) on publicly available COVID-19 CXR images.Results demonstrated an improved (85.2-92.9%)detection accuracy using VGG-16 model.Islam et al. 21proposed DL technique to automatically detect COVID-19 from CXR images by combining CNN and LSTM.This model achieved 99.4% accuracy with high degree of sensitivity and specificity.Narin et al. 22 used InceptionV3, ResNet50 and InceptionResNetV2 models to categorise CXR images into COVID-19 and normal.

Table 1 .
Confusion matrix report of LSTM model

Table 3 .
Performance parameter values of deep learning models

Table 4 .
Deep learning methods and techniques used in COVID-19 using CXR images