Exploring the effect of in-domain versus out-of-domain transfer learning for medical image classification

Purpose: Transfer learning has been widely used to improve the machine learning model’s performance in medical imaging tasks. The medical imaging domain differs significantly from natural images. They are generally grayscale, noisier, and have complex shapes in 2 and 3 dimensions. These domain differences may undermine the model’s robustness from transfer learning. Here, we aim to investigate the impact of transfer learning with in-domain RadImageNet dataset and out-of-domain ImageNet on the robustness of classifying thyroid nodules as benign or malignant.

Methods/Materials: We conducted our study using two datasets. The publicly available RadImageNet dataset, and the retrospectively collected ultrasound images of single thyroid nodules of 721 patients from our institute from 2006-2016. We trained a Resnet-18 model from scratch with the RadImageNet dataset consisting of 1.35 million medical images for classifying 165 labels. We collected the best weights from this training for our in-domain transfer learning. We split our institutional data and held 145 cases in a test set and 576 cases for a 5-fold cross-validation. A Resnet-18 model was trained with our institutional dataset to classify thyroid nodules into benign and malignant. Then we applied transfer learning from the trained model with RadImageNet. Finally, we conducted transfer learning from the ImageNet dataset as an out-of-domain source of pretrained weights.

Results: The area under the ROC curve (AUROC) was used for our models' performance evaluation. Training the Resnet-18 model on the RadImageNet dataset resulted in an AUROC of 0.98. The AUROC achieved by our model without transfer learning was 0.74. The AUROC of our model after transfer learning using the ImageNet weights was 0.80, and our model achieved the best performance with an AUROC of 0.84 from transfer learning of the RadImageNet weights.

Conclusions: We demonstrate the effectiveness of in-domain transfer learning using the RadImageNet dataset compared to out-of-domain transfer learning and no transfer learning in classifying medical images. This can be useful to increase the model’s performance in scenarios with limited data, which is usually the case with medical datasets.