Remote Sensing, Vol. 16, Pages 4695: Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing

Remote Sensing, Vol. 16, Pages 4695: Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing

Remote Sensing doi: 10.3390/rs16244695

Authors: Zhifei Chen Yang Hao Qichao Liu Yuyong Liu Mingyang Zhu Liang Xiao

Recently, there has been a surge in the adoption of deep learning (DL) techniques, especially convolutional neural networks (CNNs), to perform hyperspectral image (HSI) classification. Although deep learners have been shown to achieve impressive performance in HSI classification, they are known to be extremely sensitive to even slight perturbations to their inputs and models. When applied in safety-critical applications, it is crucial to know how robust they really are against perturbations. However, there is still limited tool support for DL testing in terms of their robustness, nor are the existing RGB testing approaches able to address the HSI-specific challenges. In this paper, we propose a mutation analysis framework specialized for DL models trained to classify HSIs, which facilitates a critical evaluation of the robustness of DL-based HSI classifiers. First, we introduce a set of mutation operators to inject faults into the inputs and models to simulate distortions of remote sensing HSI classifiers. By utilizing the mutation testing technique, we implement a novel framework which supports the multidimensional evaluation of individual DL-based classifiers. Finally, a comparative study of the robustness of seven popular CNN-based HSI classifiers (i.e., 3D-CNN, FDSSC, HybridSN, MCNN, FC3DCNN, DWTDENSE, and Tri-CNN) on six HSI datasets is provided. Results show that FDSSC and Tri-CNN achieve higher robustness in the presence of distortions, and FDSSC maintains a relatively stable level of robustness even with few training samples. These empirical findings can be partly explained by the characteristics of the classifiers’ architectures. The results substantiate the efficacy of our evaluation framework in assessing the robustness of HSI classifiers and thus confirm its contribution to the field of remote sensing image classification.