Journal of information and
communication convergence engineering
(J. Inf. Commun. Converg. Eng.)

Research on the cloud and its characteristics plays a very important role for many applications: e.g., climate modeling, weather prediction, meteorology study, solar energy production and satellite communication [1-6]. Cloud classification is an essential role in cloud observation. However, at present, existing cloud classification is done by professionally trained observers. This method is highly time consuming, and depends on the experience of observers; moreover, there are some problems that cannot be well handled by the human observers [7]. Therefore, automatic classification of the cloud is a much-needed task.

Much research on classification of cloud images has been conducted. Buch and Sun [8] applied binary decision trees to classify pixels in the whole sky imager (WSI) images into five cloud types. Singh and Glennen [9] proposed five different feature extraction methods (autocorrection, co-occurrence matrices, edge frequency, Law’s features and primitive length) and used the k-nearest neighbor and neural network for cloud classification. Calbo and Sabburg [10] applied a Fourier transform for cloud-type recognition. Heinle et al. [11] predefined several statistical features to describe color and texture and used a k-nearest neighbor classifier. Liu et al. [12] used an illumination-invariant completed local ternary pattern descriptor for cloud classification. Liu et al. [13] extracted some cloud structure features and used a simple classifier called the rectangle method for classification of infrared cloud images. Liu et al. [14] proposed a salient local binary pattern for cloud classification. Liu et al. [15] used a weighted local binary descriptor. Dev et al. [16] proposed a modified texton-based approach to categorize cloud image patches. Luo et al. [17] combined manifold features and text features and then used a support vector machine (SVM) to classify cloud images. Gan et al. [18] proposed cloud type classification using duplex norm-bounded sparse coding. Nevertheless, most of these approaches are based on hand-crafted features so each method needs to find its empirical parameters.

Recently, development of deep learning is increasing rapidly; in particular, convolutional neural networks (CNNs) have shown outstanding performance in image classification [19]. CNNs are able to “learn” features from the image data, so there is no need any feature extraction method. Some recent studies used CNNs for cloud classification. Shi et al. [20] proposed a CNN model to extract features and used SVM to classify cloud images. Ye et al. [21, 22] improved feature extraction by using both CNN and Fisher vector and used SVM to classify cloud images. Zhang et al. [23] proposed transferring deep visual information. Generally, these deep learning approaches achieved promising results; however, all the above deep learning approaches only utilized the CNN to extract features. They needed other methods for classification, and, they used pre-trained CNN models to extract features. However, for small datasets, the lack of sufficient image samples makes it difficult to converge in the end-to-end learning manner.

This paper proposes a deep learning approach for classification of cloud images on small datasets. First, we design a deep learning model that includes both the feature extraction part and classification part using CNN, and then we apply two regularization techniques, data augmentation and dropout, to generalize our model. Our experiment is performed on the SWIMCAT dataset.

The Singapore Whole-sky IMaging CATegories (SWIMCAT) dataset was introduced by Dev et al. [16] and the images were captured during 17 months from January 2013 to May 2014 in Singapore using the Wide Angle High-Resolution Sky Imaging System (WAHRSIS), a calibrated groundbased whole-sky imager [24]. The dataset has five distinct categories. The five categories (clear sky, patterned clouds, thick dark clouds, thick white clouds, and veil clouds) are defined on the basic of visual characteristics of sky/cloud conditions and consultation with experts from the Singapore Meteorological Services.

The SWIMCAT dataset has 784 images of sky/cloud patches: clear sky, 224 images; patterned cloud, 89 images; thick dark cloud, 251 images; thick white cloud, 135 images; and veil cloud, 85 images. The dimensions of all the images are 125 × 125 pixels. Some random images of each class from the SWIMCAT dataset are shown in columns in Fig. 1.

Figure 1.

Sample images of each class from the SWIMCAT dataset.

A. Convolutional Neural Networks

CNNs were inspired by the human visual system [25, 26]. They are the state-of-the-art approaches not only for pattern recognition tasks but also for object detection tasks, especially with the development of computing capacity. Krizhevsky et al. [19] won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 competition [27, 28] with brilliant deep CNNs that show the great power of deep CNNs.

Unlike many other pattern recognition algorithms, CNNs combine both feature extraction and classification. A schematic representation of a basic CNN (inspired by [29]) is shown in Fig. 2. The given network consists of five different layers: input, convolution, pooling, fully-connected, and output. The input layer specifies a fixed size for the input images, i.e., images may have to be resized accordingly. The image is then convolved with multiple learned kernels using shared weights. Next, the pooling layers reduce the size of the image while trying to maintain the contained information. These two layers comprise the feature extraction part. Afterwards, the extracted features are weighted and combined in the fully-connected layers. This represents the classification part of the CNN. Finally, there is one output neuron for each object category in the output layer.

Figure 2.

Block diagram of a CNN.

B. Model Design

As we mentioned above, the architecture of CNNs follow this pattern:

(1)$$IN\Rightarrow CONV\Rightarrow POOL\Rightarrow FC\Rightarrow OUT$$

where IN is the input layer, CONV is the convolution layer, POOL is the pooling layer, FC is the fully connected layer, and OUT is the output layer. However, for “deep learning” these layers are stacked together in a particular pattern that yields a CNN model. Basically, the most common form of CNN architecture is that many layers of CONV and POOL are repeated until the volume, width and height is small, at which point we apply some FC layers. The most common CNN architectures have the following pattern [30]:

(2)$$IN\Rightarrow \left[CONV\Rightarrow POOL?\right]*M\Rightarrow \left[FC\right]*N\Rightarrow OUT$$

where “*” indicates repetition and “?” indicates an optional pooling layer. For simplicity, we do not mention activation, but by default, the activation always follows CONV layers and FC layers. Theoretically, the larger number of convolutional layers (M is large) extracts more detailed features of input images; however, it needs a large amount of learning data. In this study, we selected less number of convolutional layers (M is small) and increased the generalization of our model by applying regularization techniques.

We selected M = 3 and N = 2, and the feature extraction part consists of three groups of a CONV layer and a POOL layer. The classification part consists of two fully connected layers.

C. Model Regularization

In this research, we used the SWIMCAT dataset which consists of only 784 image patches, which is a very small number for deep learning and makes it very easy to get overfitting. To overcome this problem we used two regularization methods. The first augments data passed into the network for training, and the second modifies the network architecture. They are data augmentation and dropout, respectively.

A. K-Fold Cross-Validation

The gold standard for machine learning model evaluation is k-fold cross-validation. In this study we randomly split our data into 5 partitions (k = 5) of equal size. For each partition i, we trained our model on the remaining four partitions, and tested it on partition i. The final score was the averages of all 5 scores obtained. The schematic of our 5-fold cross-validation is shown in Fig. 4.

Figure 4.

Schematic of 5-fold cross-validation.

B. Experimental Environment

The hardware we used to implement the proposed network was a single PC Intel Core i5-7500 with the graphics processor NVIDIA GeForce GTX 1060 with 4 GB memory equipped. The software was developed using Python programming language based on Keras deep learning library [32] with back-end as TensorFlow [33].

C. Experimental Results

We experimented on our approach as described in Section III on SWIMCAT database. We used batch size of 32 images, and RMSProp optimizer. The experimental result of five test sets after running 1,000 epochs for each fold, are shown in Table 3. We obtained an average accuracy of 0.986 with minimum and maximum accuracies at 0.975 and 0.994, respectively.

Table 3 .

Experimental results

No.	Fold	Accuracy (%)
1	Fold 1	98.73
2	Fold 2	99.36
3	Fold 3	99.36
4	Fold 4	97.45
5	Fold 5	98.06
6	Average	98.59

The confusion matrixes are shown in Fig. 5. In each confusion matrix, each column of the matrix represents the instances in a predicted class, and each row represents the instances in the actual class. Our proposed approach achieves perfect classification accuracy for most classes. The sky, patterned clouds and thick dark cloud classes always achieve 100% accuracy for all 5 folds.

Figure 5.

Confusion matrixes using the proposed approach.

This paper presented a deep learning solution for classification of cloud image patches on small datasets. For this research we used the dataset SWIMCAT, which consist of only 784 images. That is very small for deep learning applications, and it is very easy to get overfitting. To solve this problem, we designed a CNN model with enough convolutional layers for small datasets, and we applied two regularization methods, data augmentation and dropout to thoroughly treat the overfitting problem. In addition, the gold standard for the machine learning model, the k-fold cross-validation was also applied. Experimental results confirmed that we achieved perfect classification accuracy for all classes and all folds, with a minimum accuracy of 97.45% and maximum accuracies of 99.36%. These results prove that the proposed model not only achieves high accuracy but is also robust.