Journal of information and
communication convergence engineering
(J. Inf. Commun. Converg. Eng.)

The industrial structure has been changing dramatically due to the Fourth Industrial Revolution (or Industry 4.0), which dominates the mass surveillance and autonomous driving industries. Vision-based intelligent systems, such as self-driving cars and unmanned aerial vehicles, are being rapidly developed. These life-critical systems adopt highlevel object recognition algorithms to sense their environment and operate without human involvement. However, as the performance of these algorithms is subject to weather conditions, poor visibility resulting from adverse weather can trigger a cascading failure that may lead to unfortunate consequences. Therefore, studies on visibility restoration are essential for autonomous vehicles. In this research direction, haze removal (or, equivalently, image dehazing) has garnered growing interest from researchers because haze is seemingly the most frequently encountered weather in practice. In this context, haze refers to the suspended aerosols in the atmosphere. The particle-particle collision of these aerosols and light photons causes the atmospheric scattering phenomenon, reducing the visibility of captured scenes and rendering haze removal research relevant to visibility restoration.

Haze removal algorithms are generally based on the simplified Koschmieder model [1], which describes hazy image formation as follows:

where I represents the input image, J the scene radiance, t the transmission map, A the atmospheric light, and x the pixel coordinates. Assuming that H and W are the image height and width, respectively, I, J, and A take on values in ${ℝ}^{H\times W\times 3}$, whereas $t\in {ℝ}^{H\times W}$. According to (1), recovering J is an ill-posed problem because I is the only observation. Thus, early attempts in haze removal solved this problem by using multiple input images. However, as it is burdensome to acquire such input data, researchers have shifted their interest to single-image haze removal.

According to a recent systematic review [2], this haze removal category can be further partitioned into three subcategories: image processing, machine learning, and deep learning. Concerning the first, the dark channel prior (DCP) proposed by He et al. [3] is typical. The DCP states that outdoor non-sky images exhibit an extremely dark channel, whose intensity approximates zero in local patches around all pixels. They then adopted computationally intensive soft matting to refine the transmission estimate. This method demonstrated good performance in general, but it substantially prolonged the execution time due to the inherent problem in soft matting. Also, it is subject to color distortion when the input image contains a broad sky or shady objects. These limitations brought a large room for improvements, and many follow-up studies have been proposed. For example, Kim et al. [4] reduced the computational complexity by using the modified hybrid median filter—equipped with excellent edge-preserving characteristics—to eliminate the refinement step. This elimination then favored a fast and efficient hardware implementation [4,5].

In the second subcategory, a typical work is the color attenuation prior (CAP) proposed by Zhu et al. [6]. The CAP was also discovered through extensive observations on outdoor images. It states that the scene depth is closely correlated with the difference between the saturation and the value. Zhu et al. [6] modeled this correlation using a linear model, whose parameters were estimated utilizing the maximum likelihood estimates (MLE). The CAP provides a fast and effective haze removal, albeit with color distortion and background noise. In a follow-up study, Ngo et al. [7] addressed these two problems using adaptive weighting and low-pass filtering.

Finally, deep-learning techniques, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), have also found their applications in haze removal. The pioneering work of Cai et al. [8] can be taken as a prime example. They proposed a well-performed three-layer CNN for estimating the transmission map from a single input image. In subsequent work, Li et al. [9] employed serial multiscale mapping to design a CNN that estimates and refines the transmission map from coarse to fine scales. Although deep-learning-based haze removal methods generally deliver satisfactory performance, they are subject to the domain-shift problem.

This paper presents a machine-learning-based method that improves the CAP by considering two new haze-relevant features in addition to the saturation and value. More precisely, we estimate the scene depth as a linear combination of local entropy, dark channel, saturation, and value. We then present a comparative evaluation with other state-of-the-art benchmark methods to verify the efficacy of the proposed haze removal algorithm. Furthermore, we demonstrate that the software implementation per se cannot satisfy real-time processing requirements. Consequently, we design a 4K-capable hardware accelerator that can handle 4K videos at 30.8 frames per second (fps).

The rest of this paper is structured as follows. Section 2 explores the haze-relevant features and describes the proposed algorithm in detail. Section 3 presents the comparative evaluation with benchmark algorithms, and Section 4 demonstrates the necessity of a hardware accelerator for real-time processing. After that, Section 5 provides a detailed description of the proposed hardware design and interprets the implementation results. Finally, Section 6 concludes the paper.

A. Haze-relevant Features

Under the single image dehazing approach, most algorithms estimate the transmission map in two major steps: feature extraction and regression. On the one hand, these two are easily noticeable in image-processing and machine-learning- based methods. For example, He et al. [3] calculated the normalized dark channel (feature extraction) and subtracted it from unity (regression) to estimate the transmission map. On the other hand, deep learning-based methods usually introduce the multiscale mapping between these two steps to improve robustness against spatial variance in the input image. This observation demonstrates the fundamental importance of haze-relevant features in haze removal. Recently, Ngo et al. [10] explored and summarized the haze-relevant features hitherto reported in the literature. In addition, they also verified the correlation between those features and the haze distribution using representative hazy and haze-free image patches extracted from well-publicized datasets. Some of the verification results—corresponding to the saturation, value, dark channel, and local entropy— is illustrated in Fig. 1, where Figs. 1(c) and (d) are adopted from [10]. The normalized histograms demonstrate that feature values follow the normal distribution, where the means of the hazy and haze-free distributions are well separated. Also, based on the degree of overlap, it is observed that the dark channel exhibits the strongest correlation with haze distribution, followed by saturation, value, and local entropy.

Figure 1. Normalized histograms of four haze-relevant features: (a) saturation, (b) value, (c) dark channel, and (d) local entropy.

Inspired by the work of Zhu et al. [6], we also utilize a linear model to estimate the transmission map from the saturation, value, dark channel, and local entropy. The reason for using two additional features comes from the observing normalized histograms in Fig. 1. It is conspicuous that each feature correlates with the haze distribution in a different way. In addition, there are currently no features with a perfect correlation. Saturation, value, dark channel, and local entropy do not represent the haze distribution in particular circumstances. The breakdown of the dark channel in sky regions or shady objects is a prime example. Therefore, using multiple features allows the mutual compensation for their failures. The sky region is haze-free in the previous example, but its all-channel high intensities result in high dark channel values. Based on the dark channel, the sky region is misclassified as densely hazy instead of haze-free. However, as this region is also textureless, its haze condition can be recognized using the local entropy. So, this example demonstrates that the local entropy can compensate for the failure of the dark channel in the sky region.

B. Scene Depth Estimation

As discussed earlier, we improved the work of Zhu et al. [6] to estimate the scene depth from the saturation, value, dark channel, and local entropy using a linear model. This model is illustrated in (2), where d denotes the scene depth, f₁ saturation, f₂ value, f₃ dark channel, and f₄ local entropy. The corresponding parameters are θ₁, θ₂, θ₃, θ₄, while θ₀ represents the bias. The variable ε denotes the model error, and we assume that it follows the normal distribution with zero mean and σ2 variance. According to the characteristics of the normal distribution, the scene depth is also normally distributed with (θ₀ + θ₁f₁ + θ₂f₂ + θ₃f₃ + θ₄f₄) mean and σ2 variance.

Subsequently, we leverage the MLE technique to determine the parameters that maximize the likelihood function [11], wherein the synthetic training dataset is prepared as follows. We utilize the 500IMG dataset [11] whose 500 constituent haze-free images are collected from free image-sharing services. Then, we employ the enhanced equidistribution [11] to create the random depth maps, which serve as the ground-truth references in the training dataset. We also draw the random atmospheric light—whose values range from 0.8 to 1—from the enhanced equidistribution. Given the scene depth, we use the following (3) to calculate the transmission map.

where ${\text{β}}_{sc}$ is normally one as the atmospheric scattering coefficient. Because the transmission map and atmospheric light are now available, we substitute these two into (1) to produce the hazy synthetic images, whose saturation, value, dark channel, and local entropy serve as the inputs in the training dataset.

We then apply the mini-batch gradient ascent algorithm [11] on the training dataset created above to estimate the parameters. The best estimates that we obtained are θ₀ = −0.5570, θ₁ = 1.5210, θ₂ = 0.9042, θ₃ = 0.7543, and θ₄ = −0.3685. It is worth noting that this parameter estimation step is performed offline, so it does not affect the run-time of the proposed method.

C. Atmospheric Light Estimation

Researchers usually adopted the atmospheric light estimation (ALE) method of He et al. [3], which locates the atmospheric light in the “most opaque” region. He et al. [3] defined those pixels whose dark channel value is within the top 0.1% of that region. Then, the pixel with the highest intensity in the red-green-blue color space was selected as the atmospheric light.

In a different approach, Tarel and Hautiere [12] assumed that the atmospheric light was pure white if the input image was correctly white-balanced. However, this ALE method and even that of He et al. [3] are prone to incorrect estimation when the input image contains bright objects, such as white cars or light bulbs. The quad-decomposition algorithm proposed by Park et al. [13] is a good alternative. The input image is now recursively partitioned into quarters based on the average luminance. This partition procedure can eliminate bright objects effectively because of their high contrast to the background. Nevertheless, as the partition requires many frame buffers, the quad-decomposition algorithm is inefficient in memory usage. Therefore, Ngo et al. [11] developed an approximated version that is free of frame buffers. So, in this study, we utilize the approximated quaddecomposition method to estimate atmospheric light.

After that, we substitute the estimates of transmission map and atmospheric light into (1) to recover the scene radiance. Finally, we adopt the adaptive tone remapping method of Cho et al. [14] to post-process the recovered image.

This section compares the performance of the proposed method against four benchmark algorithms, including those proposed by Tarel and Hautiere [12], Zhu et al. [6], Kim et al. [4], and Ngo et al. [7]. Henceforth, we refer to these four as Tarel, Zhu, Kim, and Ngo, respectively. For comparison, we employ three full-reference metrics: structural similarity (SSIM) [15], feature similarity extended to color images (FSIMc) [16], and tone-mapped image quality index (TMQI) [17]. These metrics take on values ranging from zero to unity, wherein higher values signify a better performance. Also, we use two real datasets (I-HAZE [18] and OHAZE [19]) that comprise 30 and 45 pairs of hazy and haze-free images, respectively. Table 1 shows the average SSIM, FSIMc, and TMQI scores on I-HAZE and O-HAZE datasets, and the best results are displayed in bold. It can be observed that the proposed algorithm is the best performing under SSIM and FSIMc, regardless of whether input images are indoor or outdoor. Additionally, the performance gap between the proposed method and Zhu is easily noticeable, attributed to the use of two new haze-relevant features. The saturation, value, dark channel, and local entropy can compensate for one another, boosting performance when saturation and value fail to represent the haze distribution. So, in general, the proposed algorithm can be considered superior to the four benchmark algorithms. Fig. 2 shows hazy images and corresponding dehazing results obtained from the four benchmark methods and the proposed algorithm. The first row shows the dehazing results of a hazy image from the IVC dataset [20], which consists of 25 real hazy images. This dataset was excluded from the quantitative evaluation because it does not contain ground-truth references. In the second and third rows, haze removal was performed on images from the I-HAZE and O-HAZE datasets, respectively. It can be observed that Tarel exhibits excellent performance, but color distortion arises in the sky region. Meanwhile, the results of Zhu hinder object recognition due to excessive haze removal. In the results of Kim, the performance is average, and color distortion also arises in the upper part of the IVC and O-HAZE images. Conversely, results of Ngo are satisfactory without visually unpleasant distortion. However, in the IVC and I-HAZE images, the dehazing power is too strong, leading to the occurrence of black pixels, as witnessed in the dog’s fur and the bottom of the sofa. Finally, the proposed method removes haze effectively and well-preserves the dog’s fur color. In addition, in the I-HAZE and O-HAZE images, the dehazing results are more satisfactory than those of the benchmark methods.

Table 1 . Average structural similarity (SSIM), feature similarity extended to color images (FSIMc), and tone-mapped image quality index (TMQI) scores on I-HAZE and O-HAZE. The Best results are displayed in bold..

Dataset	I-HAZE			O-HAZE
Method	SSIM	FSIMc	TMQI	SSIM	FSIMc	TMQI
Tarel	0.7200	0.8055	0.7740	0.7263	0.7733	0.8416
Zhu	0.6864	0.8252	0.7512	0.6647	0.7738	0.8118
Kim	0.6424	0.7879	0.7026	0.4702	0.6869	0.6509
Ngo	0.7600	0.8482	0.7892	0.7322	0.8219	0.8935
Proposed	0.7642	0.8658	0.7878	0.7329	0.8920	0.8351

Figure 2. Qualitative comparison with other haze removal methods on the IVC, I-HAZE, and O-HAZE datasets.

For an image processing algorithm to be deployed in realworld systems, it should handle image data at a minimum rate of 25 fps or greater, depending on whether the color encoding standard is PAL or NTSC [21]. Therefore, we conducted a run-time comparison between several haze removal algorithms and tabulated the results in Table 2. The simulation environment is MATLAB R2019a , running on a host computer with Intel Core i9-9900K CPU, NVIDIA TITAN RTX GPU, and 64GB RAM. It can be observed from Table 3 that none of the algorithms can handle images in real-time. This finding suggests that hardware implementation is essential for coping well with the real-time processing requirement.

Table 2 . Run-time comparison of haze removal algorithms (in seconds) for three image sizes..

Size	640 × 480	1024 × 768	4096 × 2160
Method
He	12.64	32.37	470.21
Tarel	0.28	0.76	9.02
Zhu	0.22	0.55	6.39
Kim	0.16	0.43	4.81
Ngo	0.17	0.44	5.22
Proposed	0.93	2.32	26.95

Table 3 . Hardware implementation result of the proposed hardware design..

Xilinx Vivado 2019.1
Device	Xc7z045-2ffg900
Slice Logic Utilization	Available	Used	Utilization
Slice Register(#)	437,200	64,918	14.85%
Slice LUT(#)	218,600	58,126	26.59%
RAM36E1s	545	58	10.64%
Minimum Period	3.67 ns
Maximum Frequency	272.48 MHz

* The EDA tool was supported by the IC Design Education Center (IDEC), Korea..

Fig. 3 presents the hardware architecture of the proposed method, which can be partitioned into memories, logic circuits, and arithmetic circuits. Two 1024 × 32-bit SPRAMs and three 256×8-bit SPRAMs are used for the atmospheric light estimation [11] and adaptive tone remapping [14]. Other memories are used as line memories for 5 × 5 filtering operations. So, it takes time seven image lines from input to output. In addition, logic circuits consist of 10 modules. The system controller in logic circuits is responsible for inputoutput operations of the image/video data. Saturation, value, dark channel, and local entropy are calculated in parallel in the 4-feature module. Furthermore, to improve the maximum frequency, we utilized split multipliers for large multiplications where operands’ word-length is greater than 16 bits.

Figure 3. Hardware architecture of the proposed haze removal algorithm.

Table 3 summarizes the hardware implementation result in terms of slice registers, LUTs, RAM36E1s, and maximum frequency. Slice registers and LUTs represent the logic areas, whereas RAM36E1s represents the memory area. The proposed design used 64,918 registers, 58,126 LUTs, and 58 RAM36E1s, respectively. The fastest attainable frequency was 272.48 MHz. This information can be then used to obtain the maximum processing speed (MPS):

where f_max denotes the maximum frequency in Table 3; W and H denote the input image’s width and height, respectively; and HB and VB denote the horizontal and vertical blank periods. The hardware was implemented to minimize the number of blank periods corresponding to one pixel and one image line to increase the MPS. It demonstrates that the proposed design can process the DCI 4K video at 30.8 fps, satisfying the realtime processing requirement of 25 fps or greater.

Fig. 4 depicts the C/C++ platform and verification board for the real-world execution. The top and middle thirds of Fig. 4 belong to the platform, whereas the bottom third depicts the system-on-a-chip (SoC) board. Moreover, the upper part of the platform shows side-by-side input-output data for ease of performance verification. The platform control panel is responsible for providing input data to the SoC board.

Figure 4. Hardware verification using a system-on-a-chip evaluation board.

Meanwhile, the algorithm control provides a convenient graphical user interface for configuring the hardware design running on the board. This C/C++ platform is a convenient means for verifying the real-time processing of the proposed hardware design.

A high-performance haze removal algorithm and its corresponding 4K-capable hardware accelerator were presented in this paper. We proposed using two new haze-relevant features (dark channel and local entropy) to estimate the transmission map, based on the observation that they can effectively compensate for the failures of the CAP. In addition, we adopted a frame-buffer-free version of the quaddecomposition algorithm to estimate atmospheric light to reduce hardware resources. We then provided extensive experimental results to demonstrate the superiority of the proposed method over benchmark algorithms. We also conducted a run-time comparison to show that the software implementation per se was insufficient for real-time processing. Therefore, we presented a 4K-capable hardware design that can handle DCI 4K videos at 30.8 fps, rendering the proposed algorithm highly relevant for high quality, highspeed real-time systems, such as autonomous cars and drones.

This research was funded by research funds from Dong-A University, Busan, Korea.