A Novel Multi-focus Image Fusion Method Based on Non-negative Matrix Factorization

In order to efficiently extract the focused regions from the source images and improve the quality of the fused image, this paper presents a novel image fusion scheme with non-negative matrix factorization (NMF). The source images are fused by NMF to construct temporary fused image, whose region homogeneityis used to split the source images into regions.The focused regions are detected and integrated to construct the final fused image. Experimental results demonstrate that the proposedschemeis capable ofefficiently extracting the focused regions and significantly improving the fusion quality compared to other existing fusion methods,in terms of visualand quantitative evaluations.

fused image, but it doesn't appropriate for large scale data sets. Most of the existing fusion methods based on NMF use matrix factorization to extract salient feature of the source images. But NMF is confronted with two main problems which are unsatisfactory accuracy and bad generality. The processed objects of NMF are intrinsically vectors and necessaryvectorization for every matrix in the processed matrix-set, which often make the corresponding NMFbe atypical kind of small-sample learning and compromises the ability of NMF in generalization and feature representation [13].
Different from the methods mentioned above, this paper presents a novel NMF based image fusion scheme. NMF is used to construct the temporary fused image and extract the underlying salient information fromthe source images. To inhibit blocking artifacts, the source images are split based on the region homogeneity of the temporary fused image. The objective of this paper is to improve theefficiency and performance of the fusion method. The construction of temporary fused image with NMF for effective image fusion is the main contribution ofthis paper. The proposed method can efficiently extract the focused regions details from the source images and improve the visual quality of the fused image.
The rest of the paper is organized as follows. In Section 2, the basic idea of NMF will be briefly described,followed by the new method with NMF for image fusion in Section 3. In Section 4, extensive simulations are performed to evaluate the performance of the proposed method. In addition, several experimental results are presented and discussed. Finally, concluding remarks are drawn in Section 5.

Non-negative Matrix Factorization
NMF incorporates the non-negativity constraint and thus obtains the parts-based representation as well as enhances the interpretability of the issue correspondingly [14],which is a low-rank approximation technique for un-supervised multivariate data analysis and produces non-negative matrix to process an image [15]. NMF factorizes a n m  original matrix V into two factor matrices. One is a n r  non-negative basis matrix W and the other is a r m  nonnegative weight matrix H .
The two factor matrices can approximate the original matrix V according to some cost functions. The convergence of the algorithm has been proved by Lee and Seung [5], [6].
Recent years, several variants of NMF such as LNMF [16], sparse non-negative matrix factorization (SNMF) [17] and non-negative matrix factorization with sparseness constraints (NMFsc) [18] have been proposed to improve NMF from various perspectives. These extensions are mainly performed on modified models, modified constraints and modified cost functions. One important variable in the reduction of dimensions in each NMF method is commonly called the variable r (number of the basis vector). How large a data matrix will be reduced is determined by r . The larger value of r the smaller dimension is reduced and the smaller value of r the larger dimension is reduced [19].Due to NMF is a method to find a partbased representation of original data, the source images are linear and non-negative combinations of the r basis images derived from NMF on the source images when the parameter of NMF is set to r ( 1 r  ). Similarly, the source images are linear and non-negative combinations of the only one parts derived from NMF on the source images when 1 r  . The only part derived from NMF with 1 r  on the source images is the substantial feature of the source images. The substantial feature of the source images can be seen as the global feature of the source images. The fused image can be obtained from the observed images by using

Multi-focus Image Fusion Based onNMF 3.1. Fusion Algorithm
In this section, a novel fusion algorithm using NMF is proposed. The proposed fusion framework is depicted in Figure 2. 0 I is the temporary fused image of A I and B I . For the sake of simplicity, this paper assumes that there are only two registered source images, namely A I and B I , respectively. The rationale behind the proposed scheme applies to the fusion of more than two multi-focus images. The source images are assumed to be pre-registered and the image registration is not included in the framework. The fusion algorithm consists of the following three steps: Figure 2. Block diagram of proposed multi-focus image fusion framework Step  . The parameter r is set to 1. Then, the size of the basis matrix W is reset to the same size as the source images, and is used as the temporary fused image.
For more than two source images, the source images are transformed to the column vectors 1 V , 2 V ,…, 1 N V  and N V , and combined together to represent the input matrix V . The input matrix V is defined as follows: The parameter r is set to 1.
Step 2: Partition the temporary fused image 0 I into blocks by using the region homogeneity of 0 I . A I and B I are split based on the split results of 0 I , respectively.
Step 3: According to the fusion rules, the focused regions of the source images which corresponding to the salient regions of 0 I are integrated to construct the final fused image.

Fusion Rules
There are two key issues [20] involved with the fusion rules. The first is how to measure the activity level of the source images, which recognizes the sharpness of the source images. We use the energy of image gradient (EOG) to measure the activity level of the source images. The EOG of each image block can be defined as:  (4) where ( , ) I i j indicates the value of the element location ( , ) i j in the image block. The other is how to integrate the focused pixels or blocks of the source images into the counterparts of the fused image. Thus, the block with a larger EOG is chosen to construct the fused image. However, the fixed block size will lead to non-smooth transitions between blocks. In order to reduce the blocking artifacts, quad tree decomposition [21] is applied to block division. The division is first performed on the low resolution image,and then the subdivision is performed on the high resolution image based on the division of the low resolution image. Quad tree decomposition can adaptively control the block size of the subdivision of the image based on the region homogeneity of the block. Figure3indicates the decomposition of the image "Lena". It is obvious that the salient features such as edges and textures of Figure3 (a) are corresponding to the salient feature of Figure3 (b). To overcome the disadvantages of the small block in traditional block-based image fusion method, the minimum block size is set for terminating the further division when the region homogeneity of the block doesn't meet the threshold condition. The region homogeneity is defined as: is the value of the element location ( , ) i j in the image block. T is the threshold condition. In this paper, quad tree decomposition is performed on the temporary fused image and the threshold condition is set as 0.005, and the minimum block size is set as 8 8  .
where"1" in H indicates the pixel location ( , ) i j in the source image A I is in focus, while "0" in H indicates the pixel location ( , ) i j in the source image B I is in focus. However, judging by EOG alone is not sufficient to detect all the focused blocks. There are thin protrusions, narrow breaks, thin gulfs and small holes in H . To overcome these disadvantages, morphological operations [22] are performed on H . Opening, denoted as H Z o , is simply erosion of H by the structure element Z , followed by dilation of the result by Z . This process can remove thin gulfs and thin protrusions. Closing, denoted as H Z  , is dilation followed by erosion. It can join narrow breaks and thin gulfs. To correctly judge the small holes, a threshold is set to remove the holes smaller than the threshold.In this paper, the structure element Z of the proposed method is a 8 8  matrix with logical 1's and the threshold is set to 1000. Thus, the final fused image is constructed according to the rule as follows: where the ( , )

Experimental Results
In order to evaluate the performance of the proposed method, several experiments are performed on two pairs of multi-focus images [23]differing in content and texture, as shown in X5570 and 48G memory. For comparison, beside the proposed method, some existing multifocus image fusion methods are also implemented on the same set of source images. These methods are spatial frequency (SF) (Li's method [24]), NMF (Zhang's method [7]), LNMF (Ye's method [10]), SNMF [17] and NMFsc [18]. Due to the lack of original source code, this paper uses the Eduardo Fernandez Canga'sMatlab image fusion toolbox [25] as the reference for SF. The NMF toolbox [26] is used as the reference for NMF, LNMF, SNMF and NMFsc. The parameters of LNMF are set as .In order to quantitatively compare the performance of the proposed method with that of the methods mentioned above, two metrics are used to evaluate the fusion performance: (i) Mutual information (MI) [27,28], which determines the degree of dependence of the source images and the fused image, and (ii) [29], which measures the amount of edge information transferred from the source images to the fused image. In these metrics, a larger value indicates a better fusion result.   Figures 5 (b-e). The contrast of Figure 5 (e) is better than that of Figures 5 (a-d). These blurs also appear in the fused image in Figures 6(a-e), respectively, such as the cover of the left book in Figures 6 (a), (c) and (e), the edge between the two books in Figure 6 (b), and the upper edge of the left book in Figures 6 (d). The obvious blocking artifacts appear in the fused image obtained by SF, such as the upper edge of the clock in the left door frame in Figure 5 (a), and the cover of the left book in Figures 6 (a). In addition, the blocking artifacts also appear in the difference images in Figure 7 (a). There are some obvious residuals in the difference images obtained by the extension of the NMF-based fusion methods in Figures 7 (b-e), such as the right regions in Figures 7 (a), (c) and (d), the center region in Figure 7 (b). It should be noted that the residuals in Figure 7 (e) is so much that it can see the content of the right book's cover. The right region in Figure 7 (f) is smooth and flat. Upon inspecting the fused images in Figures 5 (af) and 6 (a-f), it is easy to see that the contrast of the fused image obtained by the proposed method is better than that of the fused images obtained by the other fusion methods. Therefore, the fused images of the proposed method achieve superior visual performance by containing all of the focused contents from the source images without introducing artifacts.

Quantative Analysis
For quantitative analysis, the quantitative results of the two quality measures are shown in Table 1. The running times are also shown in Table 1 values of NMF, LNMF and SNMF are almost the same and higher than that of NMFsc. The running times of NMFsc is longer than that of the other methods, which lies in the computational cost of the sparseness of the basis matrix.It can be seen that the proposed method requires longer computational time than the other methods, except for LNMF, SNMF and NMFsc. The drawback of high computational cost lies in that temporary fusion of the source images accounts for the majority of the computational load.

Conclusion
This paper proposes a novel multi-focus image fusion method with NMF to enhance the validity of focused regions extraction and blocking artifacts inhibition. The qualitative and quantitative evaluations have demonstrated that the proposed method can produce better fused image and significantly inhibit the blocking artifacts. In the future, we will consider optimizing the proposed method to reduce time consumption and improve the method's adaptability.