HAL Id: hal-02517288
https://hal-univ-rennes1.archives-ouvertes.fr/hal-02517288
Submitted on 24 Mar 2020
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Jiulou Zhang, Yunbo Gu, Hui Tang, Xiaoqing Wang, Youyong Kong, Yang
Chen, Huazhong Shu, Jean-Louis Coatrieux
To cite this version:
Jiulou Zhang, Yunbo Gu, Hui Tang, Xiaoqing Wang, Youyong Kong, et al.. Compressed sensing MR
image reconstruction via a deep frequency-division network. Neurocomputing, Elsevier, 2020, 384,
pp.346-355. �10.1016/j.neucom.2019.12.011�. �hal-02517288�
Jiulou Zhang, Yunbo Gu, Hui Tang, Youyong Kong, Yang Chen and Huazhong Shu are all with the Laboratory of Image Science and Technology, Southeast University, Nanjing 210096, China.
Xiaoqing Wang is a Postdoctoral Fellow in Uecker Lab at the Department of Interventional and Diagnostic Radiology, University Medical Center Göttingen.
Jean-Louis Coatrieux is with the Centre de Recherche en Information Biomedicale Sino-Francais, Rennes 35042, France, also with the National Institute for Health and Medical Research, Rennes F-35000, France, and also with the Laboratoire Traitement du Signal et de l’Image, Université de Rennes 1, Rennes F-35000, France.
Abstract—Compressed sensing MRI (CS-MRI) is
considered as a powerful technique for decreasing the scan time of MRI while ensuring the image quality. However, state of the art reconstruction algorithms are still subjected to two challenges including terrible parameters tuning and image details loss resulted from over-smoothing. In this paper, we propose a deep frequency-division network (DFDN) to face these two image reconstruction issues. The proposed DFDN approach applies a deep iterative reconstruction network (DIRN) to replace the regularization terms and the corresponding parameters by a stacked convolution neural network (CNN). And then multiple DIRN blocks are cascaded continuously as one deeper neural network. Data consistency (DC) layer is incorporated after each DIRN block to correct the k-space data of intermediate results. Image content loss is computed after each DC layer and frequency-division loss is gained by weighting the high frequency loss and low frequency loss after each DIRN block. The combination of image content loss and frequency-division loss is considered as the total loss for constraining the network training procedure. Validations of the proposed method have been performed on two brain datasets. Visual results and quantitative evaluations show that the proposed DFDN algorithm has better performance in sparse MRI reconstruction than other comparative methods.
Keywords—Magnetic resonance imaging, compressed
sensing, iterative reconstruction, deep learning, convolutional neural network
1. INTRODUCTION
Magnetic resonance imaging (MRI) is one of the most clinically valuable imaging modalities owing to its multiple advantages such as non-radiation damage, high resolution of soft tissue, rich information with multiple contrasts, etc. However, MRI acquisitions are still slow equipment due to its inherent physical or physiological constraints [1]. It may take minutes for image with high spatial resolutions even in parallel imaging based on multiple coils [2-4], which hinders the usage
of MRI in certain applications such as cardiac and abdominal imaging [5].
In the last decades, compressed sensing (CS) has been used to accelerate MR imaging by employing incoherent sampling schemes and by incorporating different sparsity regularization such as total variation (TV) [6, 7] or wavelet [8, 9]. Although these methods based on fixed sparsity transforms are easy to optimize, their ability to represent complex structure and texture information is restricted. Then some more flexible sparsity representations have also been exploited to further improve the reconstruction, e.g., feature learning of available high quality MR images [10, 11]. However, the above iterative optimization schemes still must face three challenging problems: 1) the high time cost and memory resource requirements; 2) the determination of appropriate regularization terms and their corresponding parameters; 3) the loss of image details resulted from over-smoothing.
Recently, CNN based deep learning methods have been successfully applied in solving many computer vision tasks [12-16]. CS-MRI is analogous to de-aliasing, image segmentation and image in-painting in essence [17-20]. Related attempts for CS-MRI reported up to now can be roughly classified into two types, i.e., expanding network architecture and direct deep network for de-aliasing.
The expanding network architecture is gained from the expansion of iterative process of numerical optimization generally. In [21], alternating direction method of multipliers (ADMM) is unfolded into one deep network to optimize iterative MRI reconstruction, which consists of a set of reconstruction layers, convolution layers, nonlinear transform Jiulou Zhang, Yunbo Gu, Hui Tang, Xiaoqing Wang, Youyong Kong, Yang Chen*, Huazhong Shu*,
Jean-
Louis CoatrieuxCompressed sensing MR image reconstruction via a
deep frequency-division network
* Corresponding authors
E-mail addresses: 101101535@seu.edu.cn (J. Zhang), 15371254460@163.com (Y. Gu), corinna@seu.edu.cn (H. Tang), xiaoqing.wang2010@gmail.com (X. Wang), kongyouyong@seu.edu.cn (Y. Kong), chenyang.list@seu.edu.cn (Y. Chen), shu.list@seu.edu.cn (H. Shu), jean-louis.coatrieux@univ-rennes1.fr (J. L. Coatrieux).
layers and multiplier update layers. Variational network model embedded in an unrolled gradient descent scheme was proposed in [22], in which the prior model as well as the data term weights were learned using an offline training procedure. Although the related parameters like penalty parameters, update rates or shrinkage functions in these implementations can be learned via training data pairs including reconstructions with under-sampled data and full k-space data, its practical effectiveness tends to be compromised by an over-smoothing of the reconstructed MR images.
Direct deep network constructed by arbitrary layers and connection types is designed for de-aliasing, which is trained to map the sparse reconstruction images to the corresponding high quality images reconstructed using full k-space data. In [23], a deep cascade of convolutional neural networks was reported for dynamic sequence reconstruction of cardiac MRI. A method termed “refine generative adversarial network” (RefineGAN) was also developed by combining convolutional auto-encoder, residual networks, generative adversarial networks (GANs) and cyclic loss [24]. Similar to the GAN theory, adversarial loss and content loss including both mean square error (MSE) and perceptual loss defined by VGG networks were combined to enhance the reconstruction accuracy in DAGAN [25]. The automated transform based on a manifold approximation (AUTOMAP) implemented with a deep neural network in [26] demonstrates the potential of manifold learning in sparse representation for reducing artifacts and noises in MRI reconstruction.
The reconstruction time cost has been decreased to an acceptable level, because there is no online iterative updating calculation in deep learning methods. Performance of above neural network based methods has proved the promising of deep learning in CS-MRI for obtaining high image quality. However, the exploitation of deep learning in CS-MRI is still in the preliminary stage, which remains to be further improved.
In this paper, we propose a deep frequency-division network (DFDN) to alleviate another two problems in MRI reconstruction. This DFDN approach first applies a deep iterative reconstruction network (DIRN), in which the regularization terms and involved parameters are substituted by the stacked CNN with multiple layers. And then multiple DIRN blocks are cascaded continuously as one deeper neural network. Data consistency layer is incorporated after each DIRN block to correct the k-space data of intermediate results. Image content loss is computed to ensure the image similarity between intermediate results and references after each DC layer. Furthermore, a frequency-division loss is gained by calculating the high frequency loss and low frequency loss with different weights after each DIRN block separately. The combination of image content loss and frequency-division loss is considered as the total loss for constraining the network training procedure, which can be conducive to preserve more image details.
The rest of paper is structured as follows. The problem formulation is described in Section 2. The main features of the proposed method are detailed in Section 3. The conducted experiments and comparisons with other approaches are given in Section 4. Discussion and perspectives are sketched in
Section 5.
2. PROBLEM FORMULATION
After collecting k-space data from receiving coils, the MR image reconstruction can be considered as a solution of the following linear system:
Fx
y
(1) where F is a Fourier encoding matrix, y and x are the raw data in k-space and MR image to be reconstructed, respectively. The inverse problem can be directly solved by the inverse fast Fourier transform (IFFT) in the case of complete k-space raw data. However, in the acceleration case, where the raw data is significantly under-sampled, the reconstructed images obtained in this way will suffer from serious artifacts. To reduce them the reconstruction problem can be solved following the general CS-MRI framework: 2 2 1 ˆ arg min - ( ) 2 K u u k k x k x F x y R x
(2)where the first term is a data fidelity term addressing the consistency between x andy , and the second one is the u
regularization term concerning sparse constraints upon the MR image as the solution. F and u y are the under-sampled u
Fourier encoding matrix and the corresponding k-space data, respectively. is used to balance the regularization term and the data fidelity term. K is the number of imposed regularization terms. In the conventional model-based CS-MRI methods, the sparsity constraint such as total generalized variation (TGV) [27], wavelets [28] or adaptive transform domain obtained by dictionary learning [29, 30], can be considered as the regularization term. Although leading to improved reconstruction quality, methods based on model (2) are often limited by high computation cost, parameter sensitivities, and even compromised solution in the case of highly under-sampled data.
3. METHODS
The procedure of iterative optimization (2) is first unfolded as the neural network, and then regularization terms and their related parameters are replaced by the stacked CNN. This expanding network is considered as one cascade block for the proposed DFDN. And a data consistency (DC) layer is incorporated behind each block. Frequency-division loss for
k-space data computed after each block and image content loss
obtained after each DC layer are combined with different weights to preserve fine image details. The reconstruction network structure, loss function are described in detail in this section.
3.1 Network Architecture
In solving Eq. (2), different optimization methods can be used, such as Bregman methods and basis pursuit methods
[31-32]. When using a gradient descent algorithm, the updating process can be expressed as follows:
1 1 ( ) ( ) ( ) K t t t T t t T t t t u u u k k k k x x F F x y R R x
(3)where the superscript T is the transpose operation for matrix, ( )
k
is the differential of k( ) in Eq. (2). t k and t
k
R take
some specific forms over iterations.
Sparsity transform learned from the CNN trained data can be applied to improve the CS-based reconstruction [33]. The regularization term in Eq. (3) has a similar form as CNN, in which t
k
R and ( t)T k
R can be regarded as the convolutions in convolution layers, and t
k
as one activation layer [34]. According to this observation, the regularization term can be written in a more general form:
1 1 1 1 1 1 ( ) (ReLu( ( ReLu( )))), W * b , W *(ReLu( )) b , ( 2, 3, , ). t t t t L L t t t t t t t t l l l l Q x f f f f x f f l L (4)
where W (lt l1, 2, , L) is the weight consisting of
n
lconvolution kernels with a uniform kernel size s , bl t l is the corresponding bias, * and Re Lu( ) denote the convolution operator and activation function, respectively. The parameter
L in the stacked CNN represents the number of convolution layers. Fig.1 illustrates the architecture of such stacked CNN with brain MR images. Accordingly, Eq. (3) is transferred into the new form as follow:
1 ( ) ( )
t t t T t t
u u u
x x F F x y Q x . (5) This overall iterative reconstruction procedure can be unfolded into neural network termed as a deep iterative reconstruction network (DIRN) for MRI, which is shown in Fig. 2(a).
Compared with overall image information, edge features only accounts for a small part in MR image, which implies that an underweighted training of the edge features for the network might lead to compromised image detail preservation. Thus, it seems to be fuzzy in reconstruction results generally. In order to reduce this problem, multiple DIRNs are cascaded together with data consistency (DC) layers to construct one deeper neural network. Furthermore, compound loss including frequency-division loss and image content loss are proposed to preserve more edge information, which would be described in next sub-section. The proposed network with compound loss including frequency-division loss is termed as deep frequency-division frequency network (DFDN), the implementation framework of which is shown in Fig. 2(b).
Fig. 2. The implementation framework of (a) DIRN and (b) DFDN network. The blue and green solid arrows denote that the low frequency and high frequency k-space data acquiring operation from intermediate results, respectively. The orange solid arrow is to obtain the intermediate images.
3.2 Loss function
An improved solution to the problem of image details missing is to train the neural network with multiple constraints. Expect for the conventional image content loss, the frequency-division loss is proposed to preserve more image edge feature information in this study.
The image content based loss function is used as an important constraint imposing to the de-aliased image obtained from each DIRN block to approach the reference one. The 1 -norm is a sound way for evaluating the above similarity img:
1
mean( )
img xDCxR , (6)
where xDC is the output from each DC layer in DFDN and x R
is the reference image.
The frequency loss can be used as an additional constraint to reduce the difference between frequency data obtained after each DIRN block and reference k-space data. Low frequency data represents the main content of images, while high frequency data implies more image details. In order to preserve more edge features and avoid over-smoothing, the high frequency loss needs to be assigned higher weights. Thus, the high frequency loss and low frequency loss should be divided and calculated separately, which is termed as the frequency-division loss. The implementation process is achieved by the under-sampling mask:
1 1 mean( ( ) ( ) ) mean( ( ) ( ) ) lf DIRN R hf DIRN R s y s y s y s y , (7)
where yDIRN and y are the k-space data obtained from the R
IFFT of de-aliased image after each DIRN block and the reference image, respectively. s denotes the low frequency data acquiring operation by under-sampling mask, while s is reverse operation to obtain high frequency data.
Fig. 3. Some example brain MR images from HCP dataset (a) - (c), and BDP dataset (d) - (f).
Fig. 4. Four typical under-sampling patterns (Radial, Cartesian, Gauss and Spiral) adopted in our experiments.
In the overall network training process, the proposed DFDN is therefore trained to minimize the following combined loss function Lcom:
com img lf hf, (8) where , and r are the weights for adv, imgand frq, respectively.
4. EXPERIMENT SETUP AND RESULTS
4.1 Experiment Setup
a) Dataset description: The first brain dataset from the
Human Connectome Project (HCP)1 and the second brain dataset from the Brain Development Project (BDP)2 were both used for evaluating the proposed method. Each brain dataset includes 500 T1-weighted MR images from ten volunteers (
320 320 and 256 256 in-plane resolution for the first and second brain dataset, separately). 5-fold cross validation was conducted to test the performance of model in this study, in which 400 images were selected from eight volunteers for training and 100 images from another two volunteers for testing. Each dataset was normalized by the maximum intensity of the dataset. Image samples from the two datasets are displayed in Fig. 3. The training inputs were generated from the k-space data using the pseudo radial under-sampling pattern.
b) Network training setup: The parameters influencing the
reconstruction quality include the filter number
n
l, the size of convolution kernels
l, the stage number N and the number of tDIRN blocks N . In this study, to reduce the computation cost s
and memory consumption, the number of convolution layers L
in the stacked CNN was set to 3. Thus, for each stage t in DIRN, estimating the parameter set t = (W , W , W , b , b , b , 1t 2t 3t 1t 2t 3t t) is the optimization target for training the DFDN network. Filter numbers
{ , , }
n n n
1 2 3 and kernel sizes{ , , }
s s s
1 2 3 , the number of stages N and DIRN blocks should be set for network ttraining in the proposed method. In this study, the default configuration for DFDN was
n
1
n
2
48
,n
3
1
,1 2 3
5 5
s
s
s
, Ns 2and Nt 10. The regularization parameter
t was initially set to 0. The learning rate, batch size and epochs are set to be 10-4, 1 and 50 for training the proposed network, respectively. For the loss function in Eq. (8),-3 -2
1, 10 , =10
are the pre-defined weight parameters in the proposed method
c) Comparison algorithms: The comparisons with our
proposed DFDN approach include: zero-filling (ZF), Total Variation (TV), DeepADMM [21], DAGAN [25] and DIRN presented in this study. The ZF reconstructions and corresponding k-space data were used as inputs for all methods,
1 https://www.humanconnectome.org/ 2 www.brain-develpoment.org/ixi-dataset
and the reference images are reconstructed from fully sampled
k-space data. TV was applied through the provided executable
solver with the regularization parameter equal to 106. For the Deep ADMM method, 10 stages in network was recommended to achieve convergence. For the DAGAN method, the pre-tuned hyper-parameters include: weights for pixel loss, perceptual loss, frequency loss and frequency loss. They were respectively set to15, 0.0025, 0.1 and 1, and the initial learning rate to 0.0001. Regarding to DIRN, the initial learning rate 0.0001 and 50 epochs for training were used in this study.
d) Under-sampling Patterns: Radial under-sampling pattern
was applied to retain 10%, 20% and 30% k-space data corresponding to 10, 5 and 3.3 acceleration factors. In order to compare the influence of different under-sampling patterns for the proposed DFDN, other patterns like Cartesian, gauss and spiral were also tested on brain dataset from BDP together. Fig. 4 depicts the four different under-sampling patterns.
e) Implementation details: In this study, data augmentation
is conducted to enhance generalization and robustness performance of the proposed DFDN. Image flipping, rotation and cropping were applied to simulate the elastic deformation of human tissues. Normalized mean squared error (NMSE), peak signal-to-noise ratio (PSNR in dB), structural similarity information (SSIM) and test time cost are adopted to assess the performance of different algorithms. This network was trained using the neural network toolkit tensorpack3 and developed on the basis of the tensorflow library. All the computations were performed on one PC equipped with Intel(R) Core(TM) i5-7500 CPU @3.40 GHz and 16 GB RAM. A 11 GB Nvidia GTX 1080 Ti is used as the GPU for network training.
4.2 Qualitative and quantitative assessment
Fig. 5 shows the reconstructed HCP brain images using different methods at 10% and 30% under-sampling rates. It can be observed from the zoomed-in ROI images that the quality of zero-filling IFFT result at 10% sampling rate has significantly decreased when compared with the reference image (i.e. the IFFT reconstruction for the original complete k-space data). TV, DeepADMM, DAGAN and DIRN comparison methods are capable of showing tissue structures relatively better. While the proposed DFDN has the best performance in preserving more image features and textures. Although the image quality reconstructed from all methods is improved obviously at 30% sampling rate, the error image related to DFDN show the smallest differences. Similar conclusions can be drawn for the DFDN when considering the second brain dataset from BDP. We can see from Fig. 6 that the proposed DFDN method leads visually to a better reconstruction with richer image details than the other methods.
In order to compare the performance of different methods comprehensively, the quantitative measurements in terms of NMSE and PSNR are summarized in Tables I and II for both brain datasets at different sampling rates. For the HCP brain
dataset at 10% and 20% under-sampling rates, deep learning based methods almost all surpass the classical TV except the DAGAN at sampling rate 20%. The proposed DFDN gained PSNR improvements of 0.79 dB and 0.83 dB compared with the best results from all selected previous methods. Although DeepADMM, DAGAN and DIRN has not advantages compared with classical TV at 30% sampling rate, the proposed DFDN perform better than the TV about 0.3 dB PSNR. Regarding to the BDP brain dataset, the proposed DFDN still has the highest PSNR and lowest NMSE compared with other comparisons. Fig. 7 provides the SSIMs for both brain datasets
at different sampling rates, which confirm the quantitative and qualitative assessments above.
Furthermore, the time cost of different algorithms are also compared and depicted in Table III. However, the deployed convolution neural network can be accelerated using GPU, then deep learning-based methods including DeepADMM, DIRN and DFDN are extremely fast (less than 1 seconds). The running test time of DIRN are decreased to about 0.05 seconds. The running time of DFDN is about twice as long because the network depth is deepen, but it still runs at lower time cost than DeepADMM.
TABLE I
QUANTITATIVE RESULTS (NMSE, PSNR) CALCULATED FOR COMPARISON PURPOSE BETWEEN THE SELECTED ALGORITHMS WITH DIFFERENT UNDER-SAMPLING RATIOS USING THE HCP BRAIN TEST DATASET.
Methods
10% 20% 30%
NMSE PSNR NMSE PSNR NMSE PSNR ZF 0.2395 ± 0.0244 24.6566 ± 1.2225 0.1584 ± 0.0187 28.2622 ± 1.1399 0.1184 ± 0.0139 30.7908 ± 1.1188 TV 0.1467 ± 0.0189 28.9424 ± 1.0318 0.0710 ± 0.0068 35.2156 ± 1.2583 0.0440 ± 0.0047 39.3860 ± 1.6568 DeepADMM 0.1194 ± 0.0137 30.7127 ± 1.0416 0.0645 ± 0.0062 36.0742 ± 1.3845 0.0465 ± 0.0046 38.8898 ± 1.5492 DAGAN 0.1330 ± 0.0143 29.7724 ± 1.0046 0.0890 ± 0.0082 33.2526 ± 1.1576 0.0699 ± 0.0059 35.3387 ± 1.2903 DIRN 0.1030 ± 0.0057 31.8783 ± 0.8842 0.0655 ± 0.0040 35.8184 ± 0.7925 0.0489 ± 0.0025 38.3550 ± 0.9405 DFDN 0.0941 ± 0.0054 32.6677 ± 0.8760 0.0577 ± 0.0032 36.9076 ± 0.9056 0.0421 ± 0.0027 39.6593± 0.8968 TABLE II
QUANTITATIVE RESULTS (NMSE, PSNR) CALCULATED FOR COMPARISON PURPOSE BETWEEN THE SELECTED ALGORITHMS WITH DIFFERENT UNDER-SAMPLING RATIOS USING THE BDP BRAIN TEST DATASET.
Methods NMSE 10% PSNR NMSE 20% PSNR NMSE 30% PSNR ZF 0.2401 ± 0.0109 27.3308 ± 1.1084 0.1720 ± 0.0080 30.2299 ± 1.0750 0.1313 ± 0.0062 32.5730 ± 1.0665 TV 0.2007 ± 0.0147 28.8994 ± 1.3313 0.1225 ± 0.0121 33.2058 ± 1.4659 0.0825 ± 0.0101 36.6602 ± 1.6222 DeepADMM 0.1831 ± 0.0144 29.7024 ± 1.3601 0.1079 ± 0.0123 34.3243 ± 1.6166 0.0749 ± 0.0085 37.4982 ± 1.5201 DAGAN 0.1951 ± 0.0157 29.1511 ± 1.4187 0.1356 ± 0.0115 32.3115 ± 1.4648 0.1071 ± 0.0112 34.3802 ± 1.6351 DIRN 0.1705 ± 0.0091 31.0110 ± 0.4604 0.1061 ± 0.0056 35.1264 ± 0.5109 0.0750 ± 0.0045 38.1452 ± 0.6515 DFDN 0.1494 ± 0.0091 32.1596 ± 0.4817 0.0906 ± 0.0060 36.5100 ± 0.5637 0.0634 ± 0.0050 39.6141 ± 0.6992
Fig. 5. Representative HCP brain MR images reconstructed using different algorithms. The first three rows show the results at 10% under-sampling rate: 2D results (the 1st row), zoomed-in images (the 2nd row), and error images (the 3rd row). The last three rows correspond to the results at 30% under-sampling rate: 2D results (the 4th row), zoomed-in images (the 5th row), and error images (the 6th row).
Fig. 6. Representative BDP brain MR images reconstructed using different algorithms. The first three rows show the results at 10% under-sampling rate: 2D results (the 1st row), zoomed-in images (the 2nd row), and error images (the 3rd row). The last three rows correspond to the results at 30% under-sampling rate: 2D results (the 4th row), zoomed-in images (the 5th row), and error images (the 6th row).
4.3 Parameter analysis in network training
Except for the hyper-parameters in network training, some parameters determining network depth and loss function should be preset, such as the stage numberN in each DIRN block and t
the number of DIRN blocks N . In this section, we evaluate s
their impact on the final reconstruction accuracy.
a) Stage number N in each DIRN block: With the number t
of DIRN blocks Ns 2 in the proposed DFDN, representative reconstruction results obtained from DFDN with different stage number in each DIRN block are shown in Fig. 8. It can be observed that the error images obtained from DFDN with
5, 10, 15 and 20 t
N has not obviously difference. However, the qualitative measurement PSNR depicted in Fig. 9 shows
that the first upward and then downward trend with the increase of stage number in each DIRN block of DFDN.
b) The number of DIRN blocks N : In the proposed DFDN, s
the number of DIRN blocks N not only determines the depth s
of network but also decides the components of compound loss function. Different DIRN block numbersNs 2, 4 , 5 and 10 were tested for the DFDN reconstruction, with the total stage number of DFDN set to be 20. Fig. 10 illustrates the results and error images obtained from DFDN with different DIRN blocks. The error images are reduced with the increase of N until it s
reach to 5. Quantitative analysis of DFDN with different N s
for each DIRN are presented in Fig. 11, from which we can see that the reconstruction accuracy is improved first and then decreased with the increase of N . s
4.4 Study on different under-sampling patterns
For a given sampling rate, the reconstruction performance is not only determined by the network model but also by the under-sampling patterns. In order to determine the optimal under-sampling pattern for the DFDN method, four various masks were tested on BDP brain test datasets with an under-sampling rate equal to 20%. From the images shown in the second and fourth rows in Fig. 12, it is obvious that the spiral
pattern leads to minimal errors. Fig. 13 plots the quantitative comparison of PSNRs in dB: 34.9790 ± 2.5078 (Radial), 34.2835±2.1655 (Cartesian), 37.4535±3.1001 (Gaussian) and 37.8754±1.9934 (Spiral). Clearly, the proposed DFDN method performs better in the case of spiral pattern.
Fig. 7. SSIM measurements on both HCP and BDP brain test datasets at the sampling rate 10%, 20% and 30% respectively for the selected methods. TABLE III TEST TIME COST COMPARISONS OF DIFFERENT METHODS ON HCP
AND BDP BRIAN TESTING DATASETS (IN SECONDS)
Methods HCP BDP with/without GPU ZF 0.0021 0.0018 without TV 1.334 1.060 without DeepADMM 0.987 0.779 with DAGAN 0.023 0.018 with DIRN 0.048 0.046 with DFDN 0.097 0.096 with
Fig. 8. Typical BDP MR images reconstructed from DFDN with
5, 10, 15 and 20
t
N in the second brain dataset at 20% sampling rate: 2D results (the first and third rows) and their corresponding error images (the second and fourth rows).
Fig. 9. PSNRs (in dB) computed by DFDN with respect to
5, 10, 15 and 20
t
Fig. 10. Typical BDP MR images reconstructed from DFDN with
2, 4, 5 and 10
s
N in the second brain dataset at 20% sampling rate: 2D results (the first and third rows) and their corresponding error images (the second and fourth rows).
Fig. 11. PSNRs (in dB) computed by DFDN with respect to
2, 4, 5 and 10
s
N in the BDP brain dataset at 20% sampling rate.
Fig. 12. Comparison of DFDN performance with respect to various patterns at 20% under-sampling rate: 2D results (the first and third rows) and their corresponding error images (the second and fourth rows).
Fig. 13. PSNRs (in dB) computed by DFDN with respect to different patterns at 20% under-sampling rate.
5. DISCUSSION AND CONCLUSION
Inspired by the effective application of stacked CNN in many computer vision applications, the iterative algorithm adopts CNN to substitute its regularization terms and corresponding parameters and then be unfolded as neural network to improve reconstruction quality of CS-MRI in this study. However, the resulting reconstructions (DIRN in this
study) still suffer from the problem of structure blurring. Therefore, a deep frequency-division based network strategy (DFDN) is proposed. Multiple DIRN blocks are cascaded continuously as one deeper neural network. Data consistency (DC) layer is incorporated after each DIRN block to correct the
k-space data of intermediate results. Image content loss is
computed after each DC layer and frequency-division loss is gained by weighting the high frequency loss and low frequency
loss after each DIRN block. The combination of image content loss and frequency-division loss is considered as the total loss for constraining the network training procedure. In this study, two brain datasets are used to validate the effectiveness of the proposed method. Visual inspections and quantitative measurements in terms of NMSE, PSNR and SSIM demonstrate that the DFDN method leads to the best performance when compared to other methods.
Nevertheless, image blurring may still be observed in the reconstructed images with the proposed DFDN method, especially for high under-sampling rates. Our work in progress will consider some other loss functions like histogram matching or entropy in order to improve the present results. Furthermore, strategies dealing with extended frequency truncations will be explored to achieve more elaborated and accurate reconstructions.
ACKNOWLEDGEMENTS
This work was supported in part by the State’s Key Project of Research and Development Plan under Grant 2017YFA0104302, Grant 2017YFC0109202 and 2017YFC0107900, in part by the National Natural Science Foundation under Grant 81530060 and 81471752, in part by the Fundamental Research Funds for the Central Universities, and in part by the joint research project of Southeast University and Nanjing Medical University.
REFERENCES
[1] M. Lustig, D. David, M. S. Juan, et al. “Compressed sensing MRI,” IEEE
signal processing magazine, vol. 25, no. 2, pp. 72-82, 2008.
[2] M. Lustig, D. David, and M. P. John. “Sparse MRI: The application of compressed sensing for rapid MR imaging,” Magnetic resonance in
medicine, vol. 58, no.6, pp. 1182-1195, 2007.
[3] K P. Pruessmann, M. Weiger, MB. Scheidegger, et al. “SENSE: sensi-tivity encoding for fast MRI,” Magn Reson Med, vol. 42, no. 5, pp. 952-962, 1999.
[4] MA. Griswold, PM. Jakob, RM. Heidemann, et al. “Generalized autocalibrating partially parallel acquisitions (GRAPPA),” Magn Reson
Med, vol. 47, no. 6, pp. 1202-1210, 2002.
[5] Y. Wang, L. Ying. “Compressed sensing dynamic cardiac cine MRI using learned spatiotemporal dictionary,” IEEE transactions on
Biomedical Engineering, vol. 61, no.4, pp. 1109-1120, 2014.
[6] K. T. Block, M. Uecker, and J. Frahm. “Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint,” Magnetic resonance in medicine, vol.57, no.6, pp. 1086-1098, 2007.
[7] J. Yang, Y. Zhang, and W. Yin. “A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data,” IEEE Journal
of Selected Topics in Signal Processing, vol. 4, no. 2, pp. 288-297, 2010.
[8] L. Chaâri, J. C. Pesquet, and A. Benazza-Benyahia. “A wavelet-based regularized reconstruction algorithm for SENSE parallel MRI with applications to neuroimaging,” Medical image analysis, vol. 15, no. 2, pp. 85-201, 2011.
[9] Z. Zhu, K. Wahid, P. Baby, et al. “Compressed sensing-based MRI reconstruction using complex double-density dual-tree DWT,” Journal
of Biomedical Imaging, vol. 5, pp.10, 2013.
[10] Z. Zhan, JF. Cai, D. Guo, et al. “Fast multiclass dictionaries learning with geometrical directions in MRI reconstruction,” IEEE Transactions
on Biomedical Engineering, vol. 63, no. 9, pp. 1850-1861, 2016.
[11] Y. Song, Z. Zhu, Y. Lu, et al. “Reconstruction of magnetic resonance imaging by three‐ dimensional dual‐ dictionary learning,” Magnetic
resonance in medicine, vol. 71, no. 3, pp. 1285-1298, 2014.
[12] S. Ramani, Z. Liu, J. Rosen, et al. “Regularization parameter selection for nonlinear iterative image restoration and MRI reconstruction using
GCV and SURE-based methods,” IEEE Transactions on Image
Processing, vol. 21, no. 8, pp. 3659-3672, 2012.
[13] T.H. Chan, K. Jia, S. Gao, et al. “PCANet: A simple deep learning baseline for image classification,” IEEE Transactions on Image
Processing, vol. 24, no.12, pp. 5017-5032, 2015.
[14] K. He, X. Zhang, S. Ren, et al. “Deep residual learning for image recognition,” Proceedings of the IEEE conference on computer vision
and pattern recognition, 2016.
[15] Y. Deng, Z. Ren, Y. Kong, et al. “A Hierarchical Fused Fuzzy Deep Neural Network for Data Classification,” IEEE Transactions on Fuzzy
Systems, vol. 25, no. 4, pp. 1006-1012, 2017.
[16] Y. Kong, Y. Deng, and Q. Dai. “Discriminative Clustering and Feature Selection for Brain MRI Segmentation,” IEEE Signal Processing
Letters, Vol. 22, no.5, pp. 573-577, 2015.
[17] J. Schmidhuber. “Deep learning in neural networks: An overview,”
Neural networks, vol. 61, pp. 85-117, 2015.
[18] J. Mehta, A. Majumdar. “RODEO: robust DE-aliasing autoencoder for real-time medical image reconstruction,” Pattern Recognition, vol. 63, pp. 499-510, 2017.
[19] Zhu Q, Du B, Turkbey B, et al. “Exploiting interslice correlation for MRI prostate image segmentation, from recursive neural networks aspect,”
Complexity, 2018.
[20] Q. Zhu, B. Du, B. Turkbey, et al. “Deeply-Supervised CNN for Prostate Segmentation,” IEEE International Joint Conference on Neural
Networks (IJCNN), pp.178-184, 2017.
[21] Y. Yang, J. Sun, H. Li, et al. “ADMM-Net: A deep learning approach for compressive sensing MRI,” arXiv preprint arXiv: 1705.06869, 2017. [22] K. Hammernik, T. Klatzer, E. Kobler, et al. “Learning a variational
network for reconstruction of accelerated MRI data,” Magnetic
resonance in medicine, vol. 79, no. 6, 2017.
[23] J. Schlemper, J. Caballero, JV. Hajnal, et al. “A deep cascade of convolutional neural networks for MR image reconstruction,”
International Conference on Information Processing in Medical Imaging. Springer, Cham, pp. 647-658, 2017.
[24] T. M. Quan, T. Nguyenduc, WK Jeong. “Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss,” IEEE Trans Med Imaging, vol. 37, no. 6, pp.1488-1497, 2018. [25] G. Yang, Simiao Yu, Hao Dong, et al. “DAGAN: Deep De-Aliasing
Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction,” IEEE Trans. Med. Imaging, vol. 37, no. 6, pp. 1310- 1321, 2018
[26] Zhu B, Liu J Z, Cauley S F, et al. “Image reconstruction by domain-transform manifold learning,” Nature, vol. 555, no. 7697, pp. 487, 2018.
[27] F. Knoll, K. Bredies, T. Pock, et al. “Second order total generalized variation (TGV) for MRI,” Magnetic resonance in medicine, vol. 65, no. 2, pp. 480-491, 2011.
[28] X. Qu, D. Guo, B. Ning, et al. “Undersampled MRI reconstruction with patch-based directional wavelets,” Magnetic resonance imaging, vol. 30, no.7, pp. 964-977, 2012.
[29] S. Ravishankar, Y. Bresler. “MR image reconstruction from highly undersampled k-space data by dictionary learning,” IEEE transactions
on medical imaging, vol. 30, no.5, pp. 1028-1041, 2011.
[30] J. Caballero, AN. Price, D. Rueckert, et al. “Dictionary learning and time sparsity for dynamic MR data reconstruction,” IEEE transactions on
medical imaging, vol. 33, no.4, pp. 979-994, 2014.
[31] T. Goldstein, S. Osher. “The split Bregman method for L1-regularized problems,” SIAM journal on imaging sciences, vol. 2, no.2, pp. 323-343, 2009.
[32] M. Lustig, D. David, and M. P. John. “Sparse MRI: The application of compressed sensing for rapid MR imaging,” Magnetic resonance in
medicine, vol. 58, no.6, pp. 1182-1195, 2007.
[33] S. Ravishankar, Y. Bresler. “Data-driven learning of a union of sparsifying transforms model for blind compressed sensing,” IEEE
Transactions on Computational Imaging, vol. 2, no. 3, pp. 294-309,
2016.
[34] H. Chen, Y. Zhang, Y. Chen, et al. “Learned Experts’ Assessment-based Reconstruction Network (LEARN) for Sparse-data CT,” arXiv preprint
arXiv: 1707.09636, 2017.
[35] D. Kingma, J. Ba. “Adam: A method for stochastic optimization,”
[36] LeCun, Yann, et al. “Efficient backprop,” Neural networks: Tricks of the
trade, vol. 1524, no. 1, pp. 9-50, 1998.
[37] J. Trzasko, A. Manduca. “Highly Undersampled Magnetic Resonance Image Reconstruction via Homotopic l0-Minimization,” IEEE
Transactions on Medical imaging, vol. 28, no. 1, pp. 106-121, 2009.
Jiulou Zhang received the B. S degree from Henan University of
Science and Technology, Luoyang, China in 2009, and M. S. degree from Southern Medical University, Guangzhou, China in 2012, and Ph. D. degree from Tsinghua University, Beijing, China in 2016. He is currently a Postdoctoral with the College of Computer Science and Engineering, Southeast University. His current research interests including machine learning, MRI reconstruction and medical image processing.
Yunbo Gu received the M. S. degree from Southeast University,
Nanjing, China, in 2018. Since 2015, he has been with the college of software engineering, Southeast University, Nanjing, China. He has been involved with medical imaging processing. His research interest is mainly about MRI reconstruction.
Hui Tang received her B.S. and M.S. degree in biomedical engineering
from Southeast University, Nanjing, China, in 2003 and 2005, respectively. She received a joint Ph. D degree in biomedical engineering from Southeast University and in signal processing and telecommunication from University of Rennes 1, France, in 2008. She is currently an Associate Professor at LIST in the School of Computer Science and Engineering, Southeast University, Nanjing, China. Her current research interests are image processing, image analysis and scientific visualization applied in medical areas.
Xiaoqing Wang received the B.Sc. and M.Sc. degrees from the
Department of Biomedical Engineering, Southeast University, China, in 2009 and 2012, respectively, and the Dr. rer. nat degree from the Max Plank Institute for Biophysical Chemistry and the Department of Physics, University of Göttingen, Germany, in 2016. Currently, he is a Postdoctoral Fellow in Uecker Lab at the Department of Interventional and Diagnostic Radiology, University Medical Center Göttingen. His research interests are in MR image reconstruction, Cardiovascular MRI and Quantitative MRI.
Youyong Kong received the B.S. and M.S. degrees in computer science
and engineering from Southeast University, Nanjing, China, in 2008 and 2011, respectively, and the Ph.D. degree in imaging and diagnostic radiology from the Chinese University of Hong Kong, Hong Kong, in 2014. He is currently an Assistant Professor with the College of Computer Science and Engineering, Southeast University. His current research interests include machine learning, and medical image processing and brain network analysis.
Yang Chen received the M.S. and Ph.D. degrees in biomedical
engineering from First Military Medical University, China, in 2004 and 2007, respectively. From April 2008 to now he is a faculty member of the Department of Computer Science and Engineering of Southeast University, China. His recent work concentrates on the medical image reconstruction, image analysis, pattern recognition, and computerized-aid diagnosis.
Huazhong Shu (M’00–SM’06) received the B.S. degree in Applied
Mathematics from Wuhan University, China, in 1987, and a Ph.D. degree in Numerical Analysis from the University of Rennes 1, Rennes, France in 1992. He is a professor of the LIST Laboratory and the Codirector of the CRIBs. His recent work concentrates on the image analysis, pattern recognition and fast algorithms of digital signal processing. Dr. Shu is a senior member of IEEE Society.
Jean Louis Coatrieux received the Ph.D. and State Doctorate in
Sciences in 1973 and 1983, respectively, from the University of Rennes 1, Rennes, France. Since 1986, he has been Director of Research at the National Institute for Health and Medical Research (INSERM), France, and since 1993 has been Professor at the New Jersey Institute of Technology, USA. He has been the Head of the Laboratoire Traitement du Signal et de l’Image, INSERM, up to 2003. His experience is related to 3D images, signal processing, pattern recognition, computational modeling and complex systems with applications in integrative biomedicine. He published more than 300 papers in journals and conferences and edited many books in these areas. He has served as the Editor-in-Chief of the IEEE Transactions on Biomedical Engineering (1996–2000) and is in the Boards of several journals. Dr. Coatrieux is a fellow member of IEEE.
