• Aucun résultat trouvé

Characterization of deep neural network feature space for inverse synthetic aperture radar automatic target recognition

N/A
N/A
Protected

Academic year: 2021

Partager "Characterization of deep neural network feature space for inverse synthetic aperture radar automatic target recognition"

Copied!
36
0
0

Texte intégral

(1)

1

Characterization of Deep Neural Network Feature Space For Inverse Synthetic Aperture Radar Automatic Target Recognition

by

Christopher Z. Au

B.S. Electrical Science and Engineering Massachusetts Institute of Technology, 2019

SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ENGINEERING IN ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY

May 2020

© 2020 Christopher Z. Au. All rights reserved.

Signature of Author: ________________________________________________________ Department of Electrical Engineering and Computer Science May 11, 2020 Certified by: _____________________________________________________________ Jing Kong, Professor of Electrical Engineering, Thesis Supervisor Certified by: _____________________________________________________________.. David Barrett, Senior Staff, Thesis Co-Supervisor Accepted by: _____________________________________________________________ Katrina LaCurts, Chair, Master of Engineering Thesis Committee

(2)

2

Characterization of Deep Neural Network Feature Space For

Inverse Synthetic Aperture Radar Automatic Target Recognition

By

Christopher Z. Au

Submitted to the Department of Electrical Engineering and Computer Science on May 11, 2020 in Partial Fulfillment of the

Requirements for the Degree of Master of Engineering in Electrical Engineering and Computer Science ABSTRACT

The Airborne Radar Systems and Techniques group at MIT Lincoln Laboratory trained neural networks to classify different targets at sea based on inverse synthetic aperture radar (ISAR) data. Simulated data was used to train these neural network based

automatic target recognition (ATR) systems. The technical challenge of this project was to find a way to evaluate the quality and adequacy of a limited set of training data. Using simulated ISAR images to train neural networks, the project determined the minimum amount of variation in terms of parameters such as aspect angle to adequately train a neural network. Establishing a correspondence between training data variation and the resulting feature space of the data informed the minimum spanning-set of training data required for future data collects.

Thesis Supervisor: Jing Kong

Title: Professor of Electrical Engineering

Thesis Co-Supervisor: David Barrett

(3)

3

1. Introduction:

Inverse synthetic aperture radar (ISAR) is a radar technique to generate two-dimensional images of a target. ISAR utilizes the movement of the target instead of the emitter to create a synthetic aperture. ISAR radars are often used in maritime patrol aircraft to classify targets at sea, such as ships and other objects. ISAR imagery is well suited for patrol at sea because ISAR can often clearly identify recognizable features that appear over the surface of the ship, such as a ship mast. ISAR has advantages over camera images because ISAR images can be used at greater distances in a variety of weather and lighting conditions.

Machine learning can improve target classification rates because machine learning can leverage large amounts of labeled data. Applying modern neural network tools to ISAR for automatic target recognition has gained interest in the past few years due to the availability of accessible and powerful tools in machine learning. In the related field of synthetic aperture radar (SAR), a 2018 Beihang University study trained neural networks using 2522 training samples taken by the Chinese Gaofen-3 satellite. The researchers found that classification rates of above 80% could be achieved using convolutional neural networks to classify objects such as boats, cargo ships, towers, and tankers [7].

Modern neural network based classifiers can be interpreted as consisting of two components: a feature extractor and a classifier. The feature extractor converts each test input into a vector describing that test input’s features. The classifier then uses these feature vectors to assign each test image a final classification. The collection of these

(4)

4

vectors constitutes a feature space, which can be visualized after reducing the dimensionality of the space to two dimensions.

It has been shown in related fields that one can achieve high target classification performance with ISAR imagery using a full aspect angle set. In March 2018, at the International Conference on Advanced Technologies for Signal and Image Processing, researchers from the University of Southern Brittany were able to obtain classification accuracies of above 92% for various air targets tests using the full aspect angle coverage from 0 to 360 degrees when trained on 840 ISAR database images including 7 planes [2].

The Airborne Radar Systems and Techniques group at MIT Lincoln trained neural networks to work with ISAR imagery. The neural networks were trained on ISAR images that varied across aspect angle and signal to noise ratio. This project helped to inform the design of future data collects by determining the necessary spanning set of these

variables required to train a neural network adequately.

The project plan was to conduct various tests to examine how these variables affect the performance of the neural network. This paper will first describe the radar geometry and data used. Then, the neural network architecture will be discussed. The methods of testing included adding noise to see its effect on performance, varying the quantity of data used to test, and training using different subsets of angles. The reason these experiments were conducted was to isolate the effects of different variables on the performance of the neural network. The classification accuracy was recorded quantify how well the neural network did under these different circumstances.

(5)

5

The feature space of the data points was also pulled out from intermediate layers in the neural network, visualized, and analyzed using a metric to quantify how different clusters of points were. The motivation for this visualization was to analyze the deeper reasons why the neural network performed the way it did under different experiments. Although classification accuracy can describe the final performance results of a neural network and help understand what angles are needed, classification accuracy does not explain where these dependencies come from. The 3d visualization was used to gain more information on how the data was being treated at intermediate layers inside of the neural network.

2. ISAR Description and Data: ISAR Simulated Training Data:

The Airborne Radar Systems and Techniques group at MIT Lincoln Laboratory can simulate thousands of ISAR images using CAD models of targets. These images were used to train neural networks in this project. Currently, ISAR images can be generated for over forty different targets at a variety of conditions including different aspect angle and motion profiles. To better understand the feature space across different kinds of ships and to keep the amount of data manageable, four distinct targets were chosen. For each target, the simulation was varied across several variables discussed below.

(6)

6

The four targets selected from the simulation were chosen to give diversity to the shapes of the ships recognized. The targets were a variety of tankers and oilers roughly around 200 meters in length. At various points in this paper when a neural network is trained with four different classes, it was useful to label the different targets by a class number. The target number was the ship’s numbered listing in a database of CAD models available to the group working on this project. The following figures show the

specifications for the four targets used in this experiment and images of their CAD models.

Target Name Class Number

Target Number Type Length

(meters)

Dhonoussa 0 16 Commercial

tanker

225

Fuchi 1 17 Commercial oiler 178.5

Boris Chilikin 2 10 Fleet

replenishment oiler

162.3

Stolt Spray 3 34 Oil/Chemical

tanker

162.6

(7)

7 Figure 2: CAD Models

Source: [11]

Data Collection Geometry:

The elevation angle of the radar was 15 degrees. This was chosen because it is within common ranges actual airborne radars operate in. The following figure shows the definition of aspect angle, where the object in the center is the target.

(8)

8 Figure 3: Aspect Angle Definition

Initial testing involved examining angles at increments of 10 degrees. The 10 degree increments were chosen to obtain some angle granularity without creating too much data that would take too long to train neural networks with. For each target and aspect angle, there were 19 sample image frames taken, for a maximum total of 361 images per class at a given motion profile. However, using the entire range from 0 to 180 degrees can cause symmetry issues. For example, 0 and 180 degree pictures may have similarities because they are the front and back ends of the ship and have similar profiles. Therefore, training on an angle that happens to give similar images could

(9)

9

more difficult to see. To prevent symmetry issues, only angles from 0 to 90 degrees were used in the experiment. A variety of different subsets of aspect angles, described in detail in later sections, were used to train neural networks.

Because the aspect angle strongly affects the final ISAR image, the first goal of this project was to explore the amount of aspect angle variation needed to adequately train a neural network. For aspect angle, it was useful to determine whether a neural network could accurately classify a target after being trained on a limited subset of angles. This gave information about the relative degradation of performance as greater numbers of aspect angles were removed.

Motion profile:

The movement of water affects a target’s motion at sea. Different motion profiles were generated by randomizing initial seed values for roll, pitch, and yaw to produce new data to train the neural network. Each target was varied over different motion profiles to increase the amount of data that the neural network has for a given target. The motion profile will be abbreviated as “Mp” or “MP” in figures. The motion model was sourced from a 2008 paper by Armin Doerry of Sandia National Laboratories [13].

(10)

10

Zero mean Gaussian noise was added to the images at various standard deviations to simulate noise conditions of real images. Higher standard deviation Gaussian noise corresponds to a noisier image. When this paper mentions an image with 10 dB noise, it will be defined to be 0 mean Gaussian noise with 10 dB standard deviation. By adding higher standard deviation Gaussian noise, one decreases the signal to noise ratio in the image.

3. Methodology:

One goal of the project was to see how the neural network responded to

limitations in the training data. These limitations included the number of training images, the signal to noise ratio, and the angle variation in the training data. Another goal was to analyze the feature space of the neural network using a metric that could describe the difference between clusters of points. The metric chosen to describe the difference between clusters of points was the Kullback-Leibler divergence, which is discussed at length in a subsequent section. The following figure shows an overall block diagram of how these experiments are carried out. After the set of input images was decided in each experiment, the noise level of the images was chosen. The images were then input into the neural network, which could produce classification accuracies and KL divergences for the images. The KL divergences produced were used to analyze the distribution of the feature vectors.

(11)

11 Figure 4: Block Diagram of Overall Experiment

Neural Network Description:

A neural network created by the MIT Lincoln Laboratory Airborne Radar Systems and Techniques group was developed in 2017 for use in ISAR and other SAR applications. This neural network performs well for ISAR imagery and uses a fully convolutional neural network [9]. Batch normalization layers were added to that neural network in 2019 before this project started. These layers make the network significantly less sensitive to different learning rates. There are 29 layers of the neural network. Salient portions of the neural network architecture are summarized in the following figure.

Frames of ISAR data are the test images that were input into the neural network. Each image resulted in a class prediction and a feature vector. The 3-dimensional dense layer that is shown in the following block diagram for the neural network was used to generate 3-dimensional feature vectors for visualization. A future extension to the project currently being worked on is to analyze the 1584 dimension dense layer in that diagram. The final softmax function step converted the results into probabilities, which the neural network uses to predict the object.

(12)

12 Figure 5: Neural Network Summary

Adding Gaussian Noise to Change Signal to Noise Ratio

Adding Gaussian noise helped to mimic the noise found in real data, which is less ideal than the images generated by the simulation. The starting images were of

dimension 500 pixels cross range by 1024 pixels of range. For a given motion profile and aspect angle, the simulation had dwells over each scenario lasting 10 seconds and collects

(13)

13

19 evenly spaced frames of the image, numbered 0 through 18. This paper will refer to cropped images as chipped images. First, the images were chipped to be 128 by 128 pixels to make the neural network process the images faster. Then, each chip was converted into dB scale and the median across the dwell was subtracted. The median over a dwell was computed and subtracted from the chips to normalize their values. Zero mean Gaussian noise was generated and added on top of those images. Finally, to match the format of the neural networks used by the group, which was written with an 8 bit color image in mind, the intensities were scaled to a 0 to 255 scale.

The following pictures depict an ISAR image of the Fuchi, a commercial oiler. First, images were cropped and the median was subtracted out. Then, Gaussian noise was added. This example images in the following figure shows the result of adding the Gaussian noise. For brevity, this paper will call images with Gaussian noise added as “noised images.”

(14)

14 Figures 6: ISAR Image Before and After Adding Noise

Classification Accuracy

Classification accuracy was one of the metrics used to characterize the neural network’s performance in response to different training image sets. The neural network takes in test images and assigns classes to them. The percentage classified correctly is the classification accuracy.

Kullback-Leibler Divergence

The Kullback-Leibler divergence (KL divergence) can measure how different one

probability distribution is from another. The Kullback-Leibler divergence is not symmetric. In other words, one probability distribution is designated as a reference distribution and the Kullback-Leibler divergence measures how different another probability distribution is from the reference. Zero KL divergence means that the distributions are identical.

Mathematically using the notation of data scientist Will Kurt, if one has probability distributions p and q, the KL divergence of q from p can be expressed as follows [10]:

In the context of this project, the amount of difference between two distributions was useful for quantifying how different clouds of datapoints in the feature space were. For example, for a given cloud of points in the feature space corresponding to a single target, the KL divergence can give information for how different angles are from the other

(15)

15

angles. As described in earlier sections, feature vectors from the three-dimensional layer described in the neural network architecture block diagram were pulled out to create this space of points. This gave a way to visualize how similar points were. This resulted in a three dimensional data point for every test image. These clusters of points could then be input into the expression for the Kullback-Leibler divergence to give measures of how similar clouds of points are.

4. Results:

Several experiments were conducted to see how the neural network responded to different limitations in the training dataset. First, the performance of the neural network was analyzed as the amount of Gaussian noise in the images was increased. Then, an experiment was conducted to see the effect of training data quantity on test accuracy. Two subsequent tests examined how the neural network responded to training on only individual angles or pair of angles to understand how those angles contributed to the neural network. Finally, an experiment involving progressively removing the adjacent angles around specific angles was conducted. The purpose of these different experiments was to isolate the effects of different training data limitations on the neural network’s performance. Finally, feature vectors of the neural network were pulled out to produce a three dimensional visualization of the data. The KL divergence between clusters of points was analyzed to analyze how the clusters were distributed to better understand how the underlying reasons for the neural network’s behavior.

(16)

16

Gaussian noise with mean 0 and standard deviation 0 to 50 at 10 dB intervals was added to see the effect of noise on classification accuracy. The classifier that was used in these experiments was the one described in the architecture under “Neural Network Description” in a previous section of this paper. Figure 8 displays the test accuracy results using a mixture of 1520 training images of the 225 meter commercial oiler Fuchi and the 178.5 meter chemical tanker Dhonoussa. 1520 images were chosen because it was the maximum number of images for two targets in the simulation dataset. CAD models of these are shown in Figure 8, both produced by ES3D studios [11].

The 1520 images represented aspect angles from 0 to 90 degrees at 10 degree increments. The test images were 76 test images of Fuchi and Dhonoussa images all at 30 degrees. For each noise level, three trials were run with three different random generator seed values and then averaged. The final average test accuracies across different noise levels were recorded as are shown in the following figures. It was notable that even at extremely high noise levels such as 40 dB, test accuracies of above 80% were achieved. This suggests that the neural network may be resistant to negative effects of Gaussian noise.

(17)

17

Figure 7: ISAR Image Across Example Different Noise Levels

Figure 8: Test Accuracies Across Different Noise Standard Deviation Levels Dhonoussa and Fuchi 0 20 40 60 80 100 120 0 10 20 30 40 50 60

Tes

t

Acc

u

rac

y

(P

er

cen

t)

Standard Deviation Noise (dB)

(18)

18

Experiment 2: Effect of Training Data Quantity on Test Accuracy

The neural network seemed to be able to classify images well even at relatively large standard deviations of Gaussian noise. For example, even 30 dB standard deviation Gaussian noised images with 1520 training images produces an average test accuracy of about 91.67% despite the targets being difficult to discern visually. The aspect angle testing was limited by possibly not having sufficient samples in each aspect angle bin. Therefore, the purpose of this experiment was to determine the minimum training sample required. 10 dB standard deviation noised images visually look moderately noised but targets remain clearly visible. The case of 30 dB standard deviation noised images was roughly where it became qualitatively difficult to discern the ships from the noise. It was hypothesized that the 30 dB standard deviation noised images would result in test accuracies that would decay much faster as training size was reduced. This proved to be true, but it seemed that test accuracies still remained roughly as 90% or more as long as there were about 300 or more samples.

(19)

19

Figure 9: Average Test Accuracy as Number of Training Samples Changed Across different dB Noise Levels

Test 3: Test Accuracy across Different Training Angles

The neural network was examined for its performance when trained on one angle and tested on thirty degrees. This was done to get a sense for how much an individual angle contributes to the neural network’s ability to classify an image. The test images were 76 test images of Fuchi and Dhonoussa images all at 30 degrees. Images of Fuchi and Dhonoussa at aspect angles of 0 to 90 degrees at 10 degree increments were used to train the neural network, with 152 images used for each angle. The 152 comes from the fact there were 1520 images for these two classes and ten different angles. For each angle, ten trials with different random seeds were used and the test accuracy results

0 10 20 30 40 50 60 70 80 90 100 0 200 400 600 800 1000 1200 1400 1600 A ver ag e Tes t Acc u racy (P er cen t)

Number of Training Images

Test Accuracy Across Different Noise

and Training Sizes

0 dB Noise 10 dB Noise 30 dB Noise

(20)

20

were averaged. 10 dB noise was used on each image to simulate a situation where there is moderate noise on the image. In general, the test accuracies were higher for smaller angles closer to 30 degrees. The test accuracies were significantly lower for angles greater than 40 degrees, which can be seen in the following figure:

Figure 10: Test Accuracy for Testing on 30 Degrees

Test 4: Test Accuracy for Training on Consecutive Angle Pairs and Testing on 30 Degrees (10 dB Noise)

The neural network was examined for its performance when trained consecutive angle pairs and tested on thirty degrees. This was done to get a sense for how much an individual angle contributes to the neural network’s ability to classify an image. The test images were 76 test images of Fuchi and Dhonoussa images all at 30 degrees. Images of

0 20 40 60 80 100 120 0 20 40 60 80 100

Tes

t

Acc

u

rac

y

Angle Used for Training (Degrees)

Test Accuracy for Testing on 30 Degrees

(21)

21

Fuchi and Dhonoussa at aspect angles of 0 to 90 degrees with consecutive angle pairs 10 degree increments were used to train the neural network, with 152 images used for each angle. For example, one angle pair would be training on 0 and 10 degrees. The next angle pair would be training on 10 and 20 degrees, and this would continue until 80 and 90 degrees were reached. For each angle pair, ten trials with different random seeds were used and the test accuracy results were averaged. 10 dB noise was used on each image to simulate a situation where there is moderate noise on the image. In general, the test accuracies were higher for smaller angles closer to 30 degrees. The test accuracies were significantly lower for angles greater than 40 degrees, which can be seen in the following figure.

Figure 11: Test Accuracy for Testing on 30 Degrees and Training on Pair of Angles

0 20 40 60 80 100 120 0 10 20 30 40 50 60 70 80 90 Av era ge T es t Acc u ra cy Training Data

Lower Number Angle of Pair of Angles (Ex: 10 means 10 and 20 Degrees)

Test Accuracy for Testing on 30 Degrees,

Training on Pairs of Angles

(22)

22

Angle Gap Experiment with No Noise Dhonoussa and Fuchi:

This experiment examined how sensitive the network was to angular granularity in the training data. Consider an experiment where, for a given test angle, that angle and also angles adjacent to that angle are removed. By removing those angles from the training data set, one can see how fast the test accuracy performance dips as one

removes even more adjacent angles. The reason this experimented was conducted was to see how sensitive the neural network is to losing angular information.

Angle Gap Definition:

For example, in the case where one is testing on the angle of 30 degrees, one could first train with all of the angles. The paper defines training on all the available angles as a gap of 0 degrees. Then, one could train with all angles except 30 degrees removed. Removing only the test angle from the training set will be referred to in this paper as a gap of 10 degrees. Then, one could train with all angles except 20, 30, and 40 removed. This removal of the test angle and the two adjacent angles will be denoted a gap of 20 degrees. Continuing this trend, the following figure depicts how an angle gap experiment could be conducted using angles 0 through 90 degrees where each

configuration is repeated three times. In each of those three times, a different random seed was used in order to average over an increased number of trials.

(23)

23

Figure 12: Setup for Gap Experiment for Testing on 30 Degrees

For each row in Figure 15, the test accuracy could be computed. To account for the effects of having less data as angles are removed, the total training data was normalized to be the same number of training images in each row. As the gap of angles removed increased, the test accuracy decreased. This matches the intuition that the neural network should perform worse when the angle being tested and nearby angles are removed from the training data set.

The results of the angle gap test for Dhonoussa and Fuchi can be summarized in a diagram as shown in the following figure. A single column in this figure corresponds to a test angle. From top to bottom, the gap of angles around that test angles is increased. Training image sizes of 304 were used for all experiments because it was found that training image sizes of about 300 seemed to the point before where test accuracy started to drop off precipitously in earlier tests with the 10 dB noise images. Using angles from 0 to 90 degrees, the maximum gap of 80 degrees is possible for angles closer to the

(24)

24

remove. However, for angles closer to the middle of 0 to 90 degrees, increasing the gap to 50 degrees would result in the all angles being removed so there would be no data in that situation. This is why there are white spaces in the following figure where there is no data.

(25)

25

The highest test accuracies are generally on the top rows of Figure 13, as

expected, since the top rows reflect fewer angles removed. The top left corner with low number test angles and low angle gaps tend to have the highest test accuracies. The drop off in performance around 90 degrees was expected because range extent diminishes around then.

One potential problem that could be explored in a future project is to understand what is causing the poor performance for aspect angles greater than roughly 60 degrees. Preliminary trials conducted with and without noise found similar patterns. Furthermore, when one examines the small number angles less than 60 degrees, there is the possibility that for greater gaps, the dropoff in performance may simply be caused by the fact that no angles between 0 and 50 degrees are used in training for greater angle gaps. One hypothesis was that the size of the gap may be less important than simply being able to have any angle represented between 0 and 50 degrees at all for those lower angle values being tested.

Testing with Targets 10 and Target 34:

To see if the trends found with the Dhonoussa and Fuchi ships would apply to other cases, another set of tests was conducted using two different shaped ships. The available ships were labeled by numbers in the database they were retrieved from. Target 10 is the Boris Chilikin-class 162.3 meter length fleet oiler. Target 34 is the Stolt Spray class, a 162.6 meter tanker. These different shapes on the hull compared to each other

(26)

26

and the earlier Dhonoussa and Fuchi targets gave the same diversity to the testing. The angle gap test conducted on the Fuchi and Dhonoussa was repeated using only the Boris Chilikin and Solt Spray with 10 dB and 304 training images. The results are shown in the following figure.

Figure 14: Test Accuracies for Boris Chilikin and Stolt Spray

Figure 14 exhibits the similar patterns as in the Dhonoussa and Fuchi experiment. The top left corner has the highest test accuracy and there is a drop-off of performance around 30 degrees of gap. However, the drop-off happens sooner and the overall test accuracies are larger.

(27)

27

The experiment was repeated for the four class problem with two changes. One change was that the noise standard deviation was reduced from 10 to 5 dB. This was done because it was found that 10 dB noise dampened performance so much that it resulted in more constant, lower test accuracies that made it difficult to see the overall pattern. 5 dB standard deviation of noise produced a plot that would better show the drop-off of performance from increasing the angle gap. Another change was made was to double the number of training images since the number of targets were doubled. This kept the number of training images to number of targets ratio even.

In the following figure with test accuracies for all four classes, the overall pattern of the test accuracies being in the top left corner was still present. However, the overall test accuracies decreased. This was to be expected because in a four class problem, the expected baseline random guess would be a 1 out of 4 chance. This is a smaller

percentage than the 1 out of 2 chance that would be expected for a baseline guess for the two class problem. There are more targets in the four class problem than the two class problem, so there is overall greater chance to misclassify targets. However, the sharp drop-off of test accuracy in the top left corner at around 30 degrees gap remained. The upper left corners of the three previous figures showed a relatively steep drop-off in performance at around 30 degrees of angle gap for test angles from 0 to 30 degrees. The way this paper defined angle gap, this would correspond to removing the test angle as well as the angles that are 10 and 20 degrees away from that angle.

(28)

28

Figure 15: Test Accuracies when Classifying all 4 Classes

3d Data Visualization

As mentioned in the section describing the neural network, a three dimensional layer in the neural network was added in so that its values could be pulled out for

analysis. This gave a way to visualize how similar points were. The classification accuracies calculated in the previous experiments can only give the final result of how the neural network classified images. But classification accuracy does not reveal the deeper underlying reasons of why the neural network made those classifications or what is happening within the neural network. The motivation for creating 3D visualizations of the

(29)

29

datapoints helped give a better picture for what is happening in the intermediate layers inside of the neural network to help understand the origins of the patterns found in the past experiments.

In the following figure with the visualization, the results of the four class problem with the maximum amount of data available was used, which was the entirety of the 3040 training images generated by the simulation. It is notable from the figure that the four classes do end up separating decently well, but the Boris Chilikin and Stolt Spray datapoints are clustered more closely together than the other two classes are. This matched the results of the angle gap tests, where the Boris Chilikin and Stolt Spray test had lower test accuracies than the Dhonoussa and Fuchi test.

As mentioned in the introduction, the goal of the visualization was to analyze the underlying reasons behind the neural network’s behavior. Using classification accuracy alone only gives the final results of which angle dependencies end up mattering, but does not explain why the neural network organizes and distinguishes these different situations internally the way it does. The 3d visualization gave a way to see how the neural network was grouping and organizing the distribution of datapoints. The neural network was trained with 3040 images across the four classes, the maximum images that could be used in this dataset to see how similar the datapoints were to each other. 3040

datapoints would be too difficult to see in a figure so one out of every thirty points were randomly selected to be shown in the Figure 16 to make the plot more readable.

(30)

30 Figure 16: 3d Data Visualization

KL Divergence Analysis:

The KL divergence between entire classes was examined to see how similar the four classes were. There exist other metrics such as the Euclidean distance between centers of two distributions. There is also the Mahalanobis distance, which measures the distance between a point P and a distribution D. However, in the problem of comparing two different classes, there are two distributions of points involved. The KL divergence was used because it takes the entirety of two different distributions into account. One can draw a comparison between the KL divergence table between classes and the 3d visualization of the feature space shown earlier. For example, the distributions of points

(31)

31

corresponding to Boris Chilikin and Solt Spray in the 3d visualization layer are closer together compared to other pairs of classes. This is reflected in the relatively lower KL divergences between the two classes.

Q: Dhonoussa Fuchi Boris Chilikin Stolt Spray P Dhonoussa 0 16.4265 16.3766 18.3446 Fuchi 13.2464 0 14.4883 18.0239 Boris Chilikin 8.3202 3.6111 0 6.416 Stolt Spray 8.7668 10.8954 8.0903 0

Figure 17: KL Divergence Between Classes

Angle Dhonoussa Fuchi

Boris Chilikin Stolt Spray 0 9.112 3.5535 7.7417 6.368 10 7.6347 3.1066 6.2364 1.9718 20 6.4781 3.7613 4.1039 0.9263 30 6.4407 3.2162 2.4983 0.5368 40 2.0669 3.5559 1.701 1.4112 50 1.5802 1.5731 0.0629 1.5533 60 1.394 1.1161 1.7122 1.0251 70 4.1422 1.5689 1.1784 1.5026 80 2.9993 7.3589 1.1732 2.8116 90 2.97 4.3839 1.3127 3.6842

Figure 18: KL Divergence of Points Corresponding to Specific Angle and Class to all the Points in The Class

(32)

32

One feature was that on average, smaller angles tended to have higher KL

divergences from the rest of the points in their class. It was also true in other experiments when testing on those angles that such angles could achieve higher test accuracies than other points. The higher KL divergence may indicate that these smaller angles were further from other points in their respective classes within the feature space, which may be related to why they were able to achieve higher test accuracies.

5. Conclusions:

Improved target recognition for surveillance and reconnaissance will continue to be an important area for national security. Modern machine learning techniques hold promise for improving in neural network performance for automatic target recognition. Analysis of the simulated target data in this project was important to build a theoretical framework for understanding the amount and variation of training data required.

The angle gap tests found that the test accuracy performance dropped off sharply at around 30 degrees of gap. This 30 degree gap, using the definition of gap defined in this paper, corresponds to the removal of the test angle as well removal of all angles plus and minus within 20 degrees in either direction. This suggests a minimum of having angles within 20 degrees of within a test angle, which may reflect the granularity of angles needed. In other words, if angles not even within 20 degrees are present in the training data, then one would expect the test accuracy performance to drop. The KL divergences collected showed that the points corresponding to small angles closer to 0 degrees had greater KL divergences away from their respective classes. This would

(33)

33

suggest that the points corresponding to smaller angles are more dissimilar to the other points in their class. With test accuracies experiments, small angles closer to 0 degrees had higher test accuracies than other angles, which may be reflected in the fact that points corresponding to small angles are more dissimilar to points corresponding to other angles in their class.

Possible Project Extensions:

Further directions for the project would be to understand how much target variation is required across other variables aside from aspect angle and SNR to train a neural network for target recognition. For example, the simulation software could be modified to allow variation in sea state and grazing angle. These variables would affect the ISAR images and their distribution in the feature space, and it would be useful to see how they would affect the minimum amount of target variation required. Another direction could be to include more targets to get a more complete view of the feature space. This would provide a more diverse feature space than the four ships selected for this project.

Another area to explore could be using real data to augment or replace the simulated data. One example where this could play a larger role could be aspect angles around 0 degrees, where real data images often have a drop-off in image quality which was not apparent in the simulation.

(34)

34

Understanding the minimum spanning set to classify neural networks is a problem that becomes increasingly more complex with when one considers the different

architectures, targets, and variables one can change. But by running experiments using common targets, one can begin to see how modern neural networks perform under limited datasets currently available.

(35)

35

Bibliography

[1] N. D. Sidiropoulos, F. Gini, R. Chellappa, and S. Theodoridis, Academic Press library in signal processing, vol. 2. Amsterdam: Elsevier, 2014.

[2] S. Zaied, A. Toumi and A. Khenchaf, "Target classification using convolutional deep learning and auto-encoder models," 2018 4th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sousse, 2018, pp. 1-6.

[3] Xue, B., Tong, N. N., (2019). Real-World ISAR Object Recognition and Relation Discovery Using Deep Relation Graph Learning. Ieee Access, 7: 43906-43914. WOS:000465380900001, https://ieeexplore.ieee.org/document/8678648

[4] Xu, B., Chen, B., et al., (2019). Target-Aware Recurrent Attentional Network for Radar HRRP Target Recognition. Signal Processing, 155: 268-280.

http://www.sciencedirect.com/science/article/pii/S0165168418303220y

[5] Long, T., Liang, Z. N., et al., (2019). Advanced technology of high-resolution radar: target detection, tracking, imaging, and recognition. Science China-Information Sciences, 62(4). WOS:000461313300001,

https://link.springer.com/article/10.1007/s11432-018-9811-0

[6] Lee, H., Golkowski, D., et al., (2019). Relationship of critical dynamics, functional connectivity, and states of consciousness in large-scale human brain networks. NeuroImage, 188: 228-238.

http://www.sciencedirect.com/science/article/pii/S1053811918321554

[7] M. Ma, J. Chen, W. Liu, and W. Yang, “Ship Classification and Detection Based on CNN Using GF-3 SAR Images,” Remote Sensing, vol. 10, no. 12, p. 2043, 2018

[8] Mao and A. Jain, "Artificial neural networks for feature extraction and multivariate data projection", IEEE Transactions on Neural Networks, vol. 6, no. 2, pp. 296-317, 1995. Available: 10.1109/72.363467.

[9] J. Barber, T. Chen, A. Majumdar, " Final Report for ACC 652: Deep Learning for SAR", MIT Lincoln Laboratory, Lexington, MA, Oct. 5, 2017.

[10] Kurt, W. (2020). Kullback-Leibler Divergence Explained — Count Bayesie. [online] Count Bayesie. Available at: https://www.countbayesie.com/blog/201 7/5/9/kullback-leibler-divergence-explained [Accessed 4 Mar. 2020].

[11] Es3dstudios.com. (2020). Game and Simulation 3DModel Outsourcing :: ES3DStudios. [online] Available at: http://www.es3dstudios.com/ [Accessed 6 Mar. 2020]. [12] Es3dstudios.com. (2020). Game and Simulation 3DModel Outsourcing :: ES3DStudios.

(36)

36

[13] Doerry, & Doerry, Armin. (2008). Ship dynamics for maritime ISAR imaging. 10.2172/929523.

Figure

Figure 1: Targets Simulated
Figure 8: Test Accuracies Across Different Noise Standard Deviation Levels Dhonoussa and  Fuchi  020406080100120 0 10 20 30 40 50 60
Figure 9: Average Test Accuracy as Number of Training Samples Changed Across different  dB Noise Levels
Figure 10: Test Accuracy for Testing on 30 Degrees
+7

Références

Documents relatifs

To have a smart microscope, we propose a methodology based on efficient Convolution Neural Network (CNN) architecture in order to classify epidemic pathogen with five deep learning

• Mehr als jeder dritte Kite Student nimmt durchschnittlich 4 bis 6 Stunden Unterricht • Zwei von drei der befragten Schulen haben Wetter und Wind bedingte Probleme • 87%

One key use of aerial photographs was the detection of changes in land features and man-made structures, such as buildings and roads by using photos from several dates

Figure 63 : Capture d'écran de la moyenne des fréquences fondamentales du signal vocal du 2eme sujet pathologique... La fréquence fondamentale moyenne est :

(d) Improving the range migration algorithm: By exploiting the characteristics of the IF signal in FMCW SAR system, an efficient modified RMA which can be used for real time

In the parametric CFAR methods explained in the preceding sec- tions, the background clutter and/or the target pixels are assumed to obey a certain distribution, and the pixels in

Recen t ly ,covar iancema tr iceshavebeenmode ledasrea l iza- t ionsofR iemann ianGauss iand is tr ibu t ions(RGDs)andused inc lass ifica t iona lgor i thmssuchask-meansorExpec ta t

We specifically wish to consider the intraspecific genomic variability as a continuum within structured populations of closely related species, and measure its role on individual