Surface Flaw Classification Based on Dual Cross Pattern
Zoheir Mentouri Iron and Steel Applied Research
Unit. Research Center in Industrial Technologies-CRTI,
P.O Box 64, Algiers, Algeria [email protected]
Hakim Doghmane Lab. of Inverse Problems Université 8 Mai 1945, BP 401
Guelma, Algeria doghmane.hakim@univ-
guelma.dz
Abdelkrim Moussaoui Lab. of Electrical Engineering Université 8 Mai 1945, BP 401
Guelma, Algeria moussaoui.abdelkrim@univ-
guelma.dz
Djalil Boudjehem Lab. of Advanced Control Université 8 Mai 1945, BP 401
Guelma, Algeria boudjehem.djalil@univ-
guelma.dz
Abstract— The evaluation of flat steel surface quality is mainly concerned with detecting and identifying product surface defects. Although the variety of the implemented techniques, this type of control still presents a challenge. In this paper, we assess the Dual Cross Pattern technique, as a feature descriptor, that should be quite discriminative, to ease the steel surface defect classification. The histograms extracted from the captured DCP features are concatenated to represent the global image feature vector. The procedure parameters, as the DCP circle radius, the number of the training images and their choice, are considered to show their impact on the results. The experiment conducted on the NEU published defect database shows that, compared to the other used techniques, the proposed approach reveals not only interesting recognition rates but presents advantages in time coast too.
Keywords—Image description, Pattern recognition, Product quality, steel surface defects, hot rolling process
I. INTRODUCTION
The pattern recognition for automatic inspection of steel surface products is a field that captured a great attention for the detection and classification of product surface flaws. The effectiveness of such an operation is based on the image defect description, which should be as discriminating as possible. In some applications, steel images have been processed to allow solely a binary segregation (defective/non defective surface) [1], whereas with the increasing surface imperfections, the multi-class identification has been necessary to help optimizing the process correction. For this aim, different techniques have been applied with regard to application needs and defect complexity. So, to classify the types of rolled steel defects as a hole, scratch, clamp or oxides of which the shape may be well defined or not, the Hough transform, the principal component analysis (PCA) and the self organizing map (SOM) have been used in [2]. Further, depending on the defect nature and its complexity, many other techniques, as histogram properties and thresholding, morphological operators or filtering in spatial and joint spatial/frequency domains, have been used to design appropriate image descriptors. For instance, to localize six defects of different types, the applied procedure in [3], has consisted in denoising rolled sheet images by a Wienner filter and in applying a Sobel edge detector, whereas as in [4], a probability pixel distribution was combined with a dynamic thresholding technique for the detection of a complex rolled-in- scale defect. This combination has allowed overcome the pixel level variability, caused by the non uniformity of the reflected light, which highly depends on the surface topography and
texture. However, in spite of their relative efficiency, the mentioned techniques present the constraints of the optimal threshold determination.
The image local transformation LBP, considered as a powerful image texture encoder, with its subsequent variants (SLBP, Mean LBP, Ext. LBP, CLBP, AECLBP, etc) have showed interesting results in the defect detection and in the improvement of the feature vector robustness against noise sensitivity [5-8]. However, in some specific applications, it remained insufficient and discarded some important texture information. Then, in a recent study [9], the Uniform LBP method has been used together with Gabor filtering and co- occurrence matrix to build a small feature space, to detect and classify six types of defects of hot rolled steel surfaces.
In this paper, a novel feature representation of defect images of hot rolled strip based on the Dual Cross pattern (DCP) descriptor is assessed in a multiclass-defect identification procedure, to show its applicability, effectiveness and real time suitability.
In the reminder of this paper, the principle of the DCP as well as its application to the image defects are presented, followed by the experimental study in section three. The results are presented and discussed in the next section, to end the paper with a conclusion.
II. DUAL CROSS PATTERN APPLIED TO STEEL SURFACE DEFECT A. DCP Principle
The Dual Cross pattern, we present in this paper, is a hand- crafted descriptor based method, that deals with the same concept of LBP in shaping a pattern and in encoding the image pixels. However, while this latter uses an encoding method based on the pixel nearest neighborhood, the DCP approach uses more than one neighborhood level. It compares both the central pixel to its nearest neighbors and the neighboring pixels between themselves, to calculate the central pixel code. Thus, as shown in Fig. 1, sampling the DCP points from the positions and according several informative orientations, on the two concentric circles, which define the considered pattern, helps the operator to capture all potential intensity variations of the pixels, since, as known, the occurring steel defects may be, not only of different sizes, but of random directions too.
Fig. 1. Central pixel and two level spatial-sampled points in several orientations
B. Steel defect DCP descriptor
In Fig. 1. the Ai points referred as {A0, .. A7} are situated at equal angles on the inner circle (R1), whereas the Bi={B0, .. B7} are similarly positioned, but on the extern circle (R2).
Basically, to get a new code of a given point with the DCP method, the textural information in each of the eight sampling directions of Fig. 1, is quantized by the equation 1, to assign a decimal number DCPi corresponding to the ith direction.
7 0
2+ − ≤ ≤
−
=S(I I ). S(I I ), i
DCPi A O B Ai
i
i (1)
With S(x), Is the sign function given by:
¯®
≥
Otherwise x if 0
0
1 (2)
And where IO, IAi and IBi represent respectively the pixel intensities of O, Ai and Bi points.
Since the second order statistics of the equation 1, implies a large number of 48 of DCP values to encode all the possible textural information in the eight directions, the computing method consists in creating the pixel codes from two subsets, constituted by the two four-point groups {Ai} and {Bi}, instead of dealing with the entire set of the sampled points; i.e: all orientations. The four points of each created subset are chosen sparse and uniformly distributed on the circles, but with an offset of /4 for the positions of the second subset points. These two groups are: DCP_0={DCP0, DCP2, DCP4, DCP6} for the first and DCP_/4={DCP1, DCP3, DCP5, DCP7} for the second, with the x, in DCPx, relates to a point position. The total number of the local patterns obtained from a such grouping method is, then, 2x44=512. Below, is the illustration of the grouping method in DCP application.
The two pixel codes obtained by the DCP_0 and DCP_/4 cross encoders are formulated as follows:
°¯
°®
=
=
¦
¦
= +
=
i
i ( i )
i
i ( i)
. DCP /
_ DCP
. DCP _
DCP
4 4
4 0
3
0 2 1
3
0 2
π (3)
The final DCP descriptor of each considered image is obtained by the concatenation of the histograms calculated from the DCP_0 and DCP_/4 output images.
III. EXPERIMENTAL STUDY A. Surface defect of steel flat products
The steel defect shape, size and orientation are random and depend on the process parameters and environment, as well as the grade of the rolled steel and the quality of the input product.
In the experiment of this study, the new descriptor is applied to a defect dataset showed in Fig. 2, [15]. This latter has been already used to assess other published methods; then, it would be useful for the result comparison. The dataset is compound of six defect types of hot rolled strip: Rolled-in-scale, pitted surface, scratch, inclusion, patch and crazing. Many variabilities are provided by the 300 variants of each surface defect.
Fig. 2. Samples of the image defects of the NorthEstern (NEU) Database B. Defect recognition procedure
The proposed scheme, in Fig. 4, shows the implemented defect identification procedure.
DCP_/4 Image DCP_0
Image
DCP Cross Encoders subsets
DCPs Defect- images
Global DCP histogram Input images
Defect Image Database
Rolled-in-scale Patch Crazing
Pitted Surface Inclusion Scrach
Fig. 4. General scheme of steel defect recognition
As a reminder, raw images acquired from a production line are mostly preprocessed at first. This step has been avoided and does not appear in Fig. 4. In fact, the random steel flaw may occur scattered in small seeds of scale or inclusions, and may be confused with noise by any filtering operation. Nevertheless, a preprocessing operation by a median filter has been attempted, with no more improvements of the image quality. Moreover, the procedure is being applied to a published dataset, of which images have already been worked and used in [15]. It is, then, worth of remark, that no more preprocessing operation could prevent from any variability information loss within the images and could allow a correct comparison. So, the defect images are used as they are.
Thus, the input defect images, from the NEU database, are processed by the DCP encoders as follows: For a given image, the encoding operation Is executed according the subsets of Fig. 3, and the corresponding histograms are calculated and concatenated. The obtained larger histograms of defect images are then organized in a single and global matrix as shown in the figure below:
Fig. 5. Global matrix of defect image histograms
In a next step, a training set is randomly chosen to create, by the LDAPCA combined method, a model representing a new reduced space. 60-70% of dimension reduction is assumed in [10] to be sufficient, and there, the experiments have been conducted with a dimension reduction about 75%.
With the lack of a very clear criterion, a reduction level around 80% seems suitable in our study, which uses a specific dataset, that are the industrial product images. The reduced subspace is used for projecting the test set images.
Finally, the matching distances are computed for the defect classification.
IV. RESULTS AND DISCUSSIONS
The DCP method is concerned with two main parameters that are the number of sampling points and the circle radii.
Inspired by the validated grouping method in [11], we assess in this application the impact of the circle radii that define the pattern around the considered pixel.
Fig. 6 shows that the combination of {1, 10}, sizing the inner and exterior circle gives the highest defect recognition rate.
All the obtained recognition scores in the tables 1 and 2, represent the average value of 50 trials of rate computing, where for each one, the training set is randomly taken from the DCP encoded set of Fig. 5.
The table 1 shows the average rate, achieved with the proposed approach compared to other assessed methods, which have been used with their optimal parameters values.
The ‘D’ and ‘S’ in Gabor method indicate respectively the orientation and scale numbers, used in creating a bank of DxS filters for a Gabor multiscale application, whereas, in the Local Phase Quantization (LPQ), the ‘W.S’ is the used window size. As for the Binarized Statistical Image Features (BSIF), the ‘S’ and ‘B’ parameters concern the filter scale and the bit string (i.e: codeword resolution).
For the classification task, all the assessed descriptors have been combined with the two widely used classifiers: the nearest neighbors (KNN) and the multi-class support vector machine (SVM), and where the most suitable parameters of each method have been considered. The KNN classifier has been applied based on three neighbors and the Euclidian matching distance. Whereas with SVM, the Gaussian Radial Basis Function K(x, x’), formulated in Eq. 4, has been used.
We found, empirically, that this kernel function allows better results compared to the other ones as Polynomial, Laplace RBF or sigmoid functions.
' 2
' ) exp( x x
x , x (
K = −σ −
(4) In this paper, the parameter in the RBF kernel function has been empirically selected (=0.0001).
Fig. 6. Defect recognition rates as a function of circle radii pairs
Surface Defect images
Training Set Partition
Test Set Partition DCP Processing
Feature Extraction LDAPCA
Encoded-Images Projection
Steel Defect Classification
…… … … …
ith image jth class
Crazing variants
Inclusion variants
TABLE I. AVERAGE RATES OF NEUDATABASE DEFECTS RECOGNITION
Processing. tools Tool parameters Feat. Vector
length Classifier Results (%)
Gabor
D=4, S=2 2048 KNN3 82.40±1.37
SVMrbf 79.22±1.12
D=8, S=5 10240 KNN3 86.63±1.11
SVMrbf 88.36±1.01
LPQ W.S=9x9 256 KNN3 96.18±0.57
SVMrbf 95.58±030
Gabor_LPQ
D=4, S=2, W.S=9x9 2048 KNN3 99.12±0.29
SVMrbf 98.67±0.32
D=8, S=5, W.S=9x9 10240 KNN3 98.98±0.25
SVMrbf 98.44±0.86
BSIF Filter
S=7, B=12 4096 KNN3 99.18±0.30
SVMrbf 84.64±1.14
DCP Cir. Radius = [1 10] 512 KNN3 99.42±0.23
SVMrbf 98.99±0.39
DCP_PCA Cir. Radius = [2 3] 512 KNN3 95.25±0.66
The tuned parameters, with the DCP approach, are the circle radii. The highest score is obtained with a pair of inner and exterior circle radii respectively of 1 and 10. Moreover, the feature vector length is the shortest compared to the ones of the other evaluated methods, what means more time saving in data processing, and shows the method suitability for a real time process application. As for the lowest value of the standard deviation, it shows the efficiency and the robustness of the proposed approach DCP_LDAPCA_KNN
This result decreases significantly when PCA method is used, alone, in data reduction. As shown in the same table, the best achieved recognition rate is around 95%, obtained with circle radii of 2 and 3. This data reduction method is known for its good representation of class data variations, however with insufficient data-class distinction.
The curves in Fig. 7, show the evolution of the rate error when the size of the training set increases. All methods are used with the combined data reduction method LDAPCA and the KNN classifier. The proposed approach performs consistently better than the others.
Furthermore, in the steel defect identification, the numerous carried out studies has multiplied, combined and innovated, to improve more and more results and assure a maximum of method robustness. Some of the results of the previous works, that dealt with the same defect database as the present study’s one, are presented in the table below. The comparison confirms the relevance of the proposed approach for such an application and shows its higher achievement.
TABLE II. COMPARISON OF RECOGNITION RATES OF NEU DATABASE DEFECTS
Work
Ref. Features Descriptor Class. R (%)
[12] DST–GLCM SVM 94.11
[13] BSIF KNN 99.18±0.30
[14] CNN CNN 99.05
[15] SCN SVM 98.60±0.59
[16] AECLBP SVM 98.93±0.63
[16] CLBP SVM 98.28±0.51
[17] GW_LPQ KNN 99.12±0.29
Present
work DCP_LDAPCA KNN 99.42±0.23
V. CONCLUSION
The presented work focused the surface defect recognition of flat steel products in hot rolling mill. The randomness of the properties of the rolled-in-scale, patches, inclusions, scratches, crazing and pitted surface defects, makes of identifying them, a challenging task. This has been a suitable issue to assess the DCP approach. The experiment has been carried out with the NEU dataset, which is publicly available and has been used in several published works. The efficient defect discrimination of the DCP method, comes from its ability to capture the maximum of the variations within the image, and in the most informative directions.
relevance of our approach is confirmed by the achievement of the highest recognition rate and the lowest standard deviation, which reflects the method robustness. These results are obtained with the shortest feature vector, what means that even for a real time operation, our approach would be suitable.
REFERENCES
[1] Smriti H. Bhandari, S. M. Deshpande, S. M. Deshpande, “A Simple Approach to Surface Defect Detection”, 2008 IEEE Region 10 , Colloquium and the Third International Conference on Industrial and Information Systems, Kharagpur, December 8-10, (2008), INDIA.
[2] Luiz A. O. Martins, Flavio L. C. Padua, Paulo E. M. Almeida, Automatic Detection of Surface Defects on Rolled Steel Using Computer Vision and Articial Neural Networks, Conference Paper (2010), DOI:10.1109/IECON.2010.5675519, Source:IEEE Xplore A.B. Smith, C.D. Jones, and E.F. Roberts, “Article Title,” Journal, vol. 1, no. 3, pp. 1-10, Mar. 2000.
[3] JM Blackledge, DA Dubovitskiy, A surface inspection machine vision system that includes fractal texture analysis, international society for advanced science and technology. J Intell Syst 3(2), 76–89 (2008)
[4] D Djukic, S Spuzic, Statistical Discriminator of Surface Defects on Hot Rolled Steel (Proceedings of Image and Vision Computing (University of Waikato,Hamilton, New Zealand, 2007), pp. 158–163 [5] Maoxiang CHU and Rongfen GONG, Invariant Feature Extraction
Method Based on Smoothed Local Binary Pattern for Strip Steel Surface Defect , ISIJ International, Vol. 55 (2015), No. 9, pp. 1956–
1962
[6] T Maenpaa, Surface Quality Assessment with Advanced Texture Analysis Techniques (Proc. of International Surface Inspection Summit, Luxembourg, 2006).
[7] Kechen Song, Yunhui Yan, A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects, Applied Surface Science 285P (2013), 858-864, available at:www.elsevier.com/locate/apsusc.
[8] Timo Ojala, Matti Pietikäinen, Senior Member, IEEE, and Topi MäÈenpää, Multiresolution Gray-Scale and Rotation Invariant
Texture Classification with Local Binary Patterns. IEEE Trans. on pattern analysis and machine intelligence, Vol. 24, N°. 7, July 2002.
[9] Mang Xiao1, Mingming Jiang2*, Guangyao Li2, Li Xie2 and Li Yi2.
An evolutionary classifier for steel surface defects with small sample set EURASIP Journal on Image and Video Processing, Processing (2017) 2017:48, DOI 10.1186/s13640-017-0197-y
[10] A. Hyvarinen, J. Hurri, P. O. Hoyer, Natural Image Statistics. A probabilistic approach to early computational vision, February 27, 2009, Springer
[11] C. Ding, J. Choi, D. Tao, and L. S. Davis, Multi-directional multi- level dual-cross patterns for robust face recognition, IEEE transactions on pattern analysis and machine intelligence, 38(3): 518–
531, 2016.
[12] M. W. Ashour, F. Khalid, A. Abdul Halin, L. N. Abdullah, S.H.
Darwish. Surface Defects Classication of Hot-Rolled Steel Strips Using Multi-directional Shearlet Features. Arabian Journal for Science and Engineering. Springer (2018).
https://doi.org/10.1007/s13369-018-3329-5
[13] Mentouri Zoheir, Moussaoui Abdelkrim, Boudjehem Djalil. Steel Strip Surface Defect Identification Based on Binarized Statistical Features. Scientific Bulletin, Series B : Chemistry and Materials Science, Vol. 80, Iss. 4, 2018, ISSN 1454-2331, U.P.B Romania, [14] Li Yi, Guangyao Li, and Mingming Jiang, An End-to-End Steel Strip
Surface Defects Recognition System Based on Convolutional Neural Networks, Steel research int. 87, 2016, No. 9999. DOI:
10.1002/srin.201600068, available at : www.steel-research.de.
[15] Kechen SONG, Shaopeng HU and Yunhui YAN, Automatic Recognition of Surface Defects on Hot-rolled Steel Strip using Scattering Convolution Network, Journal of Computational Information Systems 10: 7, 3049–3055, 2014
[16] Kechen Song, Yunhui Yan, A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects, Applied Surface Science 285P (2013), 858-864, available at : www.elsevier.com/locate/apsusc.
[17] Zoheir Mentouri, Hakim Doghmane, Abdelkrim Moussaoui, Houcine Bourouba, Adel Boudiaf. New Image Feature Descriptors for Efficient Steel Defect Recognition. 6thInternational Conférence on Computational and Experimental science and Engineering, ICCESEN-2019, October 2019, Antalya-Turkey
