Applications of Kernel Methods
KERNEL METHODS IN IMAGE PROCESSING
Applications of Kernal Methods
regression algorithm for prediction and no analytical tools exist. In this case, one can actually use an SVM to solve the part of the problem where no analytical solution exist and combine the solution with other existing analytical and closed form solutions.
The use of kernelized SVMs has been already proposed to solve a variety of digital communications problems. The decision feedback equalizer (Sebald &
Buclew, 2000) and the adaptive multi-user detector for Code Division Multiple Access (CDMA) signals in multipath channels (Chen et al., 2001)are addressed by means of binary SVM nonlinear classifiers. In (Rah-man et al., 2004) signal equalization and detection for a MultiCarrier (MC)-CDMA system is based on an SVM linear classification algorithm. Koutsogiannis et al. (2002) introduced the use of KPCA for classification and de-noising of communication signals.
KERNEL METHODS IN SIGNAL PROCESSING
Many signal processing supervised and unsupervised schemes such as discriminant analysis, clustering, principal/independent component analysis, or mutual information extraction have been addressed using kernels (see previous chapters). Also, an interesting perspective for signal processing using SVM can be found in (Mattera, 2005), which relies on a different point of view to signal processing.
The use of time series with supervised SVM al-gorithms has mainly focused on two DSP problems:
(1) non-linear system identification of the underlying relationship between two simultaneously recorded discrete-time processes, and (2) time series predic-tion (Drezet and Harrison 1998; Gretton et al., 2001;
Suykens, 2001). In both of them, the conventional SVR considers lagged and buffered samples of the available signals as its input vectors.
These approaches pose several problems and op-portunities. First, the statement of linear signal models in the primal problem, which will be called SVM primal signal models, will allow us to obtain robust estimators of the model coefficients (Rojo-Álvarez et al., 2005a) in classical DSP problems, such as ARMA modeling, the g-filter, and spectral analysis (Rojo-Álvarez et al., 2003, Camps-Valls et al., 2004, Rojo-Álvarez et al., 2004). Second, the consideration of nonlinear SVM-DSP algorithms can be addressed from two different
approaches: (1) RKHS signal models, which state the signal model equation in the feature space (Martínez-Ramón et al., 2005), and (2) dual signal models, which are based on the nonlinear regression of each single time instant with appropriate Mercer’s kernels (Rojo-Álvarez et al., 2005b).
KERNEL METHODS IN SPEECH PROCESSING
An interesting and active research field is that of speech recognition and speaker verification. First, there have been many attempts to apply SVMs to improve exist-ing speech recognition systems. Ganapathiraju (2002) uses SVMs to estimate Hidden Markov Models state likelihoods, Venkataramani et al. (2003) applied SVMs to refine the decoding search space, and in (Gales and Layton, 2004) statistical models for large vocabulary continuous speech recognition were trained using SVMs. Second, early SVM approaches by Schmidt and Gish (1996), and then by Wan and Campbell (2000), used polynomial and RBF kernels to model the distri-bution of cepstral input vectors. Further improvements considered mapping to feature space using sequence kernels (Fine et al. 2001). In the case of speaker veri-fication, the recent works of Shriberg et al. (2005) for processing high-level stylistic or lexical features are worth mentioning.
Voice processing has been performed by using KPCA. Lima et al. (2005) used sparse KPCA for voice feature extraction and then used them for speech rec-ognition. Mak et al. (2005) used KPCA to introduce speaker adaptation in voice recognition schemes.
KERNEL METHODS IN IMAGE PROCESSING
One of the first works proposing kernel methods in the context of image processing was (Osuna et al., 1997), where a face detection system was proposed. Also, in (Papagiorgiou & Poggio, 2000) a face, pedestrian, and car detection method based on SVMs and Haar wavelets to represent images was presented.
The previous global approaches demonstrated good results for detecting objects under fixed viewing condi-tions. However, problems occur when the viewpoint and pose vary. Different methods have been built to
Applications of Kernal Methods
tackle these problems. For instance, the component-
A
based approach (Heisele et al., 2001) alleviates this face detection problem. Nevertheless, the main issues in this context are related to: the inclusion of geometric relationships between components, which were partially addressed in (Mohan et al., 2001) using a two-level strategy; and automatically choose components, which was improved in (Heisele et al., 2002) based on the incorporation of 3D synthetic face models database.
Alternative approaches, completely automatic, have been later proposed in the literature (Ullman et al. 2002), and kernel direct discriminate analysis (KDDA) was used by Lu et al (2006) for face recognition.
Liu et al. (2004) used KICA to model face appear-ance, showing that the method is robust with respect to illumination, expression and pose variations. Zheng et al. (2006) used KKCA for facial expression recogni-tion. Another application is object recognirecogni-tion. In the special case where the objects are human faces, it opens to face recognition, an extremely lively research field, with applications to video surveillance and security (see http://www.face-rec.org/). For instance, (Pontil
& Verri, 1998) identified objects in the COIL database (http://www1.cs.columbia.edu/CAVE/). Vaswani et al.
(2006) use KPCA for image and video classification.
Texture classification using kernel independent compo-nent analysis has been, for example, used by Cheng et al. (2004), and KPCA, KCCA and SVM are compared in Horikawa (2005). Finally, it is worth mentioning the matching kernel (Wallraven et al., 2003), which uses local image descriptors; a modified local kernel (Boughorbel, 2005), or the pyramid local descriptions (Grauman & Darrell, 2005).
Kernel methods have been used in multi-dimensional images, i.e. those acquired in (relatively high) number N of spectral bands acquired from airborne or satel-lite sensors. Support Vector Machines (SVMs) were first applied to hyperspectral image classification in (Gualtieri & Cromp, 1998) and their capabilities were further analyzed in (Camps-Valls et al., 2004) in terms of stability, robustness to noise, and accuracy. Some other kernel methods have been recently presented to improve classification, such as the kernel Fisher dis-criminant (KFD) analysis (Dundar & Langrebe, 2004), or Support Vector Clustering (SVC) (Song, Cherian, &
Fan, 2005). In (Camps-Valls & Bruzzone, 2005), an extensive comparison of kernel-based classifiers was conducted in terms of the accuracy of methods when
working in noisy environments, high input dimension, and limited training sets. Finally, a full family of com-posite kernels for efficient combination of spatial and spectral information in the scene has been presented in (Camps-Valls, 2006).
Classification of functional magnetic resonance images (fMRI) is a novel technique that may lead to a quantity of discovery tools in neuroscience. Clas-sification in this domain is intended to automatically identify differences in distributed neural substrates resulting from cognitive tasks. The application of kernel methods has given reasonable results in accuracy and generalization ability. Recent work by Cox and Savoy (Cox and Savoy, 2003) demonstrated that linear dis-criminant analysis (LDA) and support vector machines (SVM) allow discrimination of 10 class visual activation patterns evoked by the visual presentation of various categories of objects on a trial-by-trial basis within individual subjects. LaConte et al., (2005) used a linear SVM for online pattern recognition of left and right motor activation in single subjects. Wang et al (Wang et al., 2004) applied an SVM classifier to detect brain cognitive states across multiple subjects. In (Martínez-Ramón et al., 2005a), a work has been presented that splits the activation maps into areas, applying a local (or base) classifier to each one. In (Koltchinskii et al, 2005), theoretical bounds on the performance of the method for the binary case have been presented, and in (Martínez-Ramón et al, 2006), a distributed boosting takes advantage of the fact that the distribution of the information in the brain is sparse.
FUTURE TRENDS
Kernel methods have been applied in bioinformatics, signal and speech processing, and communications, but there are many areas of science and engineering in which these techniques have not been applied, namely the emerging techniques of chemical sensing (such as olfaction), forecasting, remote sensing, and many others. Our prediction is that, provided that kernel methods are systematically showing improved results over other techniques, these methods will be applied in a growing amount of engineering areas, as long as to an increasing amount of activity in the areas surveyed in this chapter.
Applications of Kernal Methods
CONCLUSION
This chapter has revised the main applications encoun-tered in the active field of machine learning known as kernel methods. This well-established field has emerged very useful in many application domains, mainly due to the versatility of the provided solutions, the possibility to adapt the method to the application field, the math-ematical elegance and many practical properties. The interested reader can find more information on these application domains in (Camps-Valls, 2006), where a suite of applications and novel kernel developments are provided. The application and development of kernel methods to new fields and also the challenging questions answered so far ensure exciting results in the near future.
REFERENCES
Boughorbel, S. (2005). Kernels for image classification with Support Vector Machines, PhD Thesis, Université Paris 11, Orsay, July 2005.
Camps-Valls, G., & Bruzzone, L. (2005). Kernel-based methods for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 43(6), 1351-1362.
Camps-Valls, G., Gómez-Chova, L., Calpe, J., Soria, E., Martín, J. D., Alonso, L., & Moreno, J. (2004). Robust support vector method for hyperspectral data classifica-tion and knowledge discovery. IEEE Transacclassifica-tions on Geoscience and Remote Sensing, 42(7), 1530-1542.
Camps-Valls, G., Gómez-Chova, L., Muñoz-Marí, J., Vila-Francés, J., & Calpe-Maravilla, J. (2006). Com-posite kernels for hyperspectral image classification.
IEEE Geoscience and Remote Sensing Letters, 3(1), 93-97.
Camps-Valls, G., Martínez-Ramón, M., Rojo-Álva-rez, J.L., and Soria-Olivas, E. (2004). Robust g-filter using support vector machines. Neurocomputing, 62, 493-499.
Camps-Valls, G., Soria-Olivas, E., Perez-Ruixo, J. J., Perez-Cruz, F., Figueiras-Vidal, A.R., Artes-Rodriguez, A. (2002). Cyclosporine Concentration Prediction using Clustering and Support Vector Regression Methods.
IEE Electronics Letters, (12), 568-570.
Camps-Valls, G., Rojo-Álvarez, J. L, and Martínez-Ramón, M. (2006). Kernel methods in bioengineering, signal and image processing. Idea Group, Inc. Hershey, PA (USA). Nov. 2006
Chen, S., Samingan, A. K., & Hanzo, L. (2001). Sup-port vector machine multiuser receiver for DS-CDMA signals in multipath channels. IEEE Transactions on Neural Networks, 12(3), 604 - 611.
Cheng, J., Liu, Q. & Lu, H. (2004). Texture classifi-cation using kernel independent component analysis.
Proceedings of the 17th International Conference on Pattern Recognition, Vol. 1, p. 23-26 Aug. 2004 pp.
620- 623.
Cox, D. D. & Savoy, R. L. (2003). Functional Magnetic Resonance Imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage, 19(2), 261-70.
Drezet, P. and Harrison, R. (1998). Support vector machines for system identification. UKACC Interna-tional Conference on Control’98, 1, 688-692, Swansea, U.K.
Dundar, M., & Langrebe, A. (2004). A cost-effective semi-supervised classifier approach with kernels. IEEE Transactions on Geoscience and Remote Sensing, 42(1), 264-270.
Fine S., Navratil J., & Gopinath, R. A. (2001). A Hybrid GMM/SVM Approach to Speaker Identification. Proc.
IEEE International Conference on Audio Speech and Signal Processing, pp. 417-420.
Gales, M. & Layton, M. (2004). SVMs, Generative Kernels and Maximum Margin Statistical Models.
Beyond HMM Workshop on Statistical Modelling Approach for Speech Recognition.
Ganapathiraju, A. (2002). Support Vector Machines for Speech Recognition. Ph.D. thesis, Mississippi State University.
Grauman, K. & Darrell, T (2005). The Pyramid Match Kernel: Discriminative Classification with Sets of Im-age Features. In Proceedings of the IEEE International Conference on Computer Vision, Beijing, China.
Gretton, A., Doucet, A., Herbrich, R., Rayner, P., and Schölkopf, B. (2001). Support vector regression for black-box system identification. In 11th IEEE Workshop on Statistical Signal Processing, 341-344, NY.
Applications of Kernal Methods
Gualtieri, J. A., & Cromp, R. F. (1998). Support Vector
A
Machines for Hyperspectral Remote Sensing Classifi-cation, 27’th AIPR Workshop, Proceedings of the SPIE Vol. 3584, 221- 232.
Heisele, B., Verri, A., & Poggio, T. (2002). Learning and vision machines. Proc. of the IEEE, 90(7), 1164-1177.
Horikawa, Y., (2005). Modification of correlation ker-nels in SVM, KPCA and KCCA in texture classification, 2005 IEEE International Joint Conference on Neural Networks. IJCNN ‘05. Proceedings. Vol. 4, 31 July-4 Aug. 2005, pp. 2006- 2011.
Koltchinskii, V., Martínez-Ramón, M., Posse, S., 2005.
Optimal aggregation of classifiers and boosting maps in functional magnetic resonance imaging. In: Saul, L.
K., Weiss, Y., Bottou, L. (Eds.), Advances in Neural Information Processing Systems 17. MIT Press, Cam-bridge, MA, pp. 705–712.
Koutsogiannis, G.S. & Soraghan, J., (2002) Classifica-tion and de-noising of communicaClassifica-tion signals using kernel principal component analysis (KPCA), IEEE International Conference on Acoustics, Speech, and Signal Processing, 2002. Proceedings. (ICASSP ‘02).
Vol 2, 2002, pp. 1677-1680.
LaConte, S., Strother, S., Cherkassky, V., J. Anderson,
& Hu, X. (2005). Support vector machines for temporal classification of block design fmri data. Neuroimage, 26, 317-329.
Lima, A., Zen, H., Nankaku, Y., Tokuda, K., Kitamura, T. & Resende, F.G. (2005) Sparse KPCA for Feature Extraction in Speech Recognition, . Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP ‘05) ,Vol. 1, March 18-23, 2005, pp. 353- 356.
Liu, Q, Cheng, J, Lu, H & Ma, S, (2004) Modeling face appearance with nonlinear independent compo-nent analysis, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Proceed-ings, 17-19, May, 2004 Page(s): 761- 766.
Lu, J., Plataniotis, K.N. & Venetsanopoulos, A.N.
(2003). Face recognition using kernel direct discrimi-nant analysis algorithms, IEEE Transactions on Neural Networks, 14(1), 117- 126
Mak, B., Kwok, J.T. & Ho, S., (2005) Kernel Eigenvoice
Speaker Adaptation, IEEE Transactions on Speech and Audio Processing, Vol. 13, No. 5 Part 2, Sept. 2005, pp. 984- 992.
Martínez-Ramón, M., Koltchinskii, V., Heileman, G.,
& Posse, S. (2005a). Pattern classification in functional MRI using optimally aggregated AdaBoosting. In Organization of Human Brain Mapping, 11th Annual Meeting, 909, Toronto, Canada.
Martínez-Ramón, M., Koltchinskii, V., Heileman, G., &
Posse, S. (2005b). Pattern classification in functional mri using optimally aggregated AdaBoost. In Proc. Inter-national Society for Magnetic Resonance in Medicine, 13th Scientific Meeting, Miami, FL, USA.
Martinez-Ramon, M & Christodoulou, C. (2006a).
Support Vector Machines for Antenna Array Process-ing and Electromagnetics, Morgan & Claypool, CA, USA, 2006.
Martínez-Ramón, M., Koltchinskii, V., Heileman, G.,
& Posse, S. (2006b). fMRI pattern classification using neuroanatomically constrained boosting. Neuroimage, 31(3), 1129-1141.
Mattera, D. (2005). Support Vector Machines for Signal Processing. In Support Vector Machines: Theory and Applications. Lipo Wang (Ed.), Springer.
Mikolajczyk, K., & Schmid, C. (2003). A performance evaluation of local descriptors. In Proceedings of the Conference on Computer Vision and Pattern Recogni-tion, Madison, Wisconsin USA, (2) 257-263.
Mohan, A., Papageorgiou, C., & Poggio, T.(2001). Ex-ample-based object detection in images by components.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4), 349-361.
Mukherjee, S., Tamayo, P., Mesirov, J. P., Slonim, D., Verri, A., and Poggio, T. (1998). Support vector machine classification of microarray data. Technical Report 182, C.B.L.C. A.I. Memo 1677.
Osowski, S., Hoai, L. T., & Markiewicz, T. (2004).
Support vector machine-based expert system for reli-able heartbeat recognition. IEEE Trans Biomed Eng, 51(4), 582-9.
Osuna, E., Freund, R., & Girosi, F. (1997). Training Sup-port Vector Machines: an application to face detection.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, 17-19.
Applications of Kernal Methods
Papageorgiou, C. & Poggio, T. (2000) A trainable system for object detection. International Journal of Computer Vision, 38(1), pp 15-33.
Pontil, M. & Verri A. (1998) Support Vector Machines for 3D object recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 637-646.
Rahman, S., Saito, M., Okada, M., & Yamamoto, H.
(2004). An MC-CDMA signal equalization and detec-tion scheme based on support vector machines. Proc.
of 1st Int Symp on Wireless Comm Syst, 1, 11 - 15.
Rojo-Álvarez, J. L., Camps-Valls, G., Martínez-Ramón, M., Soria-Olivas, E., A. Navia Vázquez, & Figueiras-Vidal, A. R. (2005). Support vector machines frame-work for linear signal processing. Signal Proccessing, 85(12), 2316 – 2326.
Rojo-Álvarez, J. L., Martínez-Ramón, M., Figueiras- Vidal, A. R., García-Armada, A., and Artés-Rodríguez, A. (2003). A robust support vector algorithm for non-parametric spectral analysis. IEEE Signal Processing Letters, 10(11), 320-323.
Rojo-Álvarez, J., Figuera, C., Martínez-Cruz, C., Camps-Valls, G., and Martínez-Ramón, M. (2005).
Sinc kernel nonuniform interpolation of time series with support vector machines. Submitted.
Rojo-Álvarez, J., Martínez-Ramón, M., Figueiras-Vi-dal, A., dePrado Cumplido, M., and Artés-Rodríguez, A. (2004). Support vector method for ARMA system identification. IEEE Transactions on Signal Procesing 52(1), 155-164.
Schmidt, M., & Gish, H. (1996). Speaker Identification via Support Vector Classifiers. Proc. IEEE International Conference on Audio Speech and Signal Processing, pp. 105-108.
Schölkopf, B., & Smola, A. (2002). Learning with kernels. Cambridge, MA: MIT Press.
Sebald, D. & Buclew, A. (2000). Support vector machine techniques for nonlinear equalization. IEEE Transac-tions on Signal Processing, 48(11), 3217 - 3226.
Shriberg, E., Ferrer, L., Kajarekar, S., Venkataraman, A., & Stolcke, A. (2005). Modelling Prosodic Feature Sequences for Speaker Recognition. Speech Commu-nication 46, pp. 455-472. Special Issue on Quantitative Prosody Modelling for Natural Speech Description and Generation.
Song, X., Cherian, G., & Fan, G. (2005). A ν-insensi-tive SVM approach for compliance monitoring of the conservation reserve program. IEEE Geoscience and Remote Sensing Letters, 2(2), 99-103.
Srivastava, A. N., & Stroeve, J. (2003). Onboard detec-tion of snow, ice, clouds and other geophysical processes using kernel methods. In Proceedings of the ICML 2003 Workshop on Machine Learning Technologies for Autonomous Space Sciences. Washington, DC USA.
Ullman, S., Vidal-Naquet, M., & Sali, E. (2002) Visual features of intermediate complexity and their use in classification. Nature Neuroscience, 5(7), 1-6.
Vaswani, N. & Chellappa, R. (2006) Principal compo-nents space analysis for image and video classification, IEEE Transactions on Image Processing, Vol. 15 No 7, July 2006, pp.1816- 1830.
Venkataramani, V., Chakrabartty, S., & Byrne, W.
(2003). Support Vector Machines for Segmental Mini-mum Bayes Risk Decoding of Continuous Speech. In Proc. IEEE Automatic Speech Recognition and Un-derstanding Workshop.
Vert, Jean-Philippe (2006). Kernel methods in genomics and computational biology. In book: “Kernel methods in bioengineering, signal and image processing”. Eds.: G.
Camps-Valls, J. L Rojo-Álvarez, M. Martínez-Ramón.
Idea Group, Inc. Hershey, PA. USA.
Wallraven, C., Caputo, B., & Graf, A. (2003) Recogni-tion with local features: the kernel recipe. In Proceedings of the IEEE International Conference on Computer Vision, Nice, France.
Wan, V., & Campbell, W. M. (2000). Support Vector Machines for Speaker Verification and Identification.
Proc. Neural Networks for Signal Processing X, pp.
775-784.
Wang, X., Hutchinson, R., & Mitchell, T. M. (2004).
Training fMRI classifiers to discriminate cognitive states across multiple subjects. In Thrun, S., Saul, L., and Schölkopf, B., editors, Advances in Neural Information Processing Systems 16. MIT Press, Cambridge, MA.
Zheng, W., Zhou, X., Zou, C., & Zhao, L., (2006) Facial expression recognition using kernel canonical correla-tion analysis (KCCA) IEEE Transaccorrela-tions on Neural Networks, Vol. 17, No 1, Jan. 2006, pp. 233- 238.
Applications of Kernal Methods