4 SVM Application: Face Detection in Images
4.3 Future Directions in Face Detection and SVM Applications
Future research in SVM application can be divided into three main categories or topics:
1.
Simplication of the SVM:
One drawback for using SVM in some real-life applications is the large number of arithmetic operations that are necessary to classify a new input vector. Usually, this number is proportional to the dimension of the input vector and the number of support vectors obtained. In the case of face detection, for example, this is283,000 multiplications per pattern!The reason behind this overhead is in the roots of the technique, since SVM's dene the decision surface by explicitly using data points. This situation causes a lot of redundancy in most cases, and
can be solved by relaxing the constraint of using data points to dene the decision surface. This is a topic of current research conducted by Burges [6] at Bell Laboratories, and it is of great interest for us, because in order to simplify the set of support vectors, one needs to solve highly-nonlinear constrained optimization problems.
Since a closed form solution exists for the case of kernel functions that are 2nd. degree polynomials, we are using a simplied SVM [6] in our current experimental face detection system that gains an acceleration factor of 20, without degrading the quality of the classications.
2.
Detection of other objects:
We are interested in using SVM's to detect other objects in digital images, like cars, airplanes, pedestrians, etc. Notice that most of these objects have very dierent appearance, depending on the viewpoint. In the context of face detection, an interesting extension that could lead to better understanding and approach to other problems, is the detection of tilted and rotated faces. It is not clear at this point, whether these dierent view of the same object can be treated with a single classier, or if they should be treated separately.3.
Use of multiple classiers:
The use of multiple classiers oers possibilities that can be faster and/or more accurate. Rowley et al. [19] have successfully combined the output from dierent neural networks by means of dierent schemes of arbitration in the face detection problem. Sung and Poggio [22][21] use a rst classier that is very fast as a way to quickly discard patterns that are clearly non-faces. These two references are just examples of the combination of dierent classiers to produce better systems.Our current experimental face detection system performs an initial quick-discarding step using a SVM trained to separate clearly non-faces from probable faces using just 14 averages taken from dierent areas of the window pattern. This classication can be done about 300 times faster and is currently discarding more than 99% of input patterns. More work will be done in the near future in this area.
The classiers to be combined do not have to be of the same kind. An interesting type of classier that we will consider is Discriminant Adaptative Nearest Neighbor due to Hastie et al. [11][10].
5 Conclusions
In this paper we presented a novel decomposition algorithm to train Support Vector Machines. We have successfully implemented this algorithm and solved large dataset problems using acceptable amounts of computer time and memory. Work in this area can be continued, and we are currently studying new techniques to improve the performance and quality of the training process.
The Support Vector Machine is a very new technique, and, as far as we know, this paper presents the second problem-solving application to use SVM, after Vapnik et al. [5, 8, 24] used it in the character recognition problem, in 1995.
Our Face Detection System performs as well as other state-of-the-art systems, and has opened many interesting questions and possible future extensions. From the object detection point of view, our ultimate goal is to develop a general methodology that extends the results obtained with faces to handle other objects. From a broader point of view, we also consider interesting the use of the function approximation or regression extension that Vapnik [24] has done with SVM, in many dierent areas where Neural Networks are currently used.
Finally, another important contribution of this paper is the application of OR-based techniques to domains like Statistical Learning and Articial Intelligence. We believe that in the future, other tools like duality theory, interior point methods and other optimization techniques and concepts, will be useful in obtaining better algorithms and implementations with a solid mathematical background.
References
[1] M. Bazaraa, H. Sherali, and C. Shetty. NonLinear Programming Theory and Algorithms. J. Wiley, New York, 2nd edition, 1993.
[2] D. Bertsekas. Projected newton methods for optimization problems with simple constraints. SIAM J.
Control and Optimization, 20(2):221{246, March 1982.
[3] B.E. Boser, I.M. Guyon, and V.N. Vapnik. A training algorithm for optimal margin classier. In Proc.
5th ACM Workshop on Computational Learning Theory, pages 144{152, Pittsburgh, PA, July 1992.
[4] G. Burel and D. Carel. Detection and localization of faces on digital images. Pattern Recognition Letters, 15:963{967, 1994.
[5] C. Burges and V. Vapnik. A new method for constructing articial neural networks. Technical report, AT&T Bell Laboratories, 101 Crawfords Corner Road Holmdel NJ 07733, May 1995.
[6] C.J.C. Burges. Simplied support vector decision rules, 1996.
[7] T. Carpenter and D. Shanno. An interior point method for quadratic programs based on conjugate projected gradients. Computational Optimization and Applications, 2(1):5{28, June 1993.
[8] C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:1{25, 1995.
[9] F. Girosi. An equivalence between sparse approximation and Support Vector Machines.
A.I. Memo 1606, MIT Articial Intelligence Laboratory, 1997. (available at the URL:
http://www.ai.mit.edu/people/girosi/svm.html).
[10] T. Hastie, A. Buja, and R. Tibshirani. Penalized discriminant analysis. Technical report, Statistics and Data Analysis Research Department, AT&T Bell Laboratories, Murray Hill, New Jersey, May 1994.
[11] T. Hastie and R. Tibshirani. Discriminat adaptative nearest neighbor classication. Technical report, Department of Statistics and Division of Biostatistics, Stanford University, December 1994.
[12] N. Kruger, M. Potzsch, and C. v.d. Malsburg. Determination of face position and pose with learned representation based on labled graphs. Technical Report 96-03, Ruhr-Universitat, January 1996.
[13] J. Mercer. Functions of positive and negative type and their connection with the theory of integral equations. Philos. Trans. Roy. Soc. London, A 209:415{446, 1909.
[14] B. Moghaddam and A. Pentland. Probabilistic visual learning for object detection. Technical Report 326, MIT Media Laboratory, June 1995.
[15] E.H. Moore. On properly positive Hermitian matrices. Bull. Amer. Math. Soc., 23(59):66{67, 1916.
[16] J. More and G. Toraldo. On the solution of large quadratic programming problems with bound constraints. SIAM J. Optimization, 1(1):93{113, February 1991.
[17] B. Murtagh and M. Saunders. Large-scale linearly constrained optimization. Mathematical Program-ming, 14:41{72, 1978.
[18] B. Murtagh and M. Saunders. MINOS 5.4 User's Guide. System Optimization Laboratory,Stanford University, Feb 1995.
[19] H. Rowley, S. Baluja, and T. Kanade. Human face detection in visual scenes. Technical Report CMU-CS-95-158R, School of Computer Science, Carnegie Mellon University, November 1995.
[20] J. Stewart. Positive denite functions and generalizations, an historical survey. Rocky Mountain J.
Math., 6:409{434, 1976.
[21] K. Sung. Learning and Example Selection for Object and Pattern Detection. PhD thesis, Massachusetts Institute of Technology, Articial Intelligence Laboratory and Center for Bilogical and Computational Learning, December 1995.
[22] K. Sung and T. Poggio. Example-based learning for view-based human face detection. A.I. MEMO 1521, C.B.C.L Paper 112, December 1994.
[23] V. Vapnik. Estimation of Dependences Based on Empirical Data. Springer-Verlag, 1982.
[24] V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 1995.
[25] V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequences of events to their probabilities. Th. Prob. and its Applications, 17(2):264{280, 1971.
[26] V.N. Vapnik and A. Ya. Chervonenkis. The necessary and sucient conditions for consistency in the empirical risk minimization method. Pattern Recognition and Image Analysis, 1(3):283{305, 1991.
[27] G. Yang and T. Huang. Human face detection in a complex background. Pattern Recognition, 27:53{63, 1994.
[28] W.Y. Young. A note on a class of symmetric functions and on a theorem required in the theory of integral equations. Philos. Trans. Roy. Soc. London, A 209:415{446, 1909.
[29] G. Zoutendijk. Methods of Feasible Directions. D. Van Norstrand, Princeton, N.J., 1960.
Figure 8: Some false detections obtained with the rst version of the system. This false positives were later used as negative examples ( class -1 ) in the training process