Everything old is new again : a fresh look at historical approaches in machine learning

Abstract

This thesis shows that several old, somewhat discredited machine learning techniques are still valuable in the solution of modern, large-scale machine learning problems. We begin by considering Tikhonov regularization, a broad framework of schemes for binary classification. Tikhonov regularization attempts to find a function which simultaneously has small empirical loss on a training set and small norm in a Reproducing Kernel Hilbert Space. The choice of loss function determines the learning scheme. Using the hinge loss gives rise to the now well-known Support Vector Machine algorithm. We present SvmFu, a state-of-the-art SVM solver developed as part of thesis. We discuss the design and implementation issues involved in SvmFu, present empirical results on its performance, and offer general guidance on the use of SVMs to solve machine learning problems. We also consider, and advocate in many cases, the use of the more classical square loss, giving rise to the Regularized Least Squares Classifiation algorithm. RLSC is "trained" by solving a single system of linear equations. While it is widely believed that the SVM will perform substantially better than RLSC, we note that the same generalization bounds that apply to SVMs apply to RLSC, and we demonstrate empirically on both toy and real-world examples that RLSC's performance is essentially equivalent to SVMs across a wide range of problems, implying that the choice between SVM and RLSC should be based on computational tractability considerations. We demonstrate the empirical advantages and properties of RLSC, discussing the tradeoffs between RLSC and SVMs.

(cont.) We also prove leave-one-out bounds for RLSC classification. Next, we turn to the problem of multiclass classification. Although a large body of recent literature exists suggesting the use of sophisticated schemes involving joint optimization of multiple discriminant functions or the use of error-correcting codes, we instead advocate a simple "one-vs-all" approach in which one classifier is trained to discriminate each class from all the others, and a new point is classified according to which classifier fires most strongly. We present empirical evidence of the strength of this scheme on real-world problems, and, in the context of RLSC as the base classifier, compelling arguments as to why the simple one-vs-all scheme is hard to beat. We also present leave-one-out bounds for multiclass classification where the base learners belong to a broad class of Tikhonov regularizers. Finally, we consider algorithmic stability, a relatively new theory that results in very elegant generalization bounds for algorithms which are "stable". We compare and contrast Tikhonov regularization, to which algorithmic stability applies, with Ivanov regularization, the form of regularization that is the basis for structural risk minimization and its related generalization bounds. We present some interesting examples which highlight the differences between the Tikhonov and Ivanov approaches, showing that the Tikhonov form has a much stronger stability than the Ivanov form ...