Homework 5

Due: April 28, 2005

You may work with others on this assignment, but you should turn in separate writeups, and you should understand the solutions. Consult the book and your professor for help if you need it. If you work with someone else, you must acknowledge them on your report.

This assignment must be done in LaTeX and turned in printed from Postscript or PDF file format.

Announcements

Tuesday, April 12, 2005: The assignment has been posted.

Assignment

Reading. You should thoroughly read chapter 4.5 (especially 4.5.2) and chapter 12 (through 12.3.2) of your textbook.
Describe kernels. In your own words, explain the concept of what a kernel is. Does a kernel have an advantage over learning directly in a higher-dimensional space? What does the "kernel trick" mean?
Learning with support vector machines. Using the SVM^light software package, you should experiment and find classification results on the binary classification problem between the digit 4 and the digit 9 in the handwritten digits dataset that we have used in past assignments. Note that you will have to re-format the data so that SVM^light can read it.
For SVM^light, you should use the default for most parameters. However, you should try varying:
- The type of kernel using the -t parameter (and the -d/-g/-s/-r parameters as appropriate). If you are feeling creative, make your own kernel!
- The trade-off between training error and margin with the -c option (recall that larger values create a smaller margin, and allow fewer training errors).
- Performance estimation with leave-one-out cross-validation with the -x 1 option.
Here are some good questions that you should consider (and answer!) for experiments you conduct with SVMs:
- How many support vectors are chosen? What percentage of the dataset is used as support vectors?
- Which support vectors are chosen?
- What is the training error? What is the test error?
- Is the learned model interpretable in the original feature space? Give your estimate of what the model means in the original feature space.
Compare the results you obtain on this task with the results using the the 1-nearest neighbor classifier. Compare these approaches also in terms of model interpretability, overfitting issues, etc.

valid html and css