Class Activity 15: Model tuning

• train.csv
• dev.csv
• test.csv

Scaled features:

• train_scaled.csv
• dev_scaled.csv
• test_scaled.csv

As I outlined in one of my earlier videos, the development set (dev.csv and dev_scaled.csv) are used in order to tune a model. Every model has parameters that can be adjusted to increase it's predictive power (some of these are feature based and therefore separate from the machine learning model, e.g., scaling features). Here are the ones I want you to consider:

• feature type (scaled, unscaled)
• k
• distance metric (euclidean, manhattan)

Using your kNN implementation, or mine (see the gi14-solutions folder in the class Dropbox), use train.csv (or train_scaled.csv) as the "training set" and dev.csv (or dev_scaled.csv) as the "testing set". Try using the original features as well as the unscaled; try different values of k on each, and try both distance measures. If you choose 5 different values of k to try, there are 5x2x2 = 20 different combinations of parameters if you want to be exhaustive, so either choose fewer values of k or don't be exhaustive in your search for the best model.  I suggest redirecting the output of your kNN program to files with descriptive names, e.g., "dev_scaled_knn_k4_euclidean.csv" used scaled features, set k=4, and used Euclidean distance. Try at least 10 different parameter combinations.

Keep in mind that if you use the scaled version of the training data, you have to use the scaled version for development.

• list the parameter combinations you tried and evaluated before arriving at your final model
• for your final model, report the evaluation statistics for both spam and ham on:
  • the development set
  • the testing set