BIOS 362: Advanced Statistical Inference (Statistical Learning)

Instructor

Teaching Assistant

  • Nick Strayer

Dates, Time, and Location

  • First meeting: Mon. Jan. 7, 2019; Last meeting: Mon. Apr. 22, 2018
  • Monday, Wednesday, and Friday 11:00-11:55AM, Room 11139 (small classroom), 11th floor, 2525 WEA
  • Office hours: open door and by request
  • We will use the Graduate School Academic Calendar

Textbook

The book for this course is listed below, and free to download in PDF format at the book webpage: Hastie, Tibshirani, Friedman. (2009) The elements of statistical learning: data mining, inference and prediction. Springer, 2nd edition.. In the course outline and class schedule, the textbook is abbreviated "HTF", often followed by chapter or page references "Ch. X-Y" or "pp. X-Y", respectively. The BibTeX entry for the book is as follows:
@book{HTF2009,
  author = {Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome},
  title = {The elements of statistical learning: data mining, inference and prediction},
  url = {http://www-stat.stanford.edu/~tibs/ElemStatLearn/},
  publisher = {Springer},
  year = 2009,
  edition = 2
}

The wide margins of the linked PDF version of the book make it difficult to read on smart devices (e.g., an iPhone). The margins may be removed using the following ghostscript command in Linux, where "output.pdf" and "input.pdf" are substituted for the appropriate file names. Please see Dr. Shotwell for help with this.
gs -o output.pdf -sDEVICE=pdfwrite -c "[/CropBox [130 140 460 685] /PAGES pdfmark" -f input.pdf

Other Resources

Course Topics

  • Overview of Supervised Learning and Review of Linear Methods: HTF Ch. 2-4
  • Splines and Kernel Methods: HTF Ch. 5-6
  • Model Assessment, Selection, and Inference: HTF Ch. 7-8
  • Neural Networks: HTF Ch. 11
  • Support Vector Machines: HTF Ch. 12
  • Unsupervised Learning: HTF Ch. 14

Other information

  • Unless otherwise stated, assigned homework is due in one week.
  • Students are encouraged to work together on homework problems, but they must turn in their own write-ups.
  • Class participation is encouraged.
  • Please bring a laptop to class.

Grading

  • Homework: 40%
  • Take-home Midterm Exam: 30%
  • Take-home Final Exam: 30%

Schedule of Topics

Date Reading (before class) Homework Topic/Content Presentation
Mon. 1/7 HTF Ch. 1 and Ch. 2.1, 2.2, and 2.3 HW Mon. 1/08 Syllabus, introduction, least-squares, nearest-neighbors lecture-1.pdf
Wed. 1/9 HTF Ch. 2.4 none Decision theory lecture-2.pdf
Fri. 1/11 HTF Ch. 2.7, 2.8, and 2.9 none structured regression , bias-variance tradeoff lecture-3.pdf
Mon. 1/14 none none Nonlinear methods: least squares lecture-4.pdf
Wed. 1/16 HTF Ch. 3.1, 3.2, 3.3, and 3.4 none Linear methods: subset selection, ridge, and lasso lecture-5.pdf
Fri. 1/18 HTF Ch. 3.5 and 3.6 HW Fri. 1/18 Linear methods: principal components regression lecture-6.pdf
Mon. 1/21 No Class none Martin Luther King Jr. Holiday (no class)  
Wed. 1/23 none none More SVD, g-inverses, and PCA; pca-and-g-inverses.html lecture-7.pdf
Fri. 1/25 HTF Ch. 4.1, 4.2, and 4.3 none Linear methods: Linear discriminant analysis lecture-8.pdf
Mon. 1/28 HTF Ch. 4.4 and 4.5 HW Mon. 1/28 Linear methods: Reduced-rank LDA lecture-9.pdf
Wed. 1/30Fri. 2/1 HTF Ch. 5.1 and 5.2 none Basis expansions: piecewise polynomials & splines lecture-11.pdf splines-example.R
Fri. 2/1 HTF Ch. 4.4 and 4.5 none Linear methods: Logistic regression; discuss HW progress lecture-10.pdf lecture-10.R
Mon. 2/4 HTF Ch. 5.4 and 5.5 none Smoothing Splines lecture-12.pdf
Wed. 2/6 HTF Ch. 5.7 HW Wed. 2/06 Multidimensional splines lecture-12.pdf
Fri. 2/8 none none Guest lecture by Nick Strayer: Introduction to Neural networks  
Mon. 2/11 HTF Ch. 6.1-6.5 none No Class: Please use class time to read the book's introduction to kernel methods.  
Wed. 2/13 none none Kernel methods lecture-13.pdf mixture-data-complete.R
Fri. 2/15 HTF Ch. 7.1, 7.2, 7.3, 7.4 none Model assessment: Cp, AIC, BIC lecture-14.pdf
Mon. 2/18 HTF Ch. 7.10 none Cross validation lecture-15.pdf
Wed. 2/20 HTF Ch. 8.1, 8.2, 8.3 none Model-based inference lecture-17.pdf
Fri. 2/22 none none Bootstrap Iteration lecture-16.pdf
Mon. 2/25 HTF Ch. 8.7, 8.8, 8.9 none Bagging lecture-18.pdf
Wed. 2/27 HTF Ch. 9.1 none Generalized Additive Models lecture-20.pdf LAozone.R
Fri. 3/1 HTF Ch. 9.2 none No Class: Please use class time to read the book's introduction to Classification and Regression Trees.  
Mon. 3/4 none none Spring Break (no class)  
Wed. 3/6 none none Spring Break (no class)  
Fri. 3/8 none none Spring Break (no class)  
Mon. 3/11 none none Phoneme recognition, midterm assignment lecture-19.pdf
Wed. 3/13 HTF Ch. 9.2 none Classification and Regression Trees lecture-21.pdf
Fri. 3/15 HTF Ch. 10.1 none Boosting and AdaBoost.M1 (part 1) lecture-22.pdf boosting-trees.R
Mon. 3/18 HTF Ch. 10.2-10.9 none Boosting and AdaBoost.M1 (part 2) lecture-23.pdf
Wed. 3/20 HTF Ch. 10.10, 10.13 none Boosting and AdaBoost.M1 (part 3) lecture-24.pdf
Fri. 3/22 HTF Ch. 15.1, 15.2   No Class: Please use class time to read the book's introduction to Random Forest  
Mon. 3/25 none HTF Ex. 10.8, 10.9 Random Forest lecture-25.pdf
Wed. 3/27 HTF Ch. 12.1, 12.2 none Support Vector Classifier lecture-26.pdf
Fri. 3/29 HTF Ch. 12.3 none Support Vector Machine lecture-27.pdf mixture-data-svm.R
Mon. 4/1 HTF Ch. 11.1, 11.2, 11.3, 11.4, 11.5 none Neural networks lecture-31.pdf nnet.R
Wed. 4/3 HTF Ch. 11.1, 11.2, 11.3, 11.4, 11.5 none Neural networks (continued) lecture-31.pdf
Wed. 4/5 HTF Ch. 11.1, 11.2, 11.3, 11.4, 11.5 none Neural networks (continued) lecture-31.pdf
Mon. 4/8 none none Neural networks (continued) mnist-convnet.R
Wed. 4/10 HTF Ch. 14.5 none Principal curves and surfaces lecture-28.pdf principal-curves.R
Fri. 4/12 HTF 14.8 none Multidimensional scaling lecture-30.pdf
Mon. 4/15 HTF 14.5.3 none k-means, hierarchical, and spectral clustering lecture-29.pdf
Wed. 4/17 none none Clustering with mixtures lecture-32.pdf
Fri. 4/19 none none Review final exam assignment  

Homework/Laboratory (other than problems listed in HTF)

Mon. 1/08

  • Due before class Wed. 1/16
  • Write a knitr or Sweave document to implement the following tasks:
    • Download the Mixture Simulation data from the authors' website: data.
    • Using HTF expressions 2.1-2.8, implement the linear regression and nearest neighbors classifiers.
    • Reproduce HTF figures 2.1, 2.2, and 2.3. For the nearest-neighbors classifier, use the contourLines function in R.

Fri. 1/18

Write an R script or Sweave (knitr) document to implement the following tasks:
  • Download the Prostate Cancer data from the authors' website or using the ElemStatLearn R package.
  • Review section 3.2.1 "Example: Prostate Cancer"
  • Reproduce the correlation listed in HTF Table 3.1, page 50.
  • Implement the least-squares regression method using HTF expressions 3.6, 3.7, and 3.8 (pages 45-57), and the relevant discussion (i.e., do not use the 'lm' function).
  • Verify the least-squares regression test error listed on HTF page 51.
  • Implement ridge regression using HTF expression 3.44 (page 64) and the relevant discussion.
  • Implement principal components regression (PCR) using HTF expressions 3.61 and 3.62.
  • Reproduce the ridge and PCR regression coefficients and test error estimates listed in Table 3.3 (page 62) using Figure 3.7 (page 62), and expression 3.50 (page 68).
  • (optional) Use the glmnet package to reproduce the lasso estimates in Table 3.3.

Mon. 1/28

  • Retrieve the vowel data (training and testing) from the HTF website or R package.
  • Review the class notes and HTF section 4.3.3.
  • Implement reduced-rank LDA using the vowel training data. Check your work by plotting the first two discriminant variables as in HTF Figure 4.4. Hint: Center the 10 training predictors before implementing LDA. See built-in R function 'scale'. The singular value or Eigen decompositions may be computed using the built-in R functions 'svd' or 'eigen', respectively.
  • Use the vowel testing data to estimate the expected prediciton error (assuming zero-one loss), varying the number of canonical variables used for classification.
  • Plot the EPE as a function of the number of discriminant variables, and compare this with HTF Figure 4.10.
  • (Optional) Reproduce HTF Figure 4.11. Note: The reproduction need not be exact. However, the information content should be preserved.

Fri. 2/06

  • HTF exercises 5.1 and 5.4
  • Read HTF section 3.3.2 regarding forward and backward stepwise selection methods. Read the last few paragraphs of HTF section 3.4 under the header "Degrees-of-Freedom Formula for LAR and Lasso". Read through the "df-stepwise.html" and corresponding "df-stepwise.Rmd" file (linked below; and attached to the wiki page), which illustrate how the degrees-of-freedom formula in 3.60 can be used in the context of stepwise regression.
  • Consider the ridge regression degrees-of-freedom formula given by expression 3.50. Describe the key assumption that must be made in order for expression 3.50 to be equivalent to the generalized degrees-of-freedom formula given by expression 3.60?
  • Extend the analysis in the "df-stepwise.Rmd" file to address the following question and write a brief summary of your findings: If the 'true' model is fixed, how does the largest possible model in a stepwise procedure affect the D.F.?

Links

RStudio/Knitr

Topic attachments
I Attachment Action Size Date Who Comment
2016-midterm-phoneme.RR 2016-midterm-phoneme.R manage 3.7 K 25 Mar 2016 - 08:33 MattShotwell Code for solution to 2016 midterm.
HW10.pdfpdf HW10.pdf manage 44.3 K 09 Mar 2015 - 09:51 MattShotwell Homework 10
LAozone.RR LAozone.R manage 3.1 K 14 Mar 2018 - 10:57 MattShotwell  
LagrangeMultipliers-Bishop-PatternRecognitionMachineLearning.pdfpdf LagrangeMultipliers-Bishop-PatternRecognitionMachineLearning.pdf manage 1574.4 K 06 Apr 2016 - 17:53 MattShotwell Lagrange Multipliers; Bishop; Pattern Recognition and Machine Learning
MCB-20121115.pdfpdf MCB-20121115.pdf manage 676.5 K 17 Dec 2014 - 10:32 MattShotwell The Matrix Cookbook (version 15 November 2012)
airquality-EM-mixture.RR airquality-EM-mixture.R manage 2.2 K 11 Apr 2016 - 10:59 MattShotwell EM algorithm with finite normal mixture
airquality-agnes.RR airquality-agnes.R manage 1.3 K 13 Apr 2016 - 11:22 MattShotwell [Ag]glomerative [nes]ting (clustering) with airquality data
boosting-trees.RR boosting-trees.R manage 2.0 K 22 Mar 2017 - 11:56 MattShotwell Boosting a tree stump with the AdaBoost.M1 algorithm
bootstrap-calibration.RR bootstrap-calibration.R manage 3.2 K 23 Feb 2018 - 11:43 MattShotwell  
df-stepwise.RDataRData df-stepwise.RData manage 2.5 K 17 Feb 2016 - 16:43 MattShotwell  
df-stepwise.RmdRmd df-stepwise.Rmd manage 5.5 K 12 Feb 2017 - 20:50 MattShotwell  
df-stepwise.htmlhtml df-stepwise.html manage 737.8 K 12 Feb 2017 - 20:50 MattShotwell  
lab1.pdfpdf lab1.pdf manage 226.6 K 12 Jan 2015 - 14:04 GuanhuaChen BIOS362_lab1
lab2.pdfpdf lab2.pdf manage 1901.4 K 21 Jan 2015 - 11:20 GuanhuaChen slides from Dr. Jojic (UNC)'s Machine learning class
lecture-1.pdfpdf lecture-1.pdf manage 406.9 K 10 Jan 2018 - 12:09 MattShotwell  
lecture-10.RR lecture-10.R manage 3.6 K 05 Feb 2018 - 09:11 MattShotwell  
lecture-10.pdfpdf lecture-10.pdf manage 146.8 K 01 Feb 2019 - 11:00 MattShotwell  
lecture-11.pdfpdf lecture-11.pdf manage 270.5 K 30 Jan 2019 - 10:59 MattShotwell  
lecture-12.pdfpdf lecture-12.pdf manage 424.5 K 06 Feb 2019 - 10:58 MattShotwell  
lecture-13.pdfpdf lecture-13.pdf manage 376.9 K 12 Feb 2018 - 10:31 MattShotwell  
lecture-14.pdfpdf lecture-14.pdf manage 269.7 K 15 Feb 2019 - 11:02 MattShotwell  
lecture-15.pdfpdf lecture-15.pdf manage 302.3 K 21 Feb 2018 - 10:58 MattShotwell  
lecture-16.pdfpdf lecture-16.pdf manage 197.6 K 22 Feb 2019 - 10:44 MattShotwell  
lecture-17.pdfpdf lecture-17.pdf manage 372.7 K 20 Feb 2019 - 11:05 MattShotwell  
lecture-18.pdfpdf lecture-18.pdf manage 190.7 K 28 Feb 2018 - 10:43 MattShotwell  
lecture-2.pdfpdf lecture-2.pdf manage 138.5 K 10 Jan 2018 - 12:10 MattShotwell  
lecture-20.pdfpdf lecture-20.pdf manage 143.4 K 14 Mar 2018 - 10:57 MattShotwell  
lecture-21.pdfpdf lecture-21.pdf manage 468.0 K 16 Mar 2018 - 10:06 MattShotwell  
lecture-22.pdfpdf lecture-22.pdf manage 276.1 K 19 Mar 2018 - 08:59 MattShotwell  
lecture-23.pdfpdf lecture-23.pdf manage 292.7 K 18 Mar 2019 - 09:24 MattShotwell  
lecture-24.pdfpdf lecture-24.pdf manage 502.2 K 20 Mar 2019 - 10:51 MattShotwell  
lecture-25.pdfpdf lecture-25.pdf manage 396.5 K 25 Mar 2019 - 10:46 MattShotwell  
lecture-26.pdfpdf lecture-26.pdf manage 569.1 K 27 Mar 2019 - 11:01 MattShotwell  
lecture-27.pdfpdf lecture-27.pdf manage 190.7 K 29 Mar 2019 - 10:59 MattShotwell  
lecture-28.pdfpdf lecture-28.pdf manage 287.3 K 10 Apr 2019 - 11:08 MattShotwell  
lecture-29.pdfpdf lecture-29.pdf manage 1018.4 K 15 Apr 2019 - 10:44 MattShotwell  
lecture-3.pdfpdf lecture-3.pdf manage 564.7 K 18 Jan 2018 - 08:46 MattShotwell  
lecture-30.pdfpdf lecture-30.pdf manage 617.6 K 12 Apr 2019 - 10:42 MattShotwell  
lecture-31.pdfpdf lecture-31.pdf manage 3491.6 K 03 Apr 2019 - 09:28 MattShotwell  
lecture-32.pdfpdf lecture-32.pdf manage 193.9 K 17 Apr 2019 - 10:51 MattShotwell  
lecture-4.pdfpdf lecture-4.pdf manage 175.7 K 14 Jan 2019 - 10:58 MattShotwell  
lecture-5.pdfpdf lecture-5.pdf manage 610.0 K 16 Jan 2019 - 10:48 MattShotwell  
lecture-6.pdfpdf lecture-6.pdf manage 259.8 K 18 Jan 2019 - 10:02 MattShotwell  
lecture-7.pdfpdf lecture-7.pdf manage 136.6 K 23 Jan 2019 - 11:18 MattShotwell  
lecture-8.pdfpdf lecture-8.pdf manage 586.9 K 29 Jan 2018 - 10:55 MattShotwell  
lecture-9.pdfpdf lecture-9.pdf manage 1063.9 K 31 Jan 2018 - 10:40 MattShotwell  
mixture-data-complete.RR mixture-data-complete.R manage 5.7 K 10 Feb 2015 - 09:12 MattShotwell splines regression, local regression, and kernel density classification of the mixture data
mixture-data-knn-local.RR mixture-data-knn-local.R manage 4.7 K 17 Jan 2018 - 10:25 MattShotwell  
mixture-data-svm.RR mixture-data-svm.R manage 3.3 K 07 Apr 2017 - 12:31 MattShotwell SVM with mixture data; 3D graphic
mixture-data.RR mixture-data.R manage 2.1 K 30 Jan 2015 - 09:31 MattShotwell Lab 3; demo code for mixture data
mnist-convnet.RR mnist-convnet.R manage 2.6 K 08 Apr 2019 - 11:09 MattShotwell  
nlls_v2.RR nlls_v2.R manage 3.2 K 19 Jan 2018 - 10:47 MattShotwell  
nnet.RR nnet.R manage 3.9 K 01 Apr 2019 - 10:52 MattShotwell  
nonlinear-bagging.RmdRmd nonlinear-bagging.Rmd manage 1.6 K 29 Feb 2016 - 11:08 MattShotwell nonlinear bagging example
nonlinear-bagging.csvcsv nonlinear-bagging.csv manage 0.5 K 29 Feb 2016 - 11:09 MattShotwell nonlinear bagging example data
pca-and-g-inverses.htmlhtml pca-and-g-inverses.html manage 924.1 K 23 Jan 2019 - 10:43 MattShotwell  
presentation.pdfpdf presentation.pdf manage 333.3 K 11 Mar 2019 - 10:41 MattShotwell  
principal-curves.RR principal-curves.R manage 2.0 K 19 Apr 2017 - 12:06 MattShotwell Principal curves example
prostate.RR prostate.R manage 2.9 K 26 Jan 2015 - 09:45 MattShotwell least-squares, ridge, and principal components regression with prostate data.
simple-neural-network.RR simple-neural-network.R manage 3.8 K 28 Mar 2017 - 09:43 MattShotwell Neural network with one hidden layer, 20 units, fully connected
splines-example.RR splines-example.R manage 3.9 K 07 Feb 2018 - 10:53 MattShotwell Splines example
yuying_1.pdfpdf yuying_1.pdf manage 953.4 K 13 Feb 2015 - 15:30 GuanhuaChen  
yuying_2.pdfpdf yuying_2.pdf manage 2283.6 K 13 Feb 2015 - 15:31 GuanhuaChen  
Topic revision: r260 - 19 Apr 2019, MattShotwell
 

This site is powered by FoswikiCopyright © 2013-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Vanderbilt Biostatistics Wiki? Send feedback