## Context

We'll be working with a dataset containing physical measurements on 80 adult males. These measurements include body fat percentage estimates as well as body circumference measurements.

- `fatBrozek`: Percent body fat using Brozek's equation: 457/Density - 414.2
- `fatSiri`: Percent body fat using Siri's equation: 495/Density - 450
- `density`: Density determined from underwater weighing (gm/cm^3).
- `age`: Age (years)
- `weight`: Weight (lbs)
- `height`: Height (inches)
- `neck`: Neck circumference (cm)
- `chest`: Chest circumference (cm)
- `abdomen`: Abdomen circumference (cm)
- `hip`: Hip circumference (cm)
- `thigh`: Thigh circumference (cm)
- `knee`: Knee circumference (cm)
- `ankle`: Ankle circumference (cm)
- `biceps`: Biceps (extended) circumference (cm)
- `forearm`: Forearm circumference (cm)
- `wrist`: Wrist circumference (cm)

It takes a lot of effort to estimate body fat percentage accurately through underwater weighing. The goal is to build the best predictive model for `fatSiri` using just circumference measurements, which are more easily attainable. (We won't use `fatBrozek` or `density` as predictors because they're other outcome variables.)

```{r}
library(readr)
library(ggplot2)
library(dplyr)
library(broom)
library(caret)
bodyfat_train <- read_csv("https://www.dropbox.com/s/js2gxnazybokbzh/bodyfat_train.csv?dl=1")

# Remove the fatBrozek and density variables
bodyfat_train <- bodyfat_train %>%
    select(-fatBrozek, -density)
```

## Exercise 1: 4 models

Consider the 4 models below:

```{r}
mod1 <- lm(fatSiri ~ age+weight+neck+abdomen+thigh+forearm, data = bodyfat_train)
mod2 <- lm(fatSiri ~ age+weight+neck+abdomen+thigh+forearm+biceps, data = bodyfat_train)
mod3 <- lm(fatSiri ~ age+weight+neck+abdomen+thigh+forearm+biceps+chest+hip, data = bodyfat_train)
mod4 <- lm(fatSiri ~ ., data = bodyfat_train) # The . means all predictors
```

a. Which model will have the lowest training RMSE, and why?
b. Compute the RMSE for models 1 and 4 to (partially) check your answer for part a.
c. Which model do you think will perform worst on new test data? Why?



## Exercise 2: Cross-validation with `caret`

a. Complete the code below to perform 10-fold cross-validation for `mod1` to estimate the test RMSE ($\text{CV}_{(10)}$). Do we need to use `set.seed()`? Why or why not? (Is there a number of folds for which we would not need to set the seed?)    

```{r}
# Do we need to use set.seed()?

mod1_cv <- train(
    
)
```

b. **STAT 155 review:** Look at the `summary()` of `mod1_cv`. Contextually interpret the coefficient for the weight predictor. Is anything surprising? Why might this be?
c. Look at `mod1_cv$resample`, and use this to calculate the 10-fold cross-validated RMSE by hand (the idea is the same as when using MSE). (Note: We haven't done this together, but how can you adapt code that we've used before?)
d. Check your answer to part c by directly printing out the CV metrics: `mod1_cv$results`. Interpret this metric.



## Exercise 3: Looking at the evaluation metrics

Look at the completed table below of evaluation metrics for the 4 models.

a. Which model performed the best on the training data?
b. Which model performed best on the test set?
c. Explain why there's a discrepancy between these 2 answers and why CV, in general, can help prevent overfitting.


Model     Training RMSE   $\text{CV}_{(10)}$
-------- --------------- --------------------
`mod1`       3.810712         4.389568
`mod2`       3.766645         4.438637
`mod3`       3.752362         4.517281
`mod4`       3.572299         4.543343




## Exercise 4: Practical issues: choosing $k$

a. In terms of sample size, what are the pros/cons of low vs. high $k$?
b. In terms of computational time, what are the pros/cons of low vs. high $k$?
c. If possible, it is advisable to choose $k$ to be a divisor of the sample size. Why do you think that is?



## Digging deeper

If you have time, consider these exercises to further explore concepts related to todays' ideas.

1. `caret`'s `trainControl()` function also has a `"repeatedcv"` method. Just from the name, how do you think this method differs from `"cv"`? What are the pros/cons of `"repeatedcv"` as compared to `"cv"`?

2. Adapt the `train()` code to perform leave-one-out-cross-validation (LOOCV).
    - Hint: `nrow(dataset)` obtains the number of cases in the dataset.
    - Do we need `set.seed()`? Why or why not?
    - Using the information from `your_output$resample` (which is a dataset), construct a visualization to examine the variability of RMSE from case to case. What might explain any very large values? What does this highlight about the quality of estimation of the LOOCV process?



