---
title: "Maximum likelihood estimation and resampling techniques"
output:
  pdf_document: default
---


# Maximum likelihood estimation in linear regression

1. Write up the simple linear regression model with one explanatory variable. 

2. Write up an expression for the log-likelihood for `n` independent observations from the simple linear regression model.

3. Show that the maximum likelihood estimators (MLE) for the intercept $\alpha$ and slope $\beta$ are indeed the least squares estimators.
   
 Consider a simple linear regression model for the `trees` dataset 
   with `Volume` as response variable and `Girth` as explanatory variable.
   
4.  Find the MLE estimates for $\alpha$, $\beta$ and $\sigma$ numerically by maximizing the log-likelihood function.
   
5. Find the least squares estimates of $\alpha$ and $\beta$ numerically. That is, define a function that computes the least squares as a function of $\alpha$ and $\beta$ and use `optim()` to minimize the function. Compare with 4.

6. Compare 4. and 5. to the output of `summary(lm(...))`.

# Overfitting and cross validation


 We consider the `Credit` dataset from the `ISLR` package which contains data about credit card holders.
 
```{r, warning=FALSE}
library(ISLR) # Remember that this package must be installed
head(Credit)
```
Our response variable will be `Balance` which is the constumers credit card debt. As predictor we use the variable `Rating` which is the costumer's credit rating.

1. Fit a linear regression model for the relationship between `balance` and `Rating`. 

2. Use bootstrap 
   to estimate the standard errors of the parameter estimates in the 
   simple linear regression model. Compare to those obtained from `summary()`.
   
3. Use resampling of residuals to estimate the standard errors of the parameter estimates in the 
   simple linear regression model. Compare to those obtained from `summary()`.   
   
4. Use cross validation to decide between linear and polynomial regression.