# Statistics - Bias-variance trade-off (between overfitting and underfitting)

## 1 - About

The bias-variance trade-off is the point where we are adding just noise by adding model complexity (flexibility). The training error goes down as it has to, but the test error is starting to go up. The model after the bias trade-off begins to overfit.

When the nature of the problem is changing the trade-off is changing.

## 3 - Formula

The ingredients of prediction error are actually:

• bias: the bias is how far off on the average the model is from the truth.
• and variance. The variance is how much that the estimate varies around its average.

Bias and variance together gives us prediction error.

This difference can be expressed in term of variance and bias:

$e^2 = var(model) + var(chance) + bias$

where:

• $var(model)$

is the variance due to the training data set selected. (Reducible)

• $var(chance)$

is the variance due to chance (Not reducible)

• bias is the average of all $\hat{Y}$

over all training data set minus the true Y (Reducible)

As the flexibility (order in complexity) of f increases, its variance increases, and its bias decreases. So choosing the flexibility based on average test error amounts to a bias-variance trade-o ff.

## 4 - Illustration

where:

We want to find the model complexity that gives the smallest test error. When the nature of the problem is changing the trade-off is changing.

## 5 - Nature of the problem

When the nature of the problem is changing the trade-off is changing.

• the truth is wiggly and the noise is high, so the quadratic do the best

• the truth is smoother, so the linear model do really well

• the truth is wiggly and the noise is low, so the more flexible do the best