About
Loss functions define how to penalize incorrect predictions. The optimization problems associated with various linear classifiers are defined as minimizing the loss on training points (sometime along with a regularization term).
They can also be used to evaluate the quality of models.
Type
Regression
- Squared loss = <math>(y-\hat{y})^2</math>
Classification
0-1
0-1 loss: Penalty is 0 for correct prediction, and 1 otherwise
As 0-1 loss is not convex, the standard approach is to transform the categorical features into numerical features: (See Statistics - Dummy (Coding|Variable) - One-hot-encoding (OHE)) and to use a regression loss.
Log
Log loss is defined as: <MATH> \begin{align} \ell_{log}(p, y) = \begin{cases} -\log (p) & \text{if } y = 1\\\ -\log(1-p) & \text{if } y = 0 \end{cases} \end{align} </MATH> where
- <math>p</math> is a probability between 0 and 1. A base probability for a binary event will be just the mean over the training targetThen it can be compared to the output of probabilistic model such as the logistic regression
- <math>y</math> is a label of either 0 or 1.
Log loss is a standard evaluation criterion when predicting rare-events such as click-through rate prediction
Python
from math import log
def computeLogLoss(p, y):
"""Calculates the value of log loss for a given probabilty and label.
Args:
p (float): A probabilty between 0 and 1.
y (int): A label. Takes on the values 0 and 1.
Returns:
float: The log loss value.
"""
epsilon = 10e-12
if p==0:
p = epsilon
elif p==1:
p = p - epsilon
if y == 1:
return -log(p)
elif y == 0:
return -log(1-p)