10 Most Common Regression and Classification Loss Functions

Loss functions are a key component of ML algorithms.

They specify the objective an algorithm should aim to optimize during its training. In other words, loss functions tell the algorithm what it should minimize or maximize to improve its performance.

Therefore, being aware of the most common loss functions is extremely crucial.

The visual below depicts the most commonly used loss functions for regression and classification tasks.

Regression

Mean Bias Error

1. Captures the average bias in the prediction.
2. However, it is rarely used in training ML models.
3. This is because negative errors may cancel positive errors, leading to zero loss, and consequently, no weight updates.
4. Mean bias error is foundational to the more advanced regression losses discussed below.

Mean Absolute Error (or L1 loss)

1. Measures the average absolute difference between predicted and actual value.
2. Positive errors and negative errors don’t cancel out.
3. One caveat is that small errors are as important as big ones. Thus, the magnitude of the gradient is independent of error size.

Mean Squared Error (or L2 loss)

1. It measures the squared difference between predicted and actual value.
2. Larger errors contribute more significantly than smaller errors.
3. The above point may also be a caveat as it is sensitive to outliers.
4. Yet, it is among the most common loss functions for many regression models.

Root Mean Squared Error

1. Mean Squared Error with a square root.
2. Loss and the dependent variable (y) have the same units.

Huber Loss

1. It is a combination of mean absolute error and mean squared error.
2. For smaller errors, mean squared error is used, which is differentiable through (unlike MAE, which is non-differentiable at x=0).
3. For large errors, mean absolute error is used, which is less sensitive to outliers.
4. One caveat is that it is parameterized — adding another hyperparameter to the list.

Log Cosh Loss

1. For small errors, log cosh loss is approximately → x²/2 — quadratic.
2. For large errors, log cash loss is approximately → |x| - log(2) — linear.
3. Thus, it is very similar to Huber loss.
4. Also, it is non-parametric.
5. The only caveat is that it is a bit computationally expensive.

Classification

Log Cosh Loss

1. A loss function used for binary classification tasks.
2. Measures the dissimilarity between predicted probabilities and true binary labels, through the logarithmic loss.
3. Where did the log loss originate from? We discussed it here: why Do We Use log-loss To Train Logistic Regression?

Hinge Loss

1. Penalizes both wrong and right (but less confident) predictions).
2. It is based on the concept of margin, which represents the distance between a data point and the decision boundary.
3. The larger the margin, the more confident the classifier is about its prediction.
4. Particularly used to train support vector machines (SVMs).

Cross-Entropy Loss

1. An extension of Binary Cross Entropy loss to multi-class classification tasks.

KL Divergence

1. Measure information lost when one distribution is approximated using another distribution.
2. The more information is lost, the more the KL Divergence.
3. For classification, however, using KL divergence is the same as minimizing cross entropy (proved below):

4. Thus, it is recommended to use cross-entropy loss directly.
5. That said, KL divergence is widely used in many other algorithms:
   1. As a loss function in the t-SNE algorithm. We discussed it here: t-SNE article.
   2. For model compression using knowledge distillation. We discussed it here: Model compression article.

Thanks for reading!