What Is a Loss Function?

Motivation

A neural network learns by adjusting its parameters $\theta$ so that its predictions $f_\theta(x)$ are close to the ground-truth targets $y$. A loss function (also called a criterion or objective) gives this notion of "closeness" a precise scalar number that the optimizer minimizes.

Without a loss, gradient descent has no direction to follow.

The Training Objective

Given a dataset of $N$ input-target pairs $\{(x_i, y_i)\}_{i=1}^N$, the empirical training loss is

$$\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \ell\bigl(f_\theta(x_i),\, y_i\bigr)$$

where $\ell$ is the per-sample loss specific to the task. The optimizer updates parameters via

$$\theta \leftarrow \theta - \alpha \, \nabla_\theta \mathcal{L}(\theta)$$

where $\alpha$ is the learning rate. PyTorch computes $\nabla_\theta \mathcal{L}$ automatically via loss.backward().

PyTorch Convention

In PyTorch, a loss function is an nn.Module that takes two tensors — input (model prediction) and target (ground truth) — and returns a scalar:

import torch
import torch.nn as nn

criterion = nn.MSELoss()
input  = torch.randn(4, 1)   # model output
target = torch.randn(4, 1)   # ground truth
loss   = criterion(input, target)  # scalar
loss.backward()              # populate .grad for every parameter

The names input and target follow PyTorch's own documentation. In context:

• input — the raw output of the last layer (logits or probabilities, depending on the loss)
• target — the label or value you want the model to predict

Reduction Modes

Every PyTorch loss accepts a reduction keyword that controls whether the per-sample losses $\ell_i$ are aggregated:

criterion_mean = nn.L1Loss(reduction='mean')
criterion_none = nn.L1Loss(reduction='none')

l_mean = criterion_mean(input, target)      # scalar
l_each = criterion_none(input, target)      # shape (4, 1)

The Computational Graph

PyTorch builds a dynamic computation graph during the forward pass. The loss node sits at the top; calling .backward() propagates gradients back through every operation to the leaf parameters.

  x  ──► layer 1 ──► layer 2 ──► output
                                     │
                         target ──► loss ──► .backward()
                                               │
                                       ∇θ flows back

This is why the loss must be a scalar (or you must specify a gradient vector if using backward(gradient=...)). Most built-in losses produce a scalar when reduction != 'none'.

Choosing the Right Loss

The right loss encodes your statistical assumption about the target:

The remaining readings in this supplement cover each family in detail.