YouTip

## PyTorch torch.nn.BCEWithLogitsLoss `torch.nn.BCEWithLogitsLoss` is a loss function in PyTorch that measures the Binary Cross Entropy (BCE) between target and input probabilities. This loss combines a **Sigmoid layer** and the **BCELoss** into a single class. By combining these operations, it leverages the log-sum-exp trick for numerical stability, making it much more stable than using a plain `Sigmoid` followed by `BCELoss`. It is widely used for binary classification and multi-label classification tasks. --- ## Syntax and Parameters ### Function Definition ```python torch.nn.BCEWithLogitsLoss(weight=None, reduction='mean', pos_weight=None) ``` ### Parameters * **`weight`** (*Tensor, optional*): A manual rescaling weight given to the loss of each batch element. If provided, it must match the shape of the input. * **`reduction`** (*str, optional*): Specifies the reduction to apply to the output: * `'none'`: No reduction will be applied; returns a loss tensor of the same shape as the input. * `'mean'`: The sum of the output will be divided by the number of elements in the output (default). * `'sum'`: The output will be summed. * **`pos_weight`** (*Tensor, optional*): A weight of positive examples. Must be a vector with length equal to the number of classes. It is used to handle class imbalance by scaling the loss of positive classes. --- ## Code Examples ### Example 1: Basic Usage This example demonstrates how to compute the loss using raw, unnormalized logits and compares it with the manual combination of `torch.sigmoid` and `nn.BCELoss`. ```python import torch import torch.nn as nn # Initialize the loss function criterion = nn.BCEWithLogitsLoss() # Unnormalized logits (raw outputs from the model's last linear layer) logits = torch.tensor([2.0, -1.0, 0.5, -3.0]) # Binary targets (ground truth labels) targets = torch.tensor([1.0, 0.0, 1.0, 0.0]) # Calculate loss loss = criterion(logits, targets) print("BCEWithLogitsLoss:", loss.item()) # Manual verification: Sigmoid followed by BCELoss sigmoid_outputs = torch.sigmoid(logits) bce_criterion = nn.BCELoss() manual_loss = bce_criterion(sigmoid_outputs, targets) print("Manual BCE Loss: ", manual_loss.item()) ``` --- ### Example 2: Handling Class Imbalance with `pos_weight` When dealing with highly imbalanced datasets where positive samples are rare, you can use `pos_weight` to increase the importance of positive classes. ```python import torch import torch.nn as nn # Positive class weight: makes positive samples 5 times more important pos_weight = torch.tensor([5.0]) criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight) # Simulated model outputs (logits) and targets for 10 samples logits = torch.randn(10, 1) targets = torch.zeros(10, 1) targets[:2] = 1.0 # Only 2 out of 10 samples are positive (imbalanced) # Calculate weighted loss loss = criterion(logits, targets) print("Weighted BCE Loss:", loss.item()) ``` --- ### Example 3: Multi-Label Classification In multi-label classification, each sample can belong to multiple classes simultaneously. Each class is treated as an independent binary classification task. ```python import torch import torch.nn as nn # Initialize the loss function criterion = nn.BCEWithLogitsLoss() # Batch size = 4, Number of classes = 5 logits = torch.randn(4, 5) # Generate random binary labels (0 or 1) for each class labels = torch.randint(0, 2, (4, 5)).float() # Calculate multi-label loss loss = criterion(logits, labels) print("Multi-label Loss:", loss.item()) ``` --- ## Common Use Cases * **Binary Classification**: Predicting a single binary outcome (e.g., Spam vs. Not Spam). * **Multi-Label Classification**: Predicting multiple independent binary attributes for a single input (e.g., tagging an image with both "outdoor" and "sunny"). * **Imbalanced Datasets**: Using `pos_weight` to penalize false negatives more heavily than false positives. --- ## Important Considerations > ⚠️ **Warning: Do not apply Sigmoid to your model's output before passing it to this loss function.** > The input to `BCEWithLogitsLoss` must be raw, unnormalized logits. If you apply a `Sigmoid` activation function in your model's forward pass, you should use `nn.BCELoss` instead, though doing so is less numerically stable.