If an evaluation batch contains only one observed class, pass the full intended class universe through labels when calculating scikit-learn log loss. Keep the probability columns aligned with that universe. This resolves label inference; it does not make the batch representative of every class.
This situation can occur in a small daily monitoring window even when the trained classifier supports two classes. Do not invent an extra observation, remove the window from reporting without explanation, or replace probabilities with predicted labels just to silence an error.
Understand what is missing from the batch
Log loss scores the probability assigned to the observed true outcome. A correct outcome given high probability receives a smaller loss than the same outcome given low probability. The calculation uses natural logarithms. Its label universe must contain at least two classes, but an individual evaluation window need not contain examples of every one of those classes.
Without explicit labels, the function infers the universe from the supplied true labels. A batch containing only ham messages therefore fails to describe the intended ham-versus-spam problem. Supplying both legitimate class names fixes that metadata problem. It does not create evidence about spam detection in this window.
Run a small example with known probabilities
Use an environment with scikit-learn installed. This fixture supplies probabilities directly, so it neither trains a model nor reads customer data. The two probability columns below mean ham and spam, in that alphabetical order.
import math
from sklearn.metrics import log_loss
y_true = ["ham", "ham"]
probabilities = [[0.9, 0.1], [0.8, 0.2]]
try:
log_loss(y_true, probabilities)
except ValueError:
print("Implicit labels rejected the one-class batch")
else:
raise AssertionError("Expected explicit labels to be necessary")
mean_loss = log_loss(
y_true, probabilities, labels=["ham", "spam"]
)
summed_loss = log_loss(
y_true, probabilities, labels=["ham", "spam"], normalize=False
)
expected = (-math.log(0.9) - math.log(0.8)) / 2
assert math.isclose(mean_loss, expected)
assert math.isclose(summed_loss, 2 * mean_loss)
print(f"Mean log loss: {mean_loss:.6f}")
print(f"Summed log loss: {summed_loss:.6f}")
The mean rounds to 0.164252 and the sum to 0.328504. The calculation uses the ham probabilities because both observed outcomes are ham. The second column remains necessary to describe the binary probability distribution; it is not a second observed outcome.
Check column order before adding labels
The official log_loss reference specifies alphabetically ordered class columns, as used by LabelBinarizer. Inspect a fitted classifier’s classes_ alongside its predict_proba output. Do not assume that the second column means the business event you happen to call positive, especially when consuming exported probabilities from another system.
For this example, a column order of spam then ham would be wrong unless the data were explicitly reordered before scoring. Merely reversing the labels argument is not a reliable way to relabel incoming columns: construct the expected ordered columns intentionally. Validate row counts, finite values, probability bounds and row sums when integrating an external export.
A one-dimensional binary probability array has an additional convention: it represents the positive-class probability. A two-column fixture is easier to audit when teaching or debugging this boundary. Do not convert hard class predictions into apparent probabilities by changing their data type.
Report coverage alongside the score
Record the evaluation window, total observations, per-class counts and the declared class universe with each result. Here that means two ham observations and zero spam observations. A low loss on this batch does not demonstrate that the classifier catches spam, is calibrated overall, or remains suitable for deployment.
For adjacent label-reporting issues, see keeping confusion-matrix labels fixed when a test class is missing. Stable display labels help comparisons, but neither a confusion matrix nor an explicit log-loss label list repairs missing evaluation coverage.
Quick troubleshooting checklist
- Confirm that the intended task really has at least two legitimate classes.
- Check that every observed label belongs to the declared universe.
- Match probability rows to true-label rows without independently sorting either array.
- Document whether the reported value is the mean or sum; sums grow with batch size.
- If a class disappears repeatedly, investigate sampling and monitoring windows rather than hiding the condition.
Can I compare two windows of different sizes? Mean loss avoids the direct size scaling of summed loss, but class mix, sampling and weighting can still change the comparison. Report those differences rather than interpreting a smaller value as automatic model improvement.
Safety: This tutorial is for educational purposes. Test evaluation changes carefully before production use. The site owner is not responsible for misuse, damage, data loss or production issues. No production model or dataset is modified by this example.
