Keep Confusion Matrix Labels Fixed When a Test Class Is Missing

If a scikit-learn confusion matrix changes shape between evaluation batches, inspect the labels before changing your model. By default, the function uses the sorted labels present in either the true or predicted values. A batch containing only one class can therefore produce a smaller matrix. Pass an explicit, complete class order to keep reports comparable, then separately check whether the batch contains examples of every class.

This tutorial is for educational purposes. Test examples in an isolated environment before production use. You remain responsible for safe implementation; the site owner is not responsible for misuse, damage, data loss or production issues. The example uses invented labels and does not evaluate a real deployed model.

Understand why the default shape changes

The official API defines rows as true classes and columns as predicted classes. It also states that the optional labels argument controls label selection and ordering. When omitted, labels occurring in either input are used in sorted order.

Suppose a monitoring batch contains only routine events and your model predicts routine for all of them. The default matrix can contain only the routine class. If a later batch includes alert events, the default matrix includes another row and column. Comparing the first flattened value from each result without its class labels is therefore unsafe.

Explicit labels stabilise representation, not evaluation quality. They cannot create missing examples or demonstrate that the model recognises a class absent from the test batch.

Run a small fixed-label example

Use an existing Python environment with scikit-learn installed. Save this code as a separate practice script rather than modifying a production evaluation job. Both sequences describe the same three fictional observations in matching order.

from sklearn.metrics import confusion_matrix

class_order = ["routine", "alert"]
y_true = ["routine", "routine", "routine"]
y_pred = ["routine", "routine", "routine"]

observed = set(y_true) | set(y_pred)
unknown = observed - set(class_order)
if unknown:
    raise ValueError(f"Unexpected labels: {sorted(unknown)}")

counts = confusion_matrix(y_true, y_pred, labels=class_order)
print("Rows: true; columns: predicted")
print("Class order:", class_order)
print(counts)
print("True class counts:", counts.sum(axis=1))

The first row records routine observations; the second records alert observations. Inspect the printed matrix and row totals together. A zero total for the alert row means this batch has no true alert examples. It does not mean the model has achieved perfect alert detection.

The explicit unexpected-label check is important because the API permits selecting a subset of labels. An accidental new spelling should trigger investigation rather than silently become an excluded category. If your targets are numeric, use numeric labels consistently instead of mixing strings and numbers.

Keep counts alongside normalised views

The API supports normalising over true rows with normalize="true", predicted columns with normalize="pred", or the whole population with normalize="all". These views answer different questions. Row normalisation describes how observations within each true class were predicted. Column normalisation describes the true-class composition of each predicted class.

For a row-normalised companion report, add the following after computing the counts:

row_view = confusion_matrix(
    y_true, y_pred, labels=class_order, normalize="true"
)
print("Row-normalised view:")
print(row_view)

Do not interpret an unsupported class row as a measured success rate, regardless of the numeric display used for that row. Show the raw support beside it and mark the class as not evaluated for that batch. An all-population percentage can conceal a rare class, so it should not replace class-level counts.

Make report comparisons reproducible

Store the class order with each report and use it for chart tick labels.
Record the evaluation window and number of observations.
Check that truth and prediction arrays describe identical records in identical order.
Preserve raw counts before adding normalised displays.
Flag unexpected labels and missing true-class support.
Document whether sample weights were used before comparing totals.

If you rename or merge classes, version the reporting scheme instead of placing incompatible matrices side by side. A stable shape is useful only when each position continues to mean the same thing.

For an additional evaluation check, read how to compare classifier accuracy with a DummyClassifier baseline. A fixed-label matrix adds detail about error types, while a baseline helps assess whether a headline score is useful.

Common questions

Can I always unpack four values as true negatives and positives? Only when you have established a binary matrix with the intended negative-then-positive label order. Multiclass matrices and smaller default matrices do not satisfy that assumption.

Should I add artificial alert examples to make the report complete? Not to disguise missing evaluation evidence. Investigate the sampling process and collect an appropriate evaluation set; keep fabricated practice data separate from measured results.

How do I roll back this reporting change? Keep the previous script under version control and compare both reports on the same saved inputs. This example changes only local reporting and does not retrain or deploy a model.