A multilabel classifier can make only a small fraction of individual label decisions incorrectly while getting no sample’s complete label set right. That is not necessarily a scikit-learn bug: hamming_loss counts incorrect label decisions, whereas accuracy_score on a multilabel indicator matrix requires an exact match across each whole row. Check both denominators before interpreting a dashboard.
Identify what one row and one column mean
Imagine a helpdesk ticket that can carry several tags simultaneously, such as network, account and hardware. Each row represents one ticket. Each column represents one fixed tag, with one meaning present and zero meaning absent. This differs from multiclass classification, where a ticket receives exactly one class from a list.
Before scoring, confirm that truth and prediction matrices use the same ticket order and the same tag order. Matching dimensions alone do not establish alignment. If one exporter rearranges tag columns, a metric can calculate successfully while measuring the wrong comparisons. Keep the column names alongside the arrays and compare them explicitly.
Run a fixture where every row has one mistake
In an isolated Python environment with NumPy and scikit-learn installed, run this self-contained example. It does not train a model or modify data. The predictions are deliberately imperfect so the two metrics separate clearly.
import numpy as np
from sklearn.metrics import accuracy_score, hamming_loss
y_true = np.array([
[1, 0, 1, 0],
[0, 1, 0, 1],
[1, 1, 0, 0],
])
y_pred = np.array([
[1, 0, 1, 1],
[0, 1, 1, 1],
[1, 0, 0, 0],
])
assert y_true.shape == y_pred.shape
wrong_cells = int(np.count_nonzero(y_true != y_pred))
exact_rows = int(np.count_nonzero(np.all(y_true == y_pred, axis=1)))
loss = hamming_loss(y_true, y_pred)
subset = accuracy_score(y_true, y_pred)
print(f'wrong cells: {wrong_cells}/{y_true.size}')
print(f'exact rows: {exact_rows}/{len(y_true)}')
print(f'hamming loss: {loss:.2f}')
print(f'subset accuracy: {subset:.2f}')
assert wrong_cells == 3
assert exact_rows == 0
assert loss == 0.25
assert subset == 0.0
assert accuracy_score(y_true, y_true) == 1.0
assert hamming_loss(y_true, y_true) == 0.0
The expected printed lines are wrong cells: 3/12, exact rows: 0/3, hamming loss: 0.25 and subset accuracy: 0.00. There are three incorrect cells among twelve decisions. However, each of the three rows contains a mistake, so none qualifies as an exact match.
Choose the metric that matches the reader’s task
If a workflow only succeeds when every required tag is correct, subset accuracy reflects that strict sample-level requirement. If operators can correct individual tags independently, Hamming loss describes the fraction of label decisions needing correction. Neither metric alone measures the business cost of missing a critical tag.
Do not label 1 - hamming_loss simply as accuracy beside a separate accuracy_score result. If reporting that complement, name it label-wise agreement and explain the denominator. In this unweighted fixture it is the proportion of matching cells, not the proportion of entirely correct tickets.
Also examine label prevalence. When most entries are zero, an always-absent predictor can agree with many cells while failing to find positive labels. Review precision and recall for important tags instead of declaring the system useful from low Hamming loss alone. The article on undefined precision for a never-predicted class explains why missing positive predictions deserve investigation.
Troubleshoot disagreements without changing the target
- Check whether inputs are binary indicator matrices rather than probability scores. Choose thresholds on validation data before evaluating the held-out test set.
- Inspect several mismatching rows with human-readable tag names. Distinguish extra tags from missing tags.
- Keep sample weights consistent between metrics. The simple cell-count arithmetic above assumes no sample weights.
- Record the label vocabulary and threshold policy with the report so later runs remain comparable.
Quick questions before publishing a score
Is lower Hamming loss better? Yes: zero means no incorrect label decisions. Does zero subset accuracy mean every label is wrong? No: it means no complete sample-level label set matches exactly. Should I flatten the matrices? Not to reproduce subset accuracy: flattening changes the question from whole-row correctness to individual decisions.
Sources: the official Hamming loss documentation describes its label-level penalty and distinction from subset zero-one loss; the accuracy documentation specifies exact-set matching for multilabel inputs.
This tutorial is for educational purposes. Test carefully before using changes in production. The site owner is not responsible for misuse, damage, data loss or production issues.
