Before calculating binary ROC AUC, check that larger scores mean stronger evidence for the class your labels identify as positive. A useful model can appear to rank cases backwards if you pair event labels with the probability of the opposite class. Define the event, align the score direction, and verify the calculation on a tiny example before using the result in a report.
Separate the business event from the library convention
Suppose a support model predicts whether a ticket is urgent. The event of interest is urgent, but a fitted classifier might store its two classes in an order you did not expect. The official roc_auc_score documentation states that binary probability scores correspond to the class with the greater label, identified by estimator.classes_[1]. A hard-coded probability column is therefore not a substitute for checking the fitted class mapping.
There are two consistent approaches. Keep the original binary labels and supply scores for the greater label, or explicitly encode your chosen event as one and supply scores for that event. Do not combine the labels from one approach with the scores from the other. The second approach makes a business-defined event especially easy to audit.
Run an event-direction check with invented probabilities
This example uses hypothetical values, not measurements from a trained model. It requires Python 3 and scikit-learn in your project environment. Save it as auc_direction.py and run it with that environment’s Python interpreter. No files or network services are modified.
from sklearn.metrics import roc_auc_score
labels = ["routine", "urgent", "routine", "urgent"]
event = "urgent"
y_event = [int(label == event) for label in labels]
p_event = [0.1, 0.8, 0.3, 0.9]
p_other = [1.0 - value for value in p_event]
if set(y_event) != {0, 1}:
raise ValueError("Evaluation needs event and non-event cases")
aligned = roc_auc_score(y_event, p_event)
inverted = roc_auc_score(y_event, p_other)
constant = roc_auc_score(y_event, [0.5] * len(y_event))
assert aligned == 1.0
assert inverted == 0.0
assert constant == 0.5
print(f"event scores: {aligned:.1f}")
print(f"opposite scores: {inverted:.1f}")
print(f"constant scores: {constant:.1f}")
The urgent cases both score above the routine cases, producing perfect ranking in this deliberately simple example. Complementing the probabilities while keeping the event labels fixed reverses every relevant ordering. Constant scores tie all cases and produce 0.5. These checks discriminate between correct direction, reversed direction and no ranking information; they do not establish useful performance on real tickets.
Connect the check to a fitted binary classifier
Inspect the fitted estimator’s classes_ and the shape of its probability output. Identify the column for the exact event label, then encode the corresponding evaluation labels as zero and one. Keep the evaluation rows in the same order as the rows passed to the model. A correct column cannot repair shuffled labels or predictions from a different sample.
Check that the evaluation data contains both outcomes. A one-class slice cannot support an ordinary binary ROC comparison; do not replace an undefined result with zero or silently merge it into an average. For split diagnostics, see checking class counts after a stratified train-test split. Also avoid feeding predicted class labels where continuous scores are available: thresholding removes ranking information.
Investigate a surprisingly low AUC without gaming the test
First inspect the event definition and column mapping, then row alignment and data preparation. Record any implementation correction so the report remains reproducible. A low result is not permission to reverse scores simply because doing so improves the held-out result. Choosing a direction after inspecting final-test performance is another form of model selection; validate that decision on appropriate development data before an untouched evaluation.
A high AUC describes discrimination, not calibrated probabilities, a suitable alert threshold or affordable false-positive volume. Those questions need additional checks tied to the intended workflow. Keep the reported metric name, event label, evaluation population and scoring method together.
Quick review checklist and FAQ
- Is the event named explicitly, with matching zero-one labels?
- Do larger scores mean more evidence for that same event?
- Are both outcomes present and all rows aligned?
- Does a toy ranking produce the expected result?
Must the event be the second original class? No. You can explicitly recode your chosen event as one and use its score consistently. Does AUC choose the deployment threshold? No. Threshold choice requires operational costs, workload and validation beyond this direction check.
Safety: This tutorial is for educational purposes. Test carefully before production use. The site owner is not responsible for misuse, damage, data loss or production issues.
