A classifier with impressive accuracy may simply repeat the most common label. Before tuning a more complex model, compare it with DummyClassifier on exactly the same held-out examples. This establishes a simple reference: does using your features actually improve the evaluation you care about?
This tutorial concerns ordinary single-label classification. It does not establish that any particular model is safe for production, and a useful baseline is only one part of validation.
Understand what the baseline predicts
The official DummyClassifier documentation explains that its predictions ignore input feature values. With strategy="most_frequent", it learns the most common training label and predicts that label for every evaluation row. This is deliberately unambitious: a complex classifier should justify its extra operational cost.
The default prior strategy gives the same predicted labels, but its probability estimates reflect the training class distribution. By contrast, most_frequent assigns all probability to the majority class. That difference matters when you evaluate probabilities rather than just labels. Select the strategy explicitly so a colleague can understand your reference without relying on defaults.
Use a split that matches the real prediction task
Fit both models on training data and score both on the same untouched evaluation data. Do not fit the dummy classifier on the evaluation labels: that would let your reference learn the evaluation distribution.
The example below uses a small built-in dataset for practice and a stratified random split. Stratification helps retain class representation; it does not solve dependencies between rows. If repeated customers appear in your dataset, follow the customer-group splitting workflow instead. For forecasting later events, consider a chronological evaluation rather than randomly mixing past and future.
Run a side-by-side comparison
Use an isolated environment containing scikit-learn. The code reads a bundled teaching dataset and makes no external requests or changes to production records. Run it as a standalone Python script:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import (accuracy_score,
balanced_accuracy_score,
confusion_matrix)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42
)
models = {
"majority baseline": DummyClassifier(
strategy="most_frequent"
),
"small tree": DecisionTreeClassifier(
max_depth=3, random_state=42
),
}
for name, model in models.items():
model.fit(X_train, y_train)
predicted = model.predict(X_test)
print(name)
print("accuracy:", accuracy_score(y_test, predicted))
print("balanced accuracy:",
balanced_accuracy_score(y_test, predicted))
print("confusion matrix:")
print(confusion_matrix(y_test, predicted, labels=[0, 1]))The medical dataset is a software-learning example, not a clinical recommendation or diagnostic tool. The script prints metrics from your installed environment; no particular numerical result is promised here. Keep its random seed and split unchanged while comparing these two models.
Read more than the accuracy column
Accuracy counts correct labels across all evaluation rows. When one class dominates, always choosing it can look successful while missing every example of another class. Balanced accuracy averages recall across classes, making that omission harder to overlook. A confusion matrix shows the actual error pattern; with the explicit label order above, rows are true labels and columns are predicted labels.
Decide which class represents the operational event before interpreting errors. A label numbered zero is not inherently less important than one. In a support-ticket example, missed urgent cases and unnecessary escalations have different consequences; a single aggregate score cannot choose that trade-off for you.
Troubleshooting and acceptance checklist
- If the tree barely beats the baseline, inspect label quality, feature availability and split design before increasing complexity.
- If scores are unexpectedly perfect, check whether a feature reveals the answer or whether duplicate entities cross the split.
- If a class is absent from evaluation, redesign the evaluation sample rather than trusting the summary metric.
- Record dataset version, training labels, split rules, package version and both models’ metrics together.
Does beating the baseline prove deployment readiness? No. You still need suitable validation, error review, monitoring and an operational decision threshold where relevant. Repeatedly tuning against one held-out set also makes it less independent; preserve a final evaluation for the selected approach.
Safety: This tutorial is for educational purposes. Test carefully before production use. The site owner is not responsible for misuse, damage, data loss or production issues.
