Mini AMLB Benchmark¶
AutoML Benchmark (AMLB) evaluates AutoML frameworks on OpenML task suites. This notebook runs a scaled-down version: one small OpenML classification task, three sklearn classifiers, then synthbench data at the same size with corruption layered on.
The point of the comparison is what you cannot compute on the real task. On iris, a classifier scoring 0.95 might be leaving 0.05 on the table or might be at the ceiling already, and nothing in the data tells you which. On synthbench data the ceiling is known exactly, so the gap between a score and the ceiling is measurable.
import warnings
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier
from synthbench import (
BenchPipeline,
FriedmanDGP,
MissingDataCorruptor,
severity_sweep,
)
plt.rcParams["figure.dpi"] = 72
warnings.filterwarnings("ignore", category=UserWarning)
Step 1: Load an OpenML benchmark task¶
We use OpenML task 59 (iris). The notebook loads the dataset directly from OpenML. A local sklearn fallback is included so the notebook runs without internet access.
try:
import openml
task = openml.tasks.get_task(59)
dataset = task.get_dataset()
X_raw, y_raw, _, _ = dataset.get_data(target=dataset.default_target_attribute)
X = X_raw.to_numpy(dtype=float)
le = LabelEncoder()
y = le.fit_transform(y_raw.to_numpy())
data_source = f"OpenML task 59 (iris): {X.shape[0]} rows, {X.shape[1]} features"
except Exception:
from sklearn.datasets import load_iris
iris_data = load_iris()
X, y = iris_data.data, iris_data.target
data_source = f"sklearn iris ({X.shape[0]} rows, {X.shape[1]} features)"
print(f"Data source: {data_source}")
print(f"X shape: {X.shape}, classes: {np.unique(y)}")
Data source: OpenML task 59 (iris): 150 rows, 4 features X shape: (150, 4), classes: [0 1 2]
Step 2: Three classifiers on the real data¶
classifiers = [
("LogisticRegression", LogisticRegression(max_iter=300, random_state=0)),
("RandomForest", RandomForestClassifier(n_estimators=50, random_state=0)),
("DecisionTree", DecisionTreeClassifier(max_depth=4, random_state=0)),
]
real_rows = []
for name, clf in classifiers:
scores = cross_val_score(clf, X, y, cv=5, scoring="accuracy")
real_rows.append(
{
"classifier": name,
"accuracy_mean": round(scores.mean(), 3),
"accuracy_std": round(scores.std(), 3),
"bayes_error": "unknown (real data)",
}
)
df_real = pd.DataFrame(real_rows)
print("Real data results:")
df_real
Real data results:
| classifier | accuracy_mean | accuracy_std | bayes_error | |
|---|---|---|---|---|
| 0 | LogisticRegression | 0.973 | 0.025 | unknown (real data) |
| 1 | RandomForest | 0.960 | 0.025 | unknown (real data) |
| 2 | DecisionTree | 0.967 | 0.037 | unknown (real data) |
Step 3: Synthbench data at the same size¶
Generate a FriedmanDGP classification dataset matched to the OpenML task's shape. Friedman #1 needs at least 5 features and iris has 4, so the synthbench runs use max(n_features, 5).
Two caveats before reading the tables side by side. Iris is a 3-class problem and synthbench targets are binary, so the accuracies are not directly comparable — what is comparable is each score's distance from its own ceiling. And bayes_error_analytic is the floor for this label mechanism, computed from the per-sample label probabilities the DGP drew from, so 1 - bayes_error_analytic is the best accuracy any classifier could reach.
n_samples, n_features = X.shape
# Friedman #1 needs at least 5 features; iris has 4.
synth_n_features = max(n_features, 5)
dgp = FriedmanDGP(task_type="classification", complexity="medium")
synth_clean = BenchPipeline(dgp).run(
n_samples=n_samples, n_features=synth_n_features, random_state=42
)
floor = synth_clean.metadata["bayes_error_analytic"]
ceiling = 1.0 - floor
synth_rows = []
for name, clf in classifiers:
scores = cross_val_score(
clf, synth_clean.X, synth_clean.y, cv=5, scoring="accuracy"
)
synth_rows.append(
{
"classifier": name,
"accuracy_mean": round(scores.mean(), 3),
"accuracy_std": round(scores.std(), 3),
"ceiling": round(ceiling, 3),
"gap_to_ceiling": round(ceiling - scores.mean(), 3),
}
)
print(f"Exact error floor (bayes_error_analytic): {floor:.4f}")
print(f"Best achievable accuracy: {ceiling:.4f}")
empirical = synth_clean.metadata["bayes_error"]
print(f"Empirical 1-NN estimate (bayes_error): {empirical:.4f}")
print(" -> the 1-NN estimate sits above the true floor; it is a proxy, not a bound")
df_synth = pd.DataFrame(synth_rows)
df_synth
Exact error floor (bayes_error_analytic): 0.3270 Best achievable accuracy: 0.6730 Empirical 1-NN estimate (bayes_error): 0.3800 -> the 1-NN estimate sits above the true floor; it is a proxy, not a bound
| classifier | accuracy_mean | accuracy_std | ceiling | gap_to_ceiling | |
|---|---|---|---|---|---|
| 0 | LogisticRegression | 0.640 | 0.098 | 0.673 | 0.033 |
| 1 | RandomForest | 0.633 | 0.042 | 0.673 | 0.040 |
| 2 | DecisionTree | 0.607 | 0.057 | 0.673 | 0.066 |
Step 4: What corruption does to the gap¶
Now apply MissingDataCorruptor at three severity levels and impute before fitting, which is what you would do with real missing data.
Watch which number moves. The analytic floor does not: missingness damages the features, not the labels, so the irreducible error is unchanged. What grows is the gap between achieved accuracy and the ceiling — that gap is signal the corruption destroyed and imputation failed to recover, and it is exactly what you cannot quantify on real data.
Note also that bayes_error, the empirical 1-NN estimate, is None in every row here. It is computed by distance, and a distance to NaN is undefined. This is the case that makes the analytic floor worth having.
dgp = FriedmanDGP(task_type="classification", complexity="medium")
sweep = severity_sweep(
dgp,
MissingDataCorruptor,
severities=["low", "medium", "high"],
n_samples=n_samples,
n_features=synth_n_features,
random_state=42,
)
imputer = SimpleImputer(strategy="mean")
corruption_rows = []
for severity, result in sweep.items(): # severity_sweep returns a dict keyed by level
X_imputed = imputer.fit_transform(result.X)
scores = cross_val_score(
RandomForestClassifier(n_estimators=50, random_state=0),
X_imputed,
result.y,
cv=5,
scoring="accuracy",
)
floor = result.metadata["bayes_error_analytic"]
empirical = result.metadata["bayes_error"]
corruption_rows.append(
{
"severity": severity,
"missing_fraction": round(float(np.isnan(result.X).mean()), 3),
"rf_accuracy": round(scores.mean(), 3),
"ceiling": round(1.0 - floor, 3),
"gap_to_ceiling": round((1.0 - floor) - scores.mean(), 3),
"bayes_error": (
"None (X has NaN)" if empirical is None else round(empirical, 4)
),
}
)
pd.DataFrame(corruption_rows)
| severity | missing_fraction | rf_accuracy | ceiling | gap_to_ceiling | bayes_error | |
|---|---|---|---|---|---|---|
| 0 | low | 0.047 | 0.660 | 0.673 | 0.013 | None (X has NaN) |
| 1 | medium | 0.147 | 0.627 | 0.673 | 0.046 | None (X has NaN) |
| 2 | high | 0.300 | 0.553 | 0.673 | 0.120 | None (X has NaN) |
What synthbench adds to an AMLB-style benchmark¶
On the real task we can rank the three classifiers but cannot say whether the best of them is near optimal. On synthbench data 1 - bayes_error_analytic gives the ceiling outright, so a score becomes interpretable on its own: a classifier at 0.72 against a ceiling of 0.74 has essentially solved the task, and one at 0.72 against a ceiling of 0.90 has not.
Corruption sharpens the same point. The ceiling is a property of the labels and stays put, while accuracy falls as features are degraded. The widening gap measures how much recoverable signal each severity level removed — a quantity that on real data you can only guess at, because you would have to introduce the missingness yourself and then assume you knew its effect.