BenchPipeline

BenchPipeline is what you use to actually generate data. It takes a DGP, an optional chain of corruptors, and produces a BenchResult carrying the data plus a record of how it was made.

Its job is mostly bookkeeping you would otherwise do by hand: fixing the order corruptors run in, handing each component an independent seed, and tracking how much information each corruptor destroyed.

from synthbench import (
    BenchPipeline,
    LinearDGP,
    MeasurementNoiseCorruptor,
    MissingDataCorruptor,
)

pipeline = BenchPipeline(
    LinearDGP(complexity="high", task_type="regression", random_state=0),
    corruptors=[
        MeasurementNoiseCorruptor(noise_level=0.3),
        MissingDataCorruptor(proportion=0.05),
    ],
)
result = pipeline.run(n_samples=1000, n_features=20, random_state=42)

print(result.X.shape)  # (1000, 20)
print(result.y.shape)  # (1000,)

A BenchResult has three attributes: X, y, and metadata.

What lands in the metadata

result.metadata["dgp_class"]  # "LinearDGP"
result.metadata["dgp_params"]  # constructor params, plus dgp_key
result.metadata["signal_feature_importances"]  # ground truth from the DGP
result.metadata["effective_feature_importances"]  # after corruption
result.metadata["corruptor_order"]  # class names, in application order
result.metadata["corruptor_params"]  # per-corruptor settings
result.metadata["bayes_error_analytic"]  # exact error floor, or None
result.metadata["bayes_error"]  # empirical 1-NN LOO estimate, or None
result.metadata["label_mechanism"]  # how the labels were drawn
result.metadata["effective_rank"]  # Roy & Vetterli (2007)
result.metadata["synthbench_version"]  # plus numpy_version, python_version

That is enough to reconstruct the pipeline later — see Serialization.

Signal versus effective importances

signal_feature_importances is the DGP's ground truth and never changes. Each corruptor updates effective_feature_importances to reflect what survived, which lets you ask how much of the signal a given corruption chain actually removed.

signal = result.metadata["signal_feature_importances"]
effective = result.metadata["effective_feature_importances"]

print(sum(signal.values()))  # 1.0
print(sum(effective.values()))  # <= 1.0 after corruption

How hard is the dataset?

Two numbers, measuring different things. Mixing them up is easy and leads to overclaiming, so:

bayes_error_analytic is exact. Every classification DGP except GeometricDGP draws each label from an explicit per-sample probability p, so a predictor that knew p perfectly would still be wrong with probability min(p, 1 − p). Averaging that gives the irreducible floor. No model can beat it. Because it describes the labels, feature corruption does not move it, and it is None for regression.

bayes_error is an estimate — a 1-NN leave-one-out error rate on the post-corruption feature matrix. It responds to corruption, which makes it useful for ranking severity levels against each other. It is not a bound, and in more than a handful of dimensions it is biased well upwards: for a balanced LinearDGP classification task at n=500, p=10, the analytic floor is about 0.325 while this reads about 0.456.

from synthbench import BenchPipeline, LinearDGP

result = BenchPipeline(LinearDGP(task_type="classification")).run(
    n_samples=500, n_features=10, random_state=0
)
print(result.metadata["bayes_error_analytic"])  # ~0.325 — a real floor
print(result.metadata["bayes_error"])  # ~0.456 — a biased proxy
print(result.metadata["effective_rank"])

bayes_error is None for regression, for fewer than two samples, and when X contains NaN, since a distance to NaN is undefined. effective_rank is computed for both task types but is also None on NaN. So a pipeline ending in MissingDataCorruptor reports neither — the analytic floor still holds, which is another reason to prefer it.

Both generalise to more than two classes. The floor becomes mean(1 − max_k p_k), which is the same quantity: the error a predictor makes even knowing every class probability exactly.

More than two classes

Pass n_classes to any DGP except FriedmanDGP, and set the prior with class_weight:

from synthbench import BenchPipeline, LinearDGP

result = BenchPipeline(
    LinearDGP(task_type="classification", n_classes=4, class_weight="balanced")
).run(n_samples=1000, n_features=10, random_state=0)

print(sorted(set(result.y)))  # [0.0, 1.0, 2.0, 3.0]
print(result.metadata["bayes_error_analytic"])
print(result.metadata["label_mechanism"])  # "softmax"

class_weight takes three forms. A float is the positive-class prior and works only for two classes — above that it raises rather than guessing what you meant. "balanced" gives a uniform prior. A sequence of n_classes positive values summing to 1 sets the prior outright.

Which mechanism, and why it matters

label_mechanism records how the labels were drawn, because the answer changes what the dataset actually tests:

Mechanism	DGPs	Classes are
`"softmax"`	`LinearDGP`, `SparseDGP`, `TreeDGP`, `RandomNeuralDGP`	Unordered — any class can be confused with any other
`"ordinal"`	`PolynomialDGP`, `AdditiveDGP`	Ordered — only neighbouring classes are confusable
`"geometric"`	`GeometricDGP`	Regions in the plane; no analytic floor
`"binary"`	any, at `n_classes=2`	The two-class mechanism, unchanged

The split is not arbitrary. A softmax needs one latent score per class, which a DGP can only supply if its signal is parameterised rather than fixed — a coefficient matrix, a wider output layer, vector-valued leaves. PolynomialDGP and AdditiveDGP are built on a fixed term basis, so their one signal gets cut into ordered bins instead. That is a real structure, and it suits genuinely ordinal targets, but a method that internally regresses a single score will score unrealistically well on it. If you are benchmarking general multiclass, prefer a "softmax" DGP.

FriedmanDGP refuses n_classes > 2 outright. Its functions are specific published formulas; inventing multi-output variants would produce something not comparable to anything anyone has reported for Friedman #1–#3. GeometricDGP grows extra rings or spiral arms, except for shape="moons", which is two interleaving arcs by definition.

Corruptor order

Feature corruptors always run in one order, whatever order you list them in:

CollinearityCorruptor
CategoricalCorruptor
MeasurementNoiseCorruptor
OutlierCorruptor
MissingDataCorruptor

The reason is that these operations do not commute — binning after adding noise gives different data than binning before — so a fixed order is what makes two pipelines with the same components produce the same dataset. Pass them out of order and you get a UserWarning telling you what the pipeline did instead.

Label corruptors are separate. They run after all feature corruptors, in the order you give them, and go in the label_corruptors= argument.

Reproducibility

The same random_state always gives the same data. The master seed feeds numpy.random.SeedSequence, which spawns one child seed per component, so the DGP and each corruptor draw from independent streams and the global NumPy RNG is never touched.

import numpy as np

r1 = pipeline.run(n_samples=100, n_features=5, random_state=0)
r2 = pipeline.run(n_samples=100, n_features=5, random_state=0)

assert np.array_equal(r1.X, r2.X, equal_nan=True)

Reusing one DGP instance across runs is safe: run saves and restores the DGP's random_state in a finally block.

equal_nan=True when checking a pipeline that injects NaN

NaN != NaN, so a plain np.array_equal(r1.X, r2.X) returns False for any pipeline ending in MissingDataCorruptor even though both runs are byte-for-byte identical. That looks like a reproducibility failure and is not one.

Generating replicates

Varying only random_state gives independent draws from the same configuration, which is what you want for error bars.

from synthbench import BenchPipeline, LinearDGP, MeasurementNoiseCorruptor

pipeline = BenchPipeline(
    LinearDGP(task_type="regression"),
    corruptors=[MeasurementNoiseCorruptor(severity="medium")],
)
replicates = [
    pipeline.run(n_samples=300, n_features=8, random_state=seed) for seed in range(10)
]

For varying something other than the seed, use the sweep helpers rather than writing the loop — they handle seed derivation so that cells stay independent.