Skip to content

synthbench

synthbench generates synthetic datasets for benchmarking machine-learning methods. You choose how complicated the signal is, layer noise or missing data on top, and get back a dataset that carries a record of how it was made. One integer seed reproduces the whole thing.

The reason to use synthetic data here is that you know the answer in advance. On a real dataset you can measure that your model scored 0.83 and have no way to tell whether 0.85 was available or whether you already hit the ceiling. Here the ceiling is in the metadata, next to the ground truth about which features were supposed to matter.

Installation

pip install synthbench

Two optional extras: synthbench[io] adds Parquet support, and synthbench[neural] pulls in PyTorch for RandomNeuralDGP. Everything else works from the base install.

Quickstart

from synthbench import BenchPipeline, LinearDGP, MissingDataCorruptor

pipeline = BenchPipeline(
    LinearDGP(complexity="medium", task_type="classification"),
    corruptors=[MissingDataCorruptor(proportion=0.1, mechanism="mar")],
)
result = pipeline.run(n_samples=500, n_features=10, random_state=42)

print(result.X.shape)  # (500, 10)
print(result.metadata["bayes_error_analytic"])  # the error floor
print(result.metadata["signal_feature_importances"])

Data-generating processes

Each takes a complexity parameter (except RandomNeuralDGP, which is sized directly) and records which features carry signal. All but FriedmanDGP also take n_classes for multiclass targets — see BenchPipeline.

DGP Signal
LinearDGP Linear combination, with sparsity and noise set by complexity
PolynomialDGP Polynomial terms and interactions
TreeDGP Axis-aligned splits from a random tree
FriedmanDGP The Friedman (1991) benchmark functions 1, 2, 3
AdditiveDGP Sum of univariate functions, no interactions
SparseDGP Exactly k informative features, rest pure noise
GeometricDGP Moons, circles, spirals
RandomNeuralDGP A randomly initialised MLP

Where to go next

  • Corruptors — noise, outliers, missing data (MCAR/MAR/MNAR), collinearity, binning, and label noise
  • BenchPipeline — composing a DGP with corruptors, and what the metadata means
  • Sweeps — ablations over severity, complexity, or a full grid
  • Suites — named dataset collections, bundled and custom
  • Serialization — Parquet, CSV, and regenerating data from metadata alone
  • Examples — runnable notebooks
  • API reference — every public symbol