BenchSuite

A suite is a named set of datasets that runs in one call. The useful property is that the name pins the data: "we evaluated on easy-classification" is a complete description, so nobody has to ship arrays around to compare results.

Two suites ship with the package:

Name	Contents
easy-classification	Low-complexity classification across LinearDGP, TreeDGP, FriedmanDGP
hard-regression	High-complexity regression across PolynomialDGP, FriedmanDGP, AdditiveDGP

from synthbench import BenchSuite

suite = BenchSuite("easy-classification")
results = suite.run()

print(list(results))  # ['linear_low', 'tree_low', 'friedman_low']
print(results["linear_low"].X.shape)
print(BenchSuite.list_suites())

run() returns a dict keyed by each entry's label, in spec order. Running the same suite twice gives bit-identical data, provided the synthbench, NumPy, and scikit-learn versions match.

Your own suites

Pass a dict, or a path to a JSON file holding one:

from synthbench import BenchSuite

spec = {
    "name": "imputation-study",
    "description": "One clean baseline plus the same DGP under three missingness mechanisms.",
    "entries": [
        {
            "label": "clean",
            "dgp_key": "linear",
            "dgp_kwargs": {"task_type": "classification", "complexity": "medium"},
            "n_samples": 500,
            "n_features": 10,
            "random_state": 0,
        },
        *(
            {
                "label": f"missing_{mechanism}",
                "dgp_key": "linear",
                "dgp_kwargs": {"task_type": "classification", "complexity": "medium"},
                "corruptors": [
                    {
                        "key": "missing_data",
                        "params": {"proportion": 0.2, "mechanism": mechanism},
                    }
                ],
                "n_samples": 500,
                "n_features": 10,
                "random_state": 0,
            }
            for mechanism in ("mcar", "mar", "mnar")
        ),
    ],
}

results = BenchSuite.from_dict(spec).run()
print(list(results))

An entry needs label, dgp_key, n_samples, and random_state. n_features defaults to 10, and dgp_kwargs, corruptors, and label_corruptors default to empty. Registry keys are the short names — "linear", "tree", "missing_data" — not class names.

Specs are checked when the suite is constructed, not when it runs, so a typo fails immediately instead of part-way through generating data. Labels must be unique, since they key the result dict.

BenchSuite.from_json("path/to/spec.json") reads the same structure from disk. The bundled specs under synthbench/data/suites/ are working examples of the format. A bundled name always wins over a same-named file in the working directory, so BenchSuite("easy-classification") means the same thing everywhere.