Sweeps & Suite API Reference
severity_sweep
Hold the DGP fixed and vary one corruptor's severity.
A fresh corruptor is constructed per level, each pipeline run getting its
own child seed from SeedSequence.spawn(len(severities)).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dgp
|
A pre-constructed DGP instance (e.g. |
required | |
corruptor_cls
|
Corruptor class, not an instance. Built per level as
|
required | |
severities
|
list[str]
|
Severity levels to run, e.g. |
required |
n_samples
|
int
|
Number of samples passed to |
500
|
n_features
|
int
|
Number of features passed to |
10
|
random_state
|
int
|
Master seed. Same arguments in, bit-identical results out. |
0
|
**corruptor_kwargs
|
Forwarded to |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, BenchResult]
|
Keyed by severity, in the order given. |
Examples:
difficulty_sweep
Hold corruption fixed and vary the DGP's signal complexity.
A fresh DGP is constructed per level, so nothing carries over between
iterations, and each level gets its own child seed from
SeedSequence.spawn(len(complexities)).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dgp_cls
|
DGP class, not an instance. Built per level as
|
required | |
complexities
|
list[str]
|
Complexity levels to run, e.g. |
required |
corruptors
|
list | None
|
Pre-constructed corruptor instances applied to every level. |
None
|
label_corruptors
|
list | None
|
Pre-constructed label corruptor instances applied to every level. |
None
|
n_samples
|
int
|
Number of samples passed to |
500
|
n_features
|
int
|
Number of features passed to |
10
|
random_state
|
int
|
Master seed. |
0
|
**dgp_kwargs
|
Forwarded to |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, BenchResult]
|
Keyed by complexity, in the order given. |
experiment_grid
Run the full cross product of sample size, complexity, and severity.
Cell seeds come from a three-level SeedSequence.spawn hierarchy::
root = SeedSequence(random_state)
n_branch = root.spawn(len(n_samples_list))[i]
c_branch = n_branch.spawn(len(complexities))[j]
s_branch = c_branch.spawn(len(severities))[k]
seed = int(s_branch.generate_state(1)[0])
Nesting rather than flat enumeration means neighbouring cells such as
(200, "low", "low") and (200, "low", "medium") get unrelated data
despite sharing two of three coordinates.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dgp_cls
|
DGP class. Built per cell as |
required | |
corruptor_cls
|
Corruptor class. Built per cell as |
required | |
n_samples_list
|
list[int]
|
Sample sizes to cross. |
required |
complexities
|
list[str]
|
Complexity levels to cross. |
required |
severities
|
list[str]
|
Severity levels to cross. |
required |
n_features
|
int
|
Number of features passed to |
10
|
random_state
|
int
|
Master seed for the root |
0
|
**dgp_kwargs
|
Forwarded to |
{}
|
Returns:
| Type | Description |
|---|---|
dict[tuple[int, str, str], BenchResult]
|
Keyed by |
BenchSuite
A named set of benchmark datasets, generated together.
Running a suite is one call, and the same suite always yields bit-identical BenchResult objects, so a suite name is enough to identify a shared baseline in a paper or an issue report.
Two suites ship with the package — "easy-classification" and
"hard-regression". Beyond those, pass a path to a JSON spec or a spec
dict to assemble your own; see
from_dict for the shape.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
str | Path | dict
|
A bundled suite name, a path to a JSON spec file, or a spec dict. |
required |
Examples:
>>> suite = BenchSuite("easy-classification")
>>> results = suite.run()
>>> list(results.keys())
['linear_low', 'tree_low', 'friedman_low']
description
property
The suite's free-text description, or an empty string.
from_dict(spec)
classmethod
Build a suite from a spec dict.
A spec is a name, an optional description, and a list of
entries. Each entry needs a unique label, a dgp_key
("linear", "tree", ...), n_samples, and random_state;
n_features defaults to 10, and dgp_kwargs / corruptors /
label_corruptors default to empty. Corruptor entries are
{"key": ..., "params": {...}}.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
dict
|
The suite spec. |
required |
Returns:
| Type | Description |
|---|---|
BenchSuite
|
|
Examples:
>>> suite = BenchSuite.from_dict(
... {
... "name": "my-suite",
... "entries": [
... {
... "label": "linear_noisy",
... "dgp_key": "linear",
... "dgp_kwargs": {"task_type": "classification"},
... "corruptors": [
... {"key": "missing_data", "params": {"proportion": 0.1}}
... ],
... "n_samples": 200,
... "random_state": 0,
... }
... ],
... }
... )
>>> list(suite.run())
['linear_noisy']
from_json(path)
classmethod
Build a suite from a JSON file holding a spec.
The bundled specs under synthbench/data/suites/ are working
examples of the format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Path to the JSON file. |
required |
Returns:
| Type | Description |
|---|---|
BenchSuite
|
|
list_suites()
staticmethod
Return the bundled suite names, sorted.
Returns:
| Type | Description |
|---|---|
list[str]
|
|
run()
Generate every dataset in the suite.
Returns:
| Type | Description |
|---|---|
dict[str, BenchResult]
|
Keyed by each entry's |