Sweeps

synthbench has three sweep helpers for systematic ablation studies: severity_sweep, difficulty_sweep, and experiment_grid. All three functions return plain Python collections of BenchResult objects and use hierarchical SeedSequence derivation for reproducible, statistically independent seeds. The global NumPy RNG state is not modified.

severity_sweep

Purpose

Run a single DGP with one corruptor across a range of severity levels. Use this for ablation studies over corruption intensity at fixed DGP complexity. The same pre-built DGP instance is reused across all severity levels; the pipeline restores internal DGP state via try/finally so re-use is safe.

Usage

from synthbench import (
    BenchResult,
    LinearDGP,
    MeasurementNoiseCorruptor,
    severity_sweep,
)

dgp = LinearDGP(task_type="regression")
results = severity_sweep(
    dgp,
    MeasurementNoiseCorruptor,
    severities=["low", "medium", "high"],
    n_samples=500,
    n_features=10,
    random_state=42,
)

assert len(results) == 3
print(results[0].metadata["corruptor_params"])   # [{'key': 'measurement_noise', ...}]
print(results[2].metadata["bayes_error"])        # None (regression)

Parameters

Parameter	Type	Default	Description
`dgp`	DGP instance	required	Pre-constructed DGP instance. Reused for each severity level.
`corruptor_cls`	type	required	Corruptor class (not instance). Instantiated fresh per severity as `corruptor_cls(severity=sev, **corruptor_kwargs)`.
`severities`	`list[str]`	required	Ordered list of severity strings, e.g. `["low", "medium", "high"]`.
`n_samples`	`int`	`500`	Number of samples passed to `BenchPipeline.run`.
`n_features`	`int`	`10`	Number of features passed to `BenchPipeline.run`.
`random_state`	`int`	`0`	Master integer seed. Child seeds derived via `SeedSequence.spawn`.
`**corruptor_kwargs`			Extra keyword arguments forwarded to `corruptor_cls` at construction.

Reproducibility

Each severity level receives an independent child seed derived from the master random_state via numpy.random.SeedSequence.spawn. Passing the same arguments always produces bit-identical results.

Seeding contract

Two separate severity_sweep calls with the same random_state and the same len(severities) produce the same underlying child seeds. To obtain statistically independent sweeps, use different random_state values.

difficulty_sweep

Purpose

Run a DGP class across a list of complexity levels. Use this for ablation studies over signal complexity at fixed corruption. A fresh DGP instance is constructed per complexity level, so there is no state mutation across iterations.

Usage

from synthbench import LinearDGP, difficulty_sweep

results = difficulty_sweep(
    LinearDGP,
    complexities=["low", "medium", "high"],
    n_samples=300,
    n_features=8,
    random_state=0,
    task_type="classification",
)

assert len(results) == 3
print(results[0].metadata["dgp_params"]["complexity"])   # "low"
print(results[2].metadata["bayes_error"])                # float (classification)

Parameters

Parameter	Type	Default	Description
`dgp_cls`	type	required	DGP class (not instance). Instantiated per complexity level as `dgp_cls(complexity=complexity, **dgp_kwargs)`.
`complexities`	`list[str]`	required	Ordered list of complexity strings, e.g. `["low", "medium", "high"]`.
`corruptors`	`list` or `None`	`None`	Pre-constructed corruptor instances forwarded to each `BenchPipeline`. Defaults to `[]`.
`label_corruptors`	`list` or `None`	`None`	Pre-constructed label corruptor instances. Defaults to `[]`.
`n_samples`	`int`	`500`	Number of samples passed to `BenchPipeline.run`.
`n_features`	`int`	`10`	Number of features passed to `BenchPipeline.run`.
`random_state`	`int`	`0`	Master integer seed. Child seeds derived via `SeedSequence.spawn`.
`**dgp_kwargs`			Extra keyword arguments forwarded to `dgp_cls` at construction (e.g. `task_type="classification"`). Do not include `complexity` here.

Notes on DGP compatibility

All built-in non-neural DGPs (LinearDGP, TreeDGP, PolynomialDGP, FriedmanDGP, AdditiveDGP, SparseDGP, GeometricDGP) accept a complexity parameter. Pass additional DGP constructor arguments via **dgp_kwargs. Do not include complexity in dgp_kwargs; it is passed explicitly by the sweep function.

experiment_grid

Purpose

Full factorial grid: all combinations of n_samples × DGP complexity × corruptor severity. Use this for publication-quality ablation tables where you need every cell of the cross product.

Usage

from synthbench import LinearDGP, OutlierCorruptor, experiment_grid

grid = experiment_grid(
    LinearDGP,
    OutlierCorruptor,
    n_samples_list=[200, 500],
    complexities=["low", "high"],
    severities=["low", "high"],
    n_features=10,
    random_state=0,
    task_type="regression",
)

# Keys are (n_samples, complexity, severity) tuples
print(list(grid.keys()))
# [(200, 'low', 'low'), (200, 'low', 'high'), (200, 'high', 'low'), ...]

result = grid[(500, "high", "high")]
print(result.X.shape)   # (500, 10)
print(result.metadata["dgp_params"]["complexity"])   # "high"

Parameters

Parameter	Type	Default	Description
`dgp_cls`	type	required	DGP class. A fresh instance is constructed per cell as `dgp_cls(complexity=complexity, **dgp_kwargs)`.
`corruptor_cls`	type	required	Corruptor class. A fresh instance is constructed per cell as `corruptor_cls(severity=severity)`.
`n_samples_list`	`list[int]`	required	List of sample counts to include in the grid.
`complexities`	`list[str]`	required	List of complexity strings to include in the grid.
`severities`	`list[str]`	required	List of severity strings to include in the grid.
`n_features`	`int`	`10`	Number of features passed to `BenchPipeline.run`.
`random_state`	`int`	`0`	Master integer seed for the root `SeedSequence`.
`**dgp_kwargs`			Extra keyword arguments forwarded to `dgp_cls` (e.g. `task_type="regression"`).

Return value

Returns dict[tuple[int, str, str], BenchResult] where each key is (n_samples, complexity, severity). Total entries equals len(n_samples_list) × len(complexities) × len(severities).

Seeding hierarchy

Cell seeds use a three-level SeedSequence hierarchy:

master_seed
  └── n_samples branch[i]
        └── complexity branch[j]
              └── severity branch[k] → cell seed

This guarantees that adjacent cells (e.g. (200, "low", "low") vs (200, "low", "medium")) have different data even when they share all other parameters. Adding a fourth axis in the future only affects new cells; existing cells are unaffected.

Seeding contract

The three-level hierarchy means that grid[(n, c, s)] has a different seed than a standalone severity_sweep(..., severities=[s], random_state=0) call because the nesting levels differ. Use a single function throughout a study to ensure consistency.