Sweeps

Three helpers for ablations. They exist mostly to get the seeding right: writing the loop yourself invites either reusing one seed across cells (so differences between cells are partly just shared noise) or seeding by loop index (so the cells of two different sweeps line up in ways you did not intend).

All three derive per-cell seeds from numpy.random.SeedSequence and never touch the global NumPy RNG.

severity_sweep

Hold the DGP fixed, vary one corruptor's severity. Returns a dict keyed by level.

from synthbench import LinearDGP, MeasurementNoiseCorruptor, severity_sweep

results = severity_sweep(
    LinearDGP(task_type="regression"),
    MeasurementNoiseCorruptor,
    severities=["low", "medium", "high"],
    n_samples=500,
    n_features=10,
    random_state=42,
)

for level, result in results.items():
    print(level, result.metadata["effective_rank"])

Pass the corruptor class, not an instance — a fresh one is built per level as corruptor_cls(severity=level, **corruptor_kwargs). The DGP is passed as an instance and reused across levels, which is safe because the pipeline restores its random_state afterwards.

difficulty_sweep

Hold corruption fixed, vary the DGP's complexity. Also returns a dict keyed by level.

from synthbench import LinearDGP, difficulty_sweep

results = difficulty_sweep(
    LinearDGP,
    complexities=["low", "medium", "high"],
    n_samples=300,
    n_features=8,
    random_state=0,
    task_type="classification",
)

for level, result in results.items():
    print(level, result.metadata["bayes_error_analytic"])

Here it is the DGP that is a class — a fresh instance per level, so nothing carries over. Extra constructor arguments go in as keyword arguments; leave complexity out, since the sweep supplies it. Corruptors, if you pass any, are instances shared across all levels.

Every non-neural DGP takes complexity. RandomNeuralDGP does not, so it cannot be used here — size its network directly instead.

experiment_grid

The full cross product of sample size, complexity, and severity. Keys are (n_samples, complexity, severity) tuples.

from synthbench import LinearDGP, OutlierCorruptor, experiment_grid

grid = experiment_grid(
    LinearDGP,
    OutlierCorruptor,
    n_samples_list=[200, 500],
    complexities=["low", "high"],
    severities=["low", "high"],
    n_features=10,
    random_state=0,
    task_type="regression",
)

print(len(grid))  # 2 x 2 x 2 = 8
print(grid[(500, "high", "high")].X.shape)  # (500, 10)

Cell seeds come from a three-level spawn hierarchy rather than a flat enumeration:

master seed
  └── n_samples branch[i]
        └── complexity branch[j]
              └── severity branch[k] → cell seed

Nesting is what makes (200, "low", "low") and (200, "low", "medium") genuinely different datasets despite sharing two coordinates. It also means adding a fourth level later would not disturb the seeds of existing cells.

Two things to know about the seeding

Sweeps with the same random_state and the same number of levels draw the same child seeds. If you want independent replicates of a whole sweep, vary random_state — not the level list.

Seeds also depend on the nesting depth, so grid[(n, c, s)] does not contain the same data as severity_sweep(..., severities=[s], random_state=0). Pick one helper for a given study and stay with it, or the cells will not be comparable.