Generate the calibration scripts instead of keeping one by hand

[12yuens2]

Why

slurm/calibrate.sh was hardcoded to fever — the task, the twenty-task cap and the intrinsicmemory-fever arm were all written into the file. Calibrating another dataset meant editing three places and remembering a fourth: pddl serves at --max-num-seqs 256, not 512, which generate_slurm.py already knew and the hand-written script did not. Since then main has grown more of the same per-dataset knowledge — the batched-token and model-length overrides, prefix caching, --max_tokens per task — none of which a hand-written script picks up.

What

slurm/generate_calibration.py renders slurm/<task>_calibrate.sh, for every task or for the ones --task names, and slurm/calibrate.sh stops being tracked:

uv run slurm/generate_calibration.py                  # every dataset
uv run slurm/generate_calibration.py --task fever pddl

It holds what a calibration is: one seed, --max_tasks 20 at the full trial budget, every arm, a 2-hour allocation, Jericho's smaller shakedown, and the result table / tokens-per-task / failed-tasks summary the job prints afterwards. They default to $HOME/GMemory/.db/sweep/calibration, so calibrating several datasets fills one table.

The cluster, the model and the arms stay in generate_slurm.py, which exposes job_script(...) for the serve-then-run body. Its render() is experiment-or-crosstask again, and the per-variant pieces are keyword arguments rather than a variant string a branch switches on — so the calibration's options move without touching the sweep. Both generators take --task.

#SBATCH --account now comes from SLURM_ACCOUNT at generation time rather than being committed, since it is one user's allocation — it was being added by hand to the working copy of every script, which is a local edit that survives no checkout.

Unchanged

The sweep scripts this generates are byte-identical to the ones main generates — verified by generating both sides and diffing all twelve.

🤖 Generated with Claude Code

[12yuens2]

let's move the calibration to a separate python script instead, so its easier and we can also set different options for the calibration if needed

[12yuens2]

We should separate the calibrate generation to a separate python file to keep the two clean

[12yuens2]

Done — slurm/generate_calibration.py is its own script now.

It holds what a calibration is: one seed, the twenty-task cap, the 2-hour window, Jericho's smaller shakedown and the summary the job prints. generate_slurm.py keeps the cluster, the model and the arms, and exposes job_script(...) for the serve-then-run body, so its render() is back to experiment-or-crosstask with no variant branch. job_script takes the per-variant pieces as keyword arguments, so setting different options for the calibration means editing only the calibration file.

uv run slurm/generate_calibration.py                  # every dataset
uv run slurm/generate_calibration.py --task fever pddl

Also merged main, which had resized the sweep per dataset in the same three places. The calibrations now inherit its serve-block changes and take --max_tokens from MAX_TOKENS_OVERRIDES rather than a fixed 2048 — calibrating a budget the real job will not use measures the wrong thing. The sweep scripts remain byte-identical to the ones main generates.

[12yuens2]

we should make all these args with defaults so they can be changed easily when running the script

[12yuens2]

Done — --seed, --max_tasks, --max_trials, --time_limit and --db_dir, each defaulting to what the constant held:

uv run slurm/generate_calibration.py --max_tasks 5 --time_limit 00:30:00
uv run slurm/generate_calibration.py --task jericho --max_trials 10

--max_trials is new — it was only reachable before through the per-dataset table. A flag given on the command line beats that table, so --max_tasks 30 means 30 for Jericho too, rather than its entry holding it to 5.

Generating with no flags writes byte-identical scripts to the ones the constants wrote.

[12yuens2]

this is way too much info, you should /caveman just 1 or 2 lines here

[12yuens2]

Cut to two:

# Jericho's prompt tokens grow with the square of its 100-trial budget, so 20
# tasks would be ~288M tokens - an 18-hour job. Five at 20 trials is ~8M.
OVERRIDES = {"jericho": {"max_tasks": 5, "max_trials": 20}}

The curve it was quoting is in data/data.md, and the rest was reasoning about the 2-hour window that nobody reading the constant needs.