Build SWE-bench instance images on aarch64 for Isambard

[12yuens2]

What this is

The first piece of adding SWE-bench: getting an instance image onto Isambard at
all, and knowing it is a faithful instance. Nothing in mas/ or tasks/envs/
yet — the environment, loader and prompts follow, and the plan for them is at
the end.

Why the published images cannot be used

SWE-bench grades a patch by running a repository's own tests inside a
per-instance image. Docker Hub carries 4,503 of those under the swebench
namespace, every one tagged sweb.eval.x86_64.*; a search for arm64 returns
nothing. Isambard-AI is aarch64 and has no qemu handler registered:

$ ls /proc/sys/fs/binfmt_misc/
register  status
$ podman-hpc run --rm --platform linux/amd64 ubuntu:jammy uname -m
runc create failed … Exec format error: '/sbin/ldconfig'

So the images are built natively, and the task Dockerfiles published in
SWE-bench/swe-bench-tasks do not build on aarch64 as they stand.

Four things had to change, and one was not expected

1. FROM --platform=linux/amd64 — the flag.

2. The Miniconda installer, Linux-x86_64.sh to Linux-aarch64.sh.

3. The conda pins in the embedded environment.yml. Every dependency carries a
   build string — python=3.11.10=he870216_0 — and those hashes exist only
   for linux-64. The version is kept and the build string dropped so conda
   solves the same versions for linux-aarch64; six packages whose linux-64
   names have no aarch64 build at all are dropped, and conda pulls the aarch64
   equivalents in as dependencies. The pip: section is left alone, since its
   pins are == and PyPI resolves them per platform.

4. The heredoc RUN blocks — where the conda environment is created and the
   repository cloned. buildah here cannot run them:

   STEP 11/14: RUN <<EOF_a88cf31974aa
   invalid mount … /dev/pipes/buildahheredoc…: bind mounts cannot have any
   filesystem-specific options applied
   

   A heredoc whose body opens with a shebang is mounted as a file rather than
   piped to the shell, and runc refuses the SELinux options buildah puts on that
   mount. Two cheaper fixes failed: --isolation=chroot exits 127, and a
   SHELL ["/bin/bash", "-c"] directive is ignored for heredocs, so
   set -euo pipefail still reaches /bin/sh and dies on Illegal option -o pipefail. A heredoc with no shebang builds fine, so those are left alone.
   The rest are extracted to scripts named after their own delimiter and
   replaced with a COPY and a RUN bash; a nested heredoc inside the body —
   the cat <<'EOF' > environment.yml carrying the whole conda environment —
   survives because the body is written out unchanged.

The build has to be a Slurm job

loginctl enable-linger is denied for user accounts, so logind kills every user
process when the last session closes. A setsid --fork build died mid-apt, and
so did a tmux session from the brics/tmux module. Compute nodes have outbound
network (pypi:200 from nid010975) and podman-hpc, and migrate writes to
$SCRATCH, so swebench_build.sbatch does the whole thing and outlives the
session that submitted it.

Verified

django__django-16485 built in 6m23s, is 2.56 GB, and shows R/O = true
so compute nodes can read it. The instance reproduces the benchmark's verdicts —
which matters because the aarch64 solve gives the same package versions, not
the same builds:

run	FAIL_TO_PASS test_zero_values	PASS_TO_PASS (9)
no gold patch	ERROR: ValueError: valid range for prec is [1, MAX_PREC]	all pass
gold patch applied	ok — Ran 10 tests … OK	all pass

That ValueError is the issue the instance is about, so the environment is
reproducing the bug rather than a build artefact of its own.

Files

• containers/swebench_arm64.py — the rewriter, and Rewritten.write, which
  lays out a build context.
• containers/build_swebench_image.sh — copies the task directory to its own
  context, rewrites the Dockerfile into it, builds with podman-hpc --layers,
  migrates the image.
• containers/swebench_build.sbatch — that, as a job.
• containers/isambard.md — setup, the build, the faithful-instance check, and
  the limits.
• tasks/tests/test_swebench_arm64.py — ten tests over the rewriter, each
  mutation-checked.

Two smaller things the build taught us, both now in the script and the doc:
podman-hpc resolves -f inside the build context, so the generated Dockerfile
is written there; and the login node's system python is 3.6, so the rewriter
needs a uv-provided 3.12.

What comes next in this PR

The dataset itself, in dependency order: a data/swebench/verified.jsonl
manifest naming instances (the task repo clone holds the statements and patches,
as rom_dir holds Jericho's roms), a loader, SWEBenchEnv/SWEBenchRecorder
registered in ENVS/RECORDERS, a config block, a prompt module and an
intrinsicmemory-swebench template.

Two findings already shape that work:

• SWE-bench cannot use clean_action_line. Measured against the current
  cleaner, ls tests/*.py becomes ls tests/.py, python -c "print(1)" > /tmp/out loses its redirection, a fenced ./tests/runtests.py x loses its
  leading dot, sed -i "s/#comment//" loses the #, and a heredoc becomes
  cat EOF /tmp/p.py. _undecorate treats <, >, *, #, backticks and
  leading dots as decoration, which is right for the other datasets and fatal
  for a bash action space. It needs its own process_action, with a test per
  form above.
• The reward has to be milestones, not a test fraction. The median Verified
  instance has one FAIL_TO_PASS test (mean 3.0, max 438), so a fraction would be
  binary in practice. A non-empty diff, the test command running, the
  FAIL_TO_PASS fraction, PASS_TO_PASS preserved as a gate, resolved for all of
  it — with done reserved for resolved, as Jericho reserves it for victory.

The experiment the dataset is for is not a leaderboard number: the same
instances run repo-major, where a cross-task memory can accumulate knowledge of
one repository, against interleaved, where no two consecutive tasks share a
repository. The empty-memory arm is flat across both orders by construction,
which makes it the control.

[12yuens2]

Five repositories tried: four build, and all four are faithful instances

Each built as an sbatch job, in about four minutes, 2.3–2.6 GB, all migrated
(R/O = true).

instance	version	FAIL_TO_PASS unpatched	with gold patch	PASS_TO_PASS
django__django-16485	5.0	ERROR: ValueError: valid range for prec is [1, MAX_PREC]	ok — Ran 10 tests … OK	9/9 pass
psf__requests-2931	2.9	FAILED … UnicodeDecodeError	86 passed	84/84 pass
pytest-dev__pytest-10051	7.2	FAILED test_clear_for_call_stage	16 passed	15/15 pass
sympy__sympy-21379	1.9	PolynomialError: Piecewise generators do not make sense — 1 exception	93 passed, no exceptions	91/91 pass

In every case the unpatched failure is the issue the instance is about, not a
build artefact of the aarch64 environment.

pydata__xarray-7229 does not build, and will not. Its environment pins
cdms2, cdtime and libcdms — the CDAT stack — and none has a linux-aarch64
build in any channel:

PackagesNotFoundError: The following packages are not available from current channels:
  - cdms2=3.1.5*
  - cdtime=3.1.4*
  - libcdms=3.1.2*
Current channels:
  - https://repo.anaconda.com/pkgs/main/linux-aarch64
  - https://conda.anaconda.org/conda-forge/linux-aarch64

That is the failure shape to expect — a dependency never built for this
architecture — and the answer is to skip the instance. It costs little: django
alone is 231 of the 500 Verified instances, sympy 75, sphinx 44.

One finding that changes the env's log parser

eval.sh runs whole test files, so its summary line is not the verdict.
psf__requests-2931 reports 81 errors in both runs — tests wanting a live
httpbin — and none of them appear in FAIL_TO_PASS or PASS_TO_PASS. Read against
tests.json, all 84 PASS_TO_PASS pass and the single FAIL_TO_PASS test flips.
A parser that reads totals would reject a perfectly good instance; it has to
compare test names.

Now in containers/isambard.md.