Skip to content

Selftest wipes PONYTAIL_PLUGIN_DIR before the run can use it - #717

Open
KimHyeongRae0 wants to merge 1 commit into
DietrichGebert:mainfrom
KimHyeongRae0:fix/selftest-clobbers-plugin-dir-env
Open

Selftest wipes PONYTAIL_PLUGIN_DIR before the run can use it#717
KimHyeongRae0 wants to merge 1 commit into
DietrichGebert:mainfrom
KimHyeongRae0:fix/selftest-clobbers-plugin-dir-env

Conversation

@KimHyeongRae0

Copy link
Copy Markdown

I was re-running benchmarks/agentic to check the numbers in results/2026-06-18-agentic.md — different machine, plugin not installed there, so I set PONYTAIL_PLUGIN_DIR the way benchmarks/agentic/README.md suggests. Baseline cells ran fine, then the first ponytail cell died:

ponytail plugin dir not found under ~/.claude/plugins/cache/ponytail/ponytail; install the plugin or set PONYTAIL_PLUGIN_DIR

Which was confusing, because I had.

_selftest_plugin_dir() sets the var to a sentinel to prove the override is honored, then unconditionally dels it in the finally. The selftest runs ahead of every real run, so by the time _plugin_dir() is called for an actual cell the variable is gone and it falls through to the cache-dir branch.

Bit of an irony: the docstring on _plugin_dir says hardcoding one machine's path "made the ponytail/caveman arms unreproducible off that box" — and the env override that solves that is the thing getting cleared.

Fix saves the prior value and restores it. With PONYTAIL_PLUGIN_DIR=/tmp/x:

before: env after selftest = None
after:  env after selftest = '/tmp/x'

python run.py --selftest still reports all instruments valid, and the run I was after finishes now.

For what it's worth, the reproduction held up well — median src_loc on datepicker came out 401 baseline / 22.5 ponytail against your published 404 / 23, two months later on a different box.

Tried reproducing the agentic benchmark on a box without the plugin
installed, set PONYTAIL_PLUGIN_DIR the way the README says, and every
ponytail cell died with "plugin dir not found".

_selftest_plugin_dir() borrows the var for its sentinel check and then
deletes it outright in the finally block. The selftest runs ahead of
every real run, so the override never survives long enough to be used.

Save the old value, put it back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant