Selftest wipes PONYTAIL_PLUGIN_DIR before the run can use it - #717
Open
KimHyeongRae0 wants to merge 1 commit into
Open
Selftest wipes PONYTAIL_PLUGIN_DIR before the run can use it#717KimHyeongRae0 wants to merge 1 commit into
KimHyeongRae0 wants to merge 1 commit into
Conversation
Tried reproducing the agentic benchmark on a box without the plugin installed, set PONYTAIL_PLUGIN_DIR the way the README says, and every ponytail cell died with "plugin dir not found". _selftest_plugin_dir() borrows the var for its sentinel check and then deletes it outright in the finally block. The selftest runs ahead of every real run, so the override never survives long enough to be used. Save the old value, put it back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was re-running
benchmarks/agenticto check the numbers inresults/2026-06-18-agentic.md— different machine, plugin not installed there, so I setPONYTAIL_PLUGIN_DIRthe waybenchmarks/agentic/README.mdsuggests. Baseline cells ran fine, then the first ponytail cell died:Which was confusing, because I had.
_selftest_plugin_dir()sets the var to a sentinel to prove the override is honored, then unconditionallydels it in thefinally. The selftest runs ahead of every real run, so by the time_plugin_dir()is called for an actual cell the variable is gone and it falls through to the cache-dir branch.Bit of an irony: the docstring on
_plugin_dirsays hardcoding one machine's path "made the ponytail/caveman arms unreproducible off that box" — and the env override that solves that is the thing getting cleared.Fix saves the prior value and restores it. With
PONYTAIL_PLUGIN_DIR=/tmp/x:python run.py --selfteststill reports all instruments valid, and the run I was after finishes now.For what it's worth, the reproduction held up well — median
src_locon datepicker came out 401 baseline / 22.5 ponytail against your published 404 / 23, two months later on a different box.