Lockfile strategy for nightly libraries that might disappear from a pypi index

[mroeschke]

Summarizing the issue from pandas-dev/pandas#65690 (comment):

We generate a lock file that specifies a nightly version of pyarrow like 25.0.0.dev59. This nightly version auto-increments number after dev and the nightly pypi index eventually evicts older pyarrow versions.

The issue then is that our lock file will eventually reference a nightly wheel version that no longer exists. Of course updating the lock file regularly is a solution, but I am curious if there are better, alternative solutions to this with a lockfile workflow. In theory we want any nightly wheel version that exists on this index.

This workflow is run in Github Actions, so I don't think caching a nightly wheel longer term is viable across all the jobs from all the pull requests that are run.

[UnknownHawkins]

The simplest way to avoid a stale nightly wheel is to don’t lock the nightly package.
In pixi.toml keep the nightly entry under pypi-dependencies (so the index is still used) but add the no-lock = true flag for that feature. Then the lockfile will contain everything else, while the nightly wheel is always resolved at install time.

[feature.pyarrow-nightly.pypi-dependencies]
pyarrow = { version = "*", index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple", no-lock = true }

When the CI runs you can keep pixi install --locked for the rest of the environment; the nightly dependency will be fetched fresh each run, so any evicted version is automatically replaced by the newest one available on the nightly index. No extra caching or frequent lock‑file regeneration is required.

[ShreeChaturvedi]

For that CI job, skip the setup-pixi install (run-install: false) and re-solve pyarrow:

pixi update -e pyarrow-nightly pyarrow

If setup-pixi installs first, you still 404 on the pinned wheel. Don't commit the lock from CI. Keep --locked on the other envs.

no-lock on a pypi-dependency isn't a real field. If you want --locked on that job too, bump update-pixi-lock.yml to daily, or pin a url to a wheel you host.