[gatsby-source-contentful] One failed asset download strands its siblings and makes gatsby develop panic with "nodes appear to be being changed every time we run queries" #39645

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

When gatsby-source-contentful runs with downloadLocal: true, a single failed asset download makes gatsby develop die with an unrelated-looking panic:

Panicking because nodes appear to be being changed every time we run queries. This would cause the site to recompile infinitely.
Check custom resolvers to see if they are unconditionally creating or mutating nodes on every query.
This may happen if they create nodes with a field that is different every time, such as a timestamp or unique id.

There are no custom resolvers involved, so the advice in the message leads nowhere. The real cause is in distributeWorkload in packages/gatsby-source-contentful/src/download-contentful-assets.js:

async function distributeWorkload(workers, count = 50) {
  const methods = workers.slice()

  async function task() {
    while (methods.length > 0) {
      await methods.pop()()
    }
  }

  await Promise.all(new Array(count).fill(undefined).map(() => task()))
}

There is no cancellation. When one worker rejects, Promise.all rejects immediately and the rejection escapes downloadContentfulAssets, but the other ~49 task runners keep going. downloadContentfulAssets is awaited from sourceNodes, so those survivors carry on calling createNode after the sourceNodes lifecycle has already returned and Gatsby has moved on.

In gatsby develop that lands in the runningQueries state, which handles ADD_NODE_MUTATION with markNodesDirty. The machine transitions to recreatingPages, queries run again, more downloads land, and after RECOMPILE_PANIC_LIMIT (6) rounds it panics.

The diagnostic tell is Total nodes climbing on each cycle while SitePage stays constant:

info Total nodes: 7727, SitePage nodes: 128
info Total nodes: 7875, SitePage nodes: 128
info Total nodes: 7965, SitePage nodes: 128
info Total nodes: 8014, SitePage nodes: 128
info Total nodes: 8061, SitePage nodes: 128
info Total nodes: 8134, SitePage nodes: 128

With --verbose, the growth is entirely File and ImageSharp. Every one of those File nodes is owned by gatsby-source-filesystem with an absolutePath under .cache/caches/gatsby-source-contentful/ and a url on images.ctfassets.net, i.e. Contentful assets still arriving long after sourcing finished. On a space with 752 assets they climbed 168 → 228 → 323 → 427 → 491 → 569 → 645 across the six cycles.

The failure that starts it, logged just before success source and transform nodes:

success Downloading Contentful Assets - 0.239s - 0/752
 ERROR #11321  API.NODE.EXECUTION
"gatsby-source-contentful" threw an error while running the sourceNodes lifecycle:
ENOENT: no such file or directory, lstat '<project>/.cache/caches/gatsby-source-contentful/tmp-<digest>.png'

That ENOENT looks like the temp-file race in gatsby-core-utils' fetchRemoteFile described in #34297, and it is intermittent — roughly 1 in 3 gatsby develop starts here. The propagation bug is independent of it though. Any rejecting worker produces the same outcome, which is why this is filed separately: #34297 is about why a download fails, this is about one failure taking the dev server down instead of failing cleanly.

This also looks like a plausible explanation for #34524 (downloadLocal, ~950 assets, assets intermittently missing from .cache with a "No such file or directory" error) and possibly #34616.

Note that swallowing the error is not the fix. #24288 deliberately stopped ignoring failed asset downloads, because a silently null localFile resurfaces later as a confusing null reference. The problem is only the timing: the rejection escapes before its siblings have settled. Collecting errors and re-raising once all work is done keeps the failure fatal and leaves nothing downloading past the end of sourceNodes.

Reproduction Link

https://github.com/jasperchow5915/gatsby/tree/fix/contentful-settle-asset-downloads

A minimal site reproduction is not practical here, because the trigger (#34297) is an intermittent filesystem race. The mechanism itself is deterministic, so the reproduction is a unit test against the real downloadContentfulAssets, in the branch above:

packages/gatsby-source-contentful/src/__tests__/download-contentful-assets.js → waits for every download to settle before surfacing a failure

It queues one asset that fails immediately alongside four slower ones and asserts that all four have finished by the time the rejection surfaces. On current master it fails with 0 settled; the sibling downloads are still running when downloadContentfulAssets has already rejected. That is exactly the window in which nodes get created after sourceNodes returns.

Steps to Reproduce

1. Check out the branch above and run jest packages/gatsby-source-contentful/src/__tests__/download-contentful-assets.js with the fix in src/download-contentful-assets.js reverted to master.
2. Observe the new test fail on expect(settled).toBe(slowCount) with Received: 0 — downloadContentfulAssets rejected while its other workers were still in flight.

To see the end-to-end panic on a real site, a Contentful space with several hundred assets and downloadLocal: true is needed, plus a download failure during sourcing. Any error thrown from a worker will do; #34297's ENOENT is just the one that occurs naturally.

Expected Result

downloadContentfulAssets waits for all in-flight downloads to settle before rejecting, so no createNode call happens after sourceNodes has returned. A failed download is reported and remains fatal, and gatsby develop starts normally with the error visible.

Actual Result

downloadContentfulAssets rejects as soon as the first worker fails. The remaining workers keep creating File and ImageSharp nodes during query execution, and gatsby develop panics after 6 recompile rounds with a message pointing at custom resolvers.

Environment

System:
    OS: macOS 26.5.2
  Binaries:
    Node: 18.12.1
    npm: 8.19.2
  npmPackages:
    gatsby: 5.12.4
    gatsby-source-contentful: 8.16.0
    gatsby-source-filesystem: 5.12.0
    gatsby-plugin-sharp: 5.12.0
    gatsby-transformer-sharp: 5.12.0
    gatsby-core-utils: 4.16.0 (transitive)

packages/gatsby-source-contentful/src/download-contentful-assets.js is unchanged between 8.16.0, 8.17.0 (latest) and current master, so this is not version specific.

Config Flags

None.