Skip to content

Pubmed xml download updates - #1507

Closed
kkaris wants to merge 0 commit into
gyorilab:masterfrom
kkaris:pubmed-xml-refactor
Closed

Pubmed xml download updates#1507
kkaris wants to merge 0 commit into
gyorilab:masterfrom
kkaris:pubmed-xml-refactor

Conversation

@kkaris

@kkaris kkaris commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

This PR updates the pubmed client with the following:

  • The XML downloading process has been updated to allow more flexibility so that downstream usage can adopt the pubmed client's download process instead of implementing its own
  • After testing parallel download, it was determined that a speedup of at least 2-4x was possible with a modest amount of 4 workers, which greatly reduces the time it takes to download the full XML dump from scratch. Because of this, the possibility to run downloads in parallel is added, with a default of serial download (i.e. max_workers=1).

@kkaris

kkaris commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

I'll look into the failing tests

@kkaris

kkaris commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

All the failing tests are related to the pmc_client getting 429 errors, which seems to be related to the updated/new PMC API. Fixing that might be out of scope for this PR.

Update Jun 25:
See #1509 for proposed fixes to this.

@kkaris kkaris closed this Jun 30, 2026
@kkaris
kkaris force-pushed the pubmed-xml-refactor branch from 42a92c5 to 7ae3337 Compare June 30, 2026 16:42
@kkaris
kkaris deleted the pubmed-xml-refactor branch June 30, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant