fix(converter): stop deleting the article's own title, byline and tags - #89
Open
Agnik47 wants to merge 1 commit into
Open
fix(converter): stop deleting the article's own title, byline and tags#89Agnik47 wants to merge 1 commit into
Agnik47 wants to merge 1 commit into
Conversation
HTMLToMarkdown removed every <header>, <footer> and <nav> on the page before extracting the main content. Semantic HTML nests two of those inside the content itself: <article><header> carries the post's <h1> and byline, and <article><footer> carries its tags and author bio. Stripping them globally deleted the title of the very page being scraped, which is the layout used by most blog, docs and news pages. Scope the header/footer removal to elements that are not inside a main-content container, reusing the same selector list extractMainContent already uses so the two stay consistent. <nav> is navigation at any depth, so it keeps being removed everywhere. Tests cover an article whose header/footer survive while the page-level ones are dropped, the body-fallback path where the main content is under the 100-character threshold, and a nav nested in an article to pin the unchanged behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
HTMLToMarkdownstrips boilerplate before extracting the main content:doc.Findmatches at any depth, so this does not just remove site chrome — it also removes the<header>and<footer>that semantic HTML nests inside the content:That is the standard layout for blog posts, docs pages and news articles (WordPress, Ghost, Hugo, MDN, most CMS themes). The result: every scrape of such a page silently loses the post's
<h1>title, its byline and its tags. The markdown comes back as body text with no heading at all — and the title is usually the single most valuable field for anyone consumingmarkdown.Reproduction
Against
master, the test added in this PR produces:The title, byline and tags are gone; only the paragraph survives.
The fix
Split the removal into two rules:
script, style, noscript, iframe, svg, nav. These never carry readable content, and a<nav>is navigation wherever it appears, so its behaviour is unchanged.header, footer. These are dropped unless they sit inside a main-content container."Main-content container" reuses the exact selector list
extractMainContentalready uses (main,article,[role='main'],#content,#main-content,.content,.main-content), lifted into a package-levelmainContentSelectorsso the two cannot drift apart. The check usesParentsFiltered, which walks ancestors only — so an element like<header id="content">still counts as chrome rather than matching itself.Behaviour only changes for
<header>/<footer>nested inside a main-content container. Pages with no such container (where the cleaner falls back to the whole<body>) strip exactly what they stripped before.Tests
Three subtests added to
TestHTMLToMarkdown:article header and footer survive chrome stripping— the article's<header>/<footer>are kept while the page-level ones are dropped. Fails onmaster.page chrome is removed even when the body fallback is used—<main>is deliberately under the 100-character threshold so cleaning falls back to the whole body, proving page chrome is already gone by then and the nested header is not. Fails onmaster.nav is removed even inside the main content— pins the deliberately unchanged<nav>behaviour. Passes both before and after.Verification
Run against Go 1.26 locally:
go build ./...— cleango vet ./...— cleango test ./...— all packages passgofmt -l— clean on both changed filesThe existing
boilerplate tags are removedandnav content is removedsubtests still pass unchanged.No open issue or PR covers
internal/converter.