Description
When using sitemcp to save Japanese language pages, the content becomes garbled (mojibake). This occurs because the tools always treat fetched HTML and text content as UTF-8, while many Japanese websites are still served using legacy encodings such as Shift_JIS or EUC-JP.
Current Behavior
- The tools process all content as UTF-8 regardless of the actual encoding
- Character encoding specified in server response headers or HTML meta tags (Shift_JIS, EUC-JP, etc.) is not automatically detected
- The underlying libraries (Go's
net/http + readability) default to UTF-8 processing
Expected Behavior
The tools should:
- Automatically detect the character encoding from HTTP headers or HTML meta tags
- Convert non-UTF-8 content to UTF-8 before saving
- Provide an option to specify the expected encoding manually
Proposed Solutions
Add character encoding detection and conversion capabilities to the tools. This could include:
- Implementing charset detection from Content-Type headers
- Parsing HTML meta charset declarations
- Adding command-line options for manual encoding specification (e.g.,
--encoding=SHIFT_JIS)
Impact
This issue significantly affects users working with Japanese websites and potentially other non-UTF-8 encoded content. The lack of automatic encoding detection makes the tools less useful for international content.
Environment
- Common problematic encodings: Shift_JIS, EUC-JP
- Platform: All platforms
Additional Context
Many legacy Japanese websites still use Shift_JIS or EUC-JP encoding for historical reasons. While UTF-8 adoption is increasing, proper encoding handling remains crucial for web scraping and archiving tools targeting Japanese content.
Description
When using
sitemcpto save Japanese language pages, the content becomes garbled (mojibake). This occurs because the tools always treat fetched HTML and text content as UTF-8, while many Japanese websites are still served using legacy encodings such as Shift_JIS or EUC-JP.Current Behavior
net/http+readability) default to UTF-8 processingExpected Behavior
The tools should:
Proposed Solutions
Add character encoding detection and conversion capabilities to the tools. This could include:
--encoding=SHIFT_JIS)Impact
This issue significantly affects users working with Japanese websites and potentially other non-UTF-8 encoded content. The lack of automatic encoding detection makes the tools less useful for international content.
Environment
Additional Context
Many legacy Japanese websites still use Shift_JIS or EUC-JP encoding for historical reasons. While UTF-8 adoption is increasing, proper encoding handling remains crucial for web scraping and archiving tools targeting Japanese content.