/ToUnicode CMap with 2-byte source codes is wrongly applied to simple fonts (1-byte codes)

[Affenlocke]

Summary

For a simple font (/Type1, /TrueType, /MMType1) character codes are always
one byte. If such a font carries a /ToUnicode CMap whose bfchar/bfrange
entries are registered with two-byte source codes, that CMap does not apply to
the font's codes and should be ignored.

pypdf applies it anyway, because map_dict is keyed by the decoded string rather
than by the source bytes: <0044> and <44> both collapse to the key "D".
The result is silently mangled text — no warning, no exception.

Environment

• pypdf 6.10.2 (also present in main as of 2026-08-31)
• Python 3.13.12
• Windows 11

Code + PDF

Two minimal PDFs are attached, both generated by make_repro.py (also attached).
Each draws the text pypdf-cmap using standard-14 /Helvetica with
/WinAnsiEncoding, plus a /ToUnicode CMap that maps every used code to X.
The only difference is the source-code length in the CMap.

File	/ToUnicode source codes	CMap applicable?
simple_font_2byte_tounicode.pdf	<0070> <0058> (2 bytes)	no — font uses 1-byte codes
simple_font_1byte_tounicode.pdf	<70> <0058> (1 byte)	yes

from pypdf import PdfReader

for name in ("simple_font_2byte_tounicode.pdf", "simple_font_1byte_tounicode.pdf"):
    print(name, repr(PdfReader(name).pages[0].extract_text()))

Actual

simple_font_2byte_tounicode.pdf 'XXXXXXXXXX'
simple_font_1byte_tounicode.pdf 'XXXXXXXXXX'

Expected

simple_font_2byte_tounicode.pdf 'pypdf-cmap'     <-- CMap does not apply, fall back to /Encoding
simple_font_1byte_tounicode.pdf 'XXXXXXXXXX'     <-- CMap applies, unchanged

Apache PDFBox 3.0.8 produces exactly the expected output for both files:

$ java -jar pdfbox-app-3.0.8.jar export:text -i simple_font_2byte_tounicode.pdf -o out.txt
$ cat out.txt
pypdf-cmap
$ java -jar pdfbox-app-3.0.8.jar export:text -i simple_font_1byte_tounicode.pdf -o out.txt
$ cat out.txt
XXXXXXXXXX

The second file is the important control: it must keep working, so "ignore
/ToUnicode when the text looks wrong" is not an acceptable fix.

Root cause

pypdf/_cmap.py, parse_bfchar (and analogously parse_bfrange):

map_dict[-1] = len(lst[0]) // 2          # source code length is computed ...
...
map_dict[
    unhexlify(lst[0]).decode(
        "charmap" if map_dict[-1] == 1 else "utf-16-be", "surrogatepass"
    )
] = map_to                                # ... but the key is the decoded string

unhexlify(b"0044").decode("utf-16-be") yields "D", and the one-byte code
<44> decoded via "charmap" yields "D" as well. Both end up under the same
key, so the byte length stops being part of the lookup. It is retained in
map_dict[-1], but that value is never consulted when the map is applied to a
simple font's one-byte codes.

How PDFBox handles it

PDFBox keeps the mappings in three separate maps keyed by source length
(fontbox/.../cmap/CMap.java):

private final Map<Integer, String> charToUnicodeOneByte  = new HashMap<>();
private final Map<Integer, String> charToUnicodeTwoBytes = new HashMap<>();
private final Map<Integer, String> charToUnicodeMoreBytes = new HashMap<>();

and looks up with an explicit length for simple fonts
(pdfbox/.../font/PDFont.java):

if (code < 256 && !(this instanceof PDType0Font))
{
    COSName encoding = dict.getCOSName(COSName.ENCODING);
    if (encoding != null && !encoding.getName().startsWith("Identity"))
    {
        return toUnicodeCMap.toUnicode(code, 1);
    }
}
return toUnicodeCMap.toUnicode(code);

A two-byte entry therefore never answers a one-byte query. PDSimpleFont.toUnicode
then falls back to /Encoding → glyph name → Adobe Glyph List, which is the same
path the renderer uses to draw the glyph — hence the correct result.

Suggested fix

Make the source-code length part of the lookup. A minimal variant: when the font
subtype is /Type1, /TrueType or /MMType1, only honour /ToUnicode entries
whose source code length is 1, and otherwise fall through to the /Encoding
handling that already exists. The required length is already available as
map_dict[-1].

I am happy to open a PR if you agree with the direction.

make_repro.py
simple_font_1byte_tounicode.pdf
simple_font_2byte_tounicode.pdf

[stefan6419846]

Thanks for the report. In the future, please use our issue templates and follow our contribution guidelines.

I do not have enough knowledge of the corresponding aspects, but I am open to merging a PR if it does not have any side effects.

[PJBrs]

I wonder what the correct approach is here. The bug report basically suggests that a 2-byte ToUnicode CMap is illegal for a (1-byte) simple font. I decided to try and get Google Gemini's view on this, and it suggested that such a ToUnicode CMap would be fine, and if at decoding time we want to use a 2-byte ToUnicode CMap to decode a 1 byte code, we just need to 00-pad it first. I'll also note that Okular just extracts XXXXXXXXXX for both documents.

[stefan6419846]

I wonder what the correct approach is here.

Thanks for chiming in - you clearly did much more related work there lately. From my side, a fix without side effects does not hurt, but in this case, I have been unsure as well and wanted to wait for a PR to possibly check for further aspects.

I'll also note that Okular just extracts XXXXXXXXXX for both documents.

This is what pdftotext does as well, but as both rely on poppler in the background as far as I am aware, this is not surprising. For comparison, I also threw mutool convert at it as well as pdf.js, which both return XXXXXXXXXX for both documents as well.

I decided to try and get Google Gemini's view on this

From the structure of the initial report, it seems like it is mostly generated by an LLM, although undeclared, so we possibly have two LLMs battling each other here.

As the examples provided in the report are clearly artificial and not actual real documents found in the wild, I am unsure whether this is a hypothetical finding or an actual issue with different PDF libraries.

[stefan6419846]

For reference, these are the corresponding PDFBox reports and changes:

• https://issues.apache.org/jira/browse/PDFBOX-4749
• apache/pdfbox@6759654
• apache/pdfbox@b303e02
• apache/pdfbox@8707401

[PJBrs]

I've just asked a third LLM (the Github Copilot). Here's the full conversation. Let me know if this clobbers the bug report, then I'll remove again. The short of it: padding extract bytes before looking them up in Font.character_map (i.e., map_dict) would seem a more robust approach than just invalidating a 2-byte ToUnicode CMap that doesn't align with our 1-byte expectations.

The Issue
Looking at line 412-416, the key of map_dict is set to:

Python
map_dict[
    unhexlify(lst[0]).decode(
        "charmap" if map_dict[-1] == 1 else "utf-16-be", "surrogatepass"
    )
] = map_to

The problem you've identified: map_dict[-1] stores the byte length of the source codes (set at line 396 or 359). For a simple font, this should theoretically be 1 byte, but the code uses:

"charmap" encoding when map_dict[-1] == 1 (1-byte source)
"utf-16-be" encoding when map_dict[-1] > 1 (multi-byte source)

However, a CMap can legally contain mixed byte lengths. In parse_bfrange (lines 327-389), the byte length is determined per-line from the hex tokens:

Python

nbi = max(len(lst[0]), len(lst[1]))
map_dict[-1] = (nbi + 1) // 2  # line 359

This gets overwritten for each range/char block parsed. So the last byte length encountered is what gets stored in map_dict[-1].

PDF Specification Perspective
According to PDF Reference 1.7, Section 5.8.4 (Character Mappings):

Simple fonts (non-composite) use 1-byte character codes (0-255)
Composite fonts (Type 0) use 1 or 2-byte character codes (typically 2-byte via CID)

The specification explicitly states that beginbfchar and beginbfrange map source character codes to Unicode sequences. For a simple font, those source codes should always be 1 byte.

The Bug
This code has a potential bug: If a PDF declares a simple font but provides a ToUnicode CMap with 2-byte source codes, the code will:

• Set map_dict[-1] = 2 (from the 2-byte hex token)
• Decode the key using "utf-16-be" encoding
• Store a 2-byte Unicode string as the dictionary key

This violates the PDF spec expectation that simple fonts only use 1-byte character codes. The stored keys would be inaccessible when the font's actual character codes are 1 byte.

Recommendation
This appears to be a defensive design to handle malformed PDFs, but it should include:

• Validation that map_dict[-1] matches the font type (1 byte for simple fonts, 2 for composite)
• Logging when this mismatch is detected
• Possibly clamping to the expected byte length for the font subtype

The current code silently accepts spec-violating mappings, which could cause character encoding failures downstream.

My follow-up question:
"The byte length mismatch only becomes a problem during text extraction. What about instead padding extracted bytes before looking them up in map_dict?"

Answer:
That's an insightful observation. You're right that the mismatch is only problematic when text is extracted and those bytes need to be looked up in map_dict.

[...]

Padding extracted bytes before lookup is a much cleaner approach than validating during parsing. Here's why it makes sense:

Current Problem
Currently, text extraction fails silently when byte lengths don't match. The lookup at line 237 in _text_extraction/init.py:

Python
x = font.character_map.get(raw_character, raw_character)
If raw_character comes in as a 1-byte encoded string (e.g., \x42) but the character_map keys are 2-byte encoded (e.g., \x00B), the lookup fails and returns the default raw_character unchanged.

Padding Solution Benefits
Padding extracted bytes at lookup time would:

• Solve the mismatch transparently — Convert all extracted characters to match the key format in character_map
• Avoid complexity during parsing — No need to validate or reject spec-violating ToUnicode streams
• Handle PDFs gracefully — Works with malformed PDFs that mix byte lengths
• Minimal performance impact — Padding is cheap (a few byte operations per character)
• Implementation Strategy

The padding should happen in the text extraction pipeline. There are two candidate locations:

Option 1: In get_display_str() (line 237)
Python

# Store the expected key byte length from character_map
key_byte_length = map_dict.get(-1, 1)  # Fallback to 1 if not set

for raw_character in text_operands:
    # Pad raw_character to match expected key length
    if key_byte_length > 1 and len(raw_character) < key_byte_length:
        padded_char = '\x00' * (key_byte_length - len(raw_character)) + raw_character
    else:
        padded_char = raw_character
    
    x = font.character_map.get(padded_char, raw_character)

Option 2: Better — Create a lookup wrapper
Store map_dict[-1] in the Font object and create a helper method:

Python

class Font:
    character_map_key_length: int = 1  # New attribute
    
    def get_character(self, raw_char: str) -> str:
        """Lookup character with automatic byte-length padding."""
        if self.character_map_key_length > 1 and len(raw_char) < self.character_map_key_length:
            padded = '\x00' * (self.character_map_key_length - len(raw_char)) + raw_char
        else:
            padded = raw_char
        return self.character_map.get(padded, raw_char)

Then update line 237:

Python
x = font.get_character(raw_character)

Key Design Points

• Preserve map_dict[-1] through Font initialization — Pass it from get_encoding() → Font.from_font_resource()
• Left-pad with null bytes — This matches the PDF convention (big-endian byte order)
• Fallback gracefully — If padding fails or key still doesn't exist, return the original character
• No changes to CMap parsing — Accept all spec-violating mappings as-is; let lookup handle mismatches

This approach is robust, performant, and spec-compliant (it handles both well-formed and malformed PDFs).

*** EDIT ***
An actual PDF that exhibits problematic extraction would be much more helpful than an artificial example.