Files
pdf-odin-llm/docs/corpus-notes.md
T
2026-09-25 23:32:29 +02:00

1.9 KiB

What real PDFs actually use

Feature counts across the 51 PDFs on this machine (39 in ~/Downloads, 12 ISO spec documents) — contracts, invoices, standards, scans, exports from Word, InDesign and web tooling. Percentages are of files that contain the marker at all.

Feature Files Note
FlateDecode 51 (100%) the only filter that must be fast
DCTDecode 19 (37%) own JPEG decoder is not optional
ASCII85Decode 3 (6%)
CCITTFaxDecode 1 (2%) long tail
LZWDecode 1 (2%) long tail
JPXDecode / JBIG2Decode 0 never seen; pass through
xref streams + object streams 26 (51%) half the corpus is PDF 1.5+ structure
Linearized 13 (25%)
Encrypted 3 (6%)
AcroForm 13 (25%)
Annots 27 (53%) appearance streams matter for rendering
ExtGState 44 (86%)
SMask 30 (59%) transparency is mainstream, not exotic
Group (transparency) 25 (49%)
BM (blend mode) 23 (45%)
Type0 (composite) fonts 27 (53%)
CIDFontType2 29 (57%)
FontFile2 (TrueType) 31 (61%)
FontFile3 (CFF) 11 (22%)
ToUnicode 32 (63%) text extraction has a real chance
Type3 fonts 1 (2%) long tail
ICCBased 12 (24%) render via /Alternate
Indexed 8 (16%)
Separation 3 (6%)
Lab 3 (6%)
Pattern 7 (14%)
Shading 3 (6%)
DeviceN, CalRGB 0 implement, but do not tune for

Reproduce with the loop in the project history; it is a plain grep for each marker across every PDF, which works because stream dictionaries are never themselves inside an object stream.

What this ordered: Flate and DCT decode quality, TrueType and CFF glyph outlines, Type0/CID mapping, /ToUnicode, and the transparency model (soft masks, groups, blend modes) are the load-bearing work. CCITT, LZW, JPX, JBIG2, Type3, DeviceN and the exotic shading types are correctness obligations, not performance ones.