pdf-odin-llm/ARCHITECTURE.md

pdf-odin — architecture and contributor contract

A self-contained PDF reader and renderer in pure Odin. No C dependencies: everything outside of core: is written here.

Pipeline

bytes
  │  lexer.odin, parser.odin          COS syntax  →  Object
  │  xref.odin, objstm.odin           file structure, object streams
  │  crypt.odin                       decryption (transparent, at parse time)
  ▼
Document  ── document.odin, pagetree.odin ─────────────────────────────────┐
  │  filters.odin + filter_*.odin     stream decoding                      │
  ▼                                                                        │
Page      ── page.odin                                                     │
  │  content.odin                     content-stream operators             │
  │  gstate.odin                      graphics state stack                 │
  │  colorspace.odin, function.odin, shading.odin, pattern.odin            │
  │  font*.odin, cmap.odin            glyph outlines and widths            │
  ▼                                                                        │
Device (vtable)  ── device.odin ───────────────────────────────────────────┘
  ├── device_raster.odin   → Bitmap (RGBA8)      path.odin, raster.odin,
  ├── device_text.odin     → extracted text      stroke.odin, blend.odin
  └── device_list.odin     → display list         image.odin

Everything above Device is format knowledge; everything below is pixels. The three devices share one interpreter, which is why text extraction agrees with what is drawn instead of being a second, subtly different parse.

File ownership

One concern per file. Do not add procedures to a file you do not own.

File	Contents
pdf.odin	package doc, version constants
error.odin	Error, Repair, error_string
object.odin	Object, Dict, Array, Stream, accessors
geom.odin	Point, Matrix, Rect
lexer.odin	COS tokeniser
parser.odin	token stream → Object
xref.odin	xref tables, xref streams, trailer chain, rebuild-by-scan
objstm.odin	/Type /ObjStm compressed object streams
document.odin	Document, open_*, close, resolve, object cache
pagetree.odin	page tree walk, inherited attributes, Page
crypt.odin	standard security handler, RC4, AES-CBC, key derivation
filters.odin	filter chain dispatch, stream_data, predictors
filter_flate.odin	tolerant inflate
filter_lzw.odin	LZW
filter_ascii.odin	ASCIIHex, ASCII85
filter_runlength.odin	RunLength
filter_ccitt.odin	CCITT G3/G4 fax
filter_dct.odin	JPEG: baseline + progressive, grey/YCbCr/CMYK/YCCK
encoding.odin	text strings, PDFDocEncoding, dates, name/number trees
function.odin	PDF functions type 0/2/3/4
colorspace.odin	all colour spaces → RGB
path.odin	Path, construction, transform, flatten
raster.odin	anti-aliased scanline fill
stroke.odin	stroke → fill path (joins, caps, dashes)
bitmap.odin	Bitmap: allocation, rows, premultiplication, RGBA conversion
blend.odin	blend modes, compositing, soft masks
image.odin	image XObjects → samples, image masks, /SMask
shading.odin	shading types 1–7
pattern.odin	tiling and shading patterns
cmap.odin	CMap parsing, /ToUnicode, predefined CMaps
font.odin	font loading and dispatch, widths, encodings
font_cff.odin	CFF / Type2 charstrings
font_truetype.odin	glyf/loca/cmap/hmtx/post
font_type1.odin	Type1, eexec, Type1 charstrings
font_standard.odin	standard-14 metrics, built-in encodings
content.odin	content-stream lexer and operator dispatch, Type3 glyph execution
gstate.odin	graphics state and its stack
device.odin	the Device vtable
device_raster.odin	rasterising device
device_text.odin	text-extraction device
device_list.odin	display-list device
page.odin	render_page, extract_text, public page API
annot.odin	annotation appearance streams
outline.odin	document outline, destinations

Conventions

• 4 spaces, no tabs. Types Pascal_Snake (Xref_Entry, Graphics_State) except the handful of short public nouns already fixed (Object, Dict, Page, Matrix). Procedures and variables snake_case. Enum members Pascal_Snake.
• Procedures are prefixed by the noun they act on: path_transform, font_glyph, dict_get. This keeps a flat 40-file package navigable.
• @(private) anything that is not part of the public API or used across files. The public API is exactly what the package doc comment at the top of pdf.odin names, plus the types those procedures take and return: open_file, open_bytes, Open_Options, close, page_count, page_get, Page, the object accessors (resolve, get_object, get_dict, get_stream, dict_*, array_*, to_*, stream_data), Error, Repair, error_string, the encryption accessors, and -- once the device layer exists -- render_page, extract_text and Bitmap. Everything else is @(private), tests included: a test lives in the same package and reaches private names fine, so being called by one is not a reason to export. Widening the public API means editing that doc comment in the same commit, which is what keeps the two in step.
• Comments explain why, especially why a tolerance exists: "a /Length that disagrees with the bytes is common enough that …". Do not narrate what the next line does.
• Cite the spec by clause number when the behaviour is non-obvious: // ISO 32000-2, 7.5.8.2.

Memory

• A Document owns a growing arena (core:mem/virtual). Every object, dictionary, stream and decoded buffer produced while reading lives there and is reclaimed in one call by close. Allocate with doc.allocator.
• Anything handed to a caller that is not owned by the Document takes an explicit allocator := context.allocator parameter and says so in its doc comment. That is render_page's Bitmap, extract_text's string, and nothing else.
• Per-page rendering scratch goes in a temporary arena that is reset between pages, never in the Document arena — a 900-page render must not grow without bound.
• Never call free or delete on Document-arena memory.

Robustness — the rule that matters most

Malformed PDFs are the normal case, not the exception. A reader that rejects a file every other viewer opens is a broken reader.

1. Never panic. No unchecked slicing, no assert on file-derived data, no unwrapped or_else on parsed values. Bounds-check every index that came from the file.
2. Never loop forever. Every loop driven by file data needs a bound: a recursion depth limit (MAX_DEPTH), an iteration fuel counter, or a visited-set. Cyclic /Parent chains, self-referencing object streams and xref /Prev loops all occur in real files.
3. Degrade, don't fail. A broken font renders as blanks; a broken image is skipped; an unparsable content fragment is stepped over to the next operator. Record a Repair on the Document and carry on. Only return an Error when there is genuinely nothing to hand back.
4. Clamp allocations. A /Length, /W, /Height or /Size from a file is attacker-controlled. Check it against the remaining file size before allocating, and against MAX_* limits.
5. No NaN or infinity below the Device line. Numbers coming out of a file are checked with is_finite at the boundary.

Tests — write them with the code, not after

Every file foo.odin gets a foo_test.odin in the same package, using core:testing. A layer is not finished until its tests pass.

• Unit tests are self-contained: byte literals in, expected values out. Codec tests use vectors from the format's own specification where one exists.
• Golden PDFs live in testdata/ and are tiny, hand-written, and committed. They cover the structural cases: classic xref, xref stream, object stream, incremental update, broken xref needing a rebuild, wrong /Length, each encryption revision.
• Corpus tests run over whatever is in the directory named by the PDF_ODIN_CORPUS environment variable and skip silently when it is unset, so they never depend on files that are not in the repository. They assert only invariants: opens without error, page count > 0, every page renders without crashing, no leaks.
• Fuzz-shaped tests: truncate and bit-flip the golden files at many offsets and assert the library returns an error rather than crashing or hanging.

Run scripts/check.sh (type-check + vet) and scripts/test.sh before declaring anything done.

Limits

Defined in document.odin, referenced everywhere:

MAX_DEPTH            32     // nested arrays/dicts, form XObject recursion
MAX_OBJECT_NUMBER    8388607
MAX_XREF_CHAIN       64     // /Prev hops
MAX_CONTENT_OPERANDS 32
MAX_BITMAP_PIXELS    (1 << 28)

Status: what is written and what remains

The library renders. Every file in the table above is written, tested and driven over a real corpus. open_file -> page_get -> render_page produces a correct picture of a real page, extract_text produces its text off the same interpreter, and page_display_list records it for a caller with a backend of their own. There is no skeleton left.

Complete: pdf error object geom document lexer parser xref objstm pagetree crypt encoding filters filter_flate filter_lzw filter_ascii filter_runlength filter_ccitt filter_dct path bitmap gstate device raster stroke blend colorspace function cmap font_standard font_cff font_truetype font_type1 font image shading pattern content device_raster device_text device_list page annot outline.

Type3 was planned as font_type3.odin and does not need a file: a Type3 glyph is a content stream, so loading it belongs to font.odin with the other font dictionaries (font_load_type3, font_type3_proc) and running it belongs to the interpreter that runs every other content stream (content_run_type3, content_type3_d1, and the Type3_Glyph flag that makes d0/d1 legal and stops a glyph that shows itself from recursing). Splitting it out would have meant a file that could only call back into content.odin. It is finished and proven end to end: a Type3 font whose glyph is 0 0 750 750 re f at 40 pt rasterises to exactly the three 30-point squares its /FontMatrix and /Widths call for.

What remains

Nothing structural. What is left is one real gap, two deliberate approximations, and the review debt from the earlier batches.

Non-embedded fonts draw nothing. This is the one visible defect in the whole layer and the render sweep below found it. A font dictionary with no /FontFile* still loads, still lays out and still extracts text correctly -- that is what the render-layer contract asks for and it works -- but font_glyph_path has no outline to return, so the text is invisible. The library ships no font programs of its own and reads none from the system, so there is nothing to substitute with: fixing it means either bundling a metric-compatible face for the standard 14 or adding a system-font lookup, and both are new subsystems rather than a repair. Fifty-nine of the corpus's 894 fonts are unembedded, and they are why fifteen corpus pages come back blank or nearly blank.

Two transparency approximations, both stated at the top of device_raster.odin where they are made: a knockout group is rendered as a non-knockout one, and a non-isolated group whose own alpha is not 1 or whose own blend mode is not Normal is rendered isolated. Both are the approximations every renderer without a backdrop-removal pass makes, and docs/corpus-notes.md ranks the cases they differ on as rare.

Still owed from the earlier batches, unchanged: the two adversarial review lenses -- visibly-wrong pixels, and the data tables audited line by line against the AFMs and the AGL. Every table reached by binary search (STD_TEXT_GLYPHS, STD_SYMBOL_GLYPHS, STD_ZAPF_GLYPHS, AGL_TABLE, AGL_BY_RUNE) has been verified sorted and in range, and spot values across all five width tables and both AGL directions match their source documents, but that is a smoke test, not the audit.

What the integration checks have covered

Three batches were built in parallel and each has since been integrated. The first was the six rendering primitives — raster stroke blend colorspace function cmap font_standard; the second was the seven files above them — font font_cff font_truetype font_type1 image shading pattern; the third was the render layer itself — content device_raster device_text device_list page annot outline.

From the first batch:

• The layer composes. path_flatten -> raster_fill -> blend_span and stroke_flatten -> raster_fill -> blend_span were driven end to end into a Bitmap and checked pixel by pixel against the geometry: aligned and half-covered rectangles, both fill rules, butt and round joins and caps, a dashed line, a hollow stroked square, a hairline under a 4x CTM, and a clip not anchored at the origin. All correct, with no fringing and no seams.
• The three conventions the primitives meet on hold everywhere: coverage is 0..255 with 255 meaning fully covered; every f64 colour is straight and 0..1; every Bitmap below the Device line is premultiplied. blend.odin's header states this rule and every entry point in it obeys it.
• One composition bug, in path.odin rather than in the six: path_flatten dropped the connecting segment of a l that followed an h, because its .Line arm did not seed the new subpath with the current point the way its .Curve arm always has. Fixed, with a regression test that holds the two arms to each other.
• Two parallel-authorship seams, both fixed: object_identity (the cache key both colorspace_load and function_load use) had been defined in function.odin and reached into from colorspace.odin, and now lives in object.odin with the rest of the Object accessors; font_standard.odin carried its own hex-digit decoder alongside object.odin's hex_value.
• One internal contradiction, fixed: the bare /Pattern singleton declared n = 1 directly under a comment saying a colour in it has no components. n is what tells the interpreter how many operands scn takes, and /P1 scn takes none, so it is 0.

From the second batch:

• The seven compose with the primitives and with each other. A font dictionary with an embedded TrueType program was driven through font_load -> font_decode -> font_glyph_path -> path_flatten -> raster_fill -> blend_span and the ink checked pixel by pixel against where the glyph's em square lands under a real text matrix; an image was driven through image_load -> image_decode -> blend_span as a stencil and through blend_span_rgba8 as colour; an axial shading and the same shading behind a /PatternType 2 were painted and checked against exact pixel values computed by hand. Every one fitted first time.
• One parallel-authorship seam, fixed: image.odin carried its own img_byte — the 0..1-to-byte quantiser, NaN handling and half-up rounding and all — beside blend.odin's blend_quantize, which is the same procedure and which shading.odin, written by a different author against the same contract, had correctly reused. image.odin now calls blend_quantize too. Quantising a colour to a byte is blend.odin's to own: it is the file whose header states the byte conventions the whole device layer meets on.
• The per-format byte readers (cff_u16, tt_u16, mesh_bits, img_sample) were checked against each other and left alone. They look alike and are not: each font-program reader is required to be self-contained, and the two sample unpackers differ in kind — one is indexed and capped at 16 bits, the other is a sequential bitstream reader up to 32.
• corpus_test.odin grew test_corpus_resources, which loads and drives every font, image, shading and pattern named by every page of every corpus file, recursing into form XObjects and tiling-pattern resources. Over 71 documents it reaches 894 distinct fonts and 1579 images, probes 123 000 glyph outlines and decodes 372 000 character codes, and found no failure of any kind. That is the first time the library met real subsetted font programs at scale, and the invariants it holds them to are the ones the interpreter will depend on: a Font always comes back, an embedded program either has glyphs or sets program_failed, every outline is finite, font_decode consumes exactly the bytes it is given, and an image either decodes to a bitmap of its own dimensions or degrades to an error with no bitmap at all.
• Three things the sweep taught, all recorded here because they will otherwise be rediscovered as bugs: a subsetter leaves maxp.numGlyphs at the original count while emptying loca, so a font can declare 22 026 glyphs and hold 25 — sampling glyph ids rather than scanning them will find none of them; a Type1C subset is routinely stripped to .notdef and space, both of which are a single endchar byte, so "no glyph in this font draws" is a normal file and not a defect; and a font with no embedded program sets neither a glyph count nor program_failed, because nothing failed — the standard-14 metrics are the whole of what the file gave.
• 296 of the 976 tests belong to these seven files. None are vacuous: the ones that appear to assert little all go through a local helper that asserts, and the two that end in expect(t, true) are the truncation and bit-flip fuzzers, where reaching the last line is the property under test — the same spelling xref_test.odin and objstm_test.odin have used since the COS layer.

From the third batch — the render layer, and the first time anything in this repository produced a picture:

• The seven files fitted together with no conflicts to resolve. Nothing had to be moved between files, no procedure was duplicated across two authors, and no frozen type had to change: check.sh and the 1294-test suite were both green on the first run after the seven landed. That is what freezing Device, Bitmap, Render_Options and the Span_Proc contract ahead of the code bought.
• The render sweep (test_corpus_render in corpus_test.odin) is the check this layer exists for and the most informative thing run on the project so far. It renders the first page of every corpus file — every page of every file under twenty — at 72 dpi, holds each render to page_size's dimensions, a 4-component straight-alpha bitmap and a time budget, and then measures the fraction of non-background pixels. Over 71 documents and 315 pages: zero errors, zero leaks, mean ink 18.3% (Downloads) and 7.1% (the ISO corpus), slowest page 1.4 s, no page's dimensions disagreeing with page_size.
• Blankness is the invariant that finds real bugs, and it found one. A page that draws nothing is invisible to every other check: it parses, loads every font, decodes every image, and returns .None. So the sweep also asks the text device what the interpreter saw on any page under 1% ink, and reports three different things where a bare ink number reports one: a page that draws nothing and extracts nothing is an empty sheet, a page that draws nothing and extracts 3 000 characters is a defect, and a page with a paragraph on it under 1% ink is the partial version of the same defect. Of the 19 light pages in the two corpora, 6 are genuinely blank paper and 14 are the non-embedded font gap. Not one is an interpreter, clip or base-matrix bug — which was the thing most likely to be wrong and is the reason the sweep was written.
• The diagnosis is unambiguous on Videcom/VRS XML Service 2020 02FEB2023.pdf page 1, the clearest case: the embedded Courier New draws its code blocks perfectly, every heading rule and the hyperlink rule land exactly where they belong — and every word set in Arial or Times New Roman, neither of which the file embeds, is missing. The geometry, the clipping, the colour and the stroking are all correct; the outlines simply are not in the file and are not in this library either.
• Text extraction (test_corpus_text) runs the same interpreter into the text device over every page of every file: 3 066 pages, 5.5 MB, 5.37 M runes. The ISO corpus extracts text from 100% of its 1 234 pages with zero U+FFFD. The Downloads corpus yields text on 85% of pages at 0.55% U+FFFD, and every one of the 21 pages above 10% U+FFFD is in a single Arabic-language ITU recommendation whose subset fonts name their glyphs by contextual presentation form — lamwithaleffinal, hahwithmeeminitial — which is in neither the AGL nor the file's own /ToUnicode. Those glyphs render correctly; there is simply nothing in the file that says what they mean, so U+FFFD is the honest answer and not a defect. The unicode chain is otherwise clean across the whole corpus.
• examples/pdfrender and examples/pdftext were added and are compiled by check.sh alongside pdfinfo, which is what keeps render_page, extract_text, Bitmap, bitmap_row and scale_matrix genuinely public rather than accidentally reachable only from inside the package.
• 307 of the 1294 tests belong to these seven files. None are vacuous: the 21 that carry no testing.expect of their own all go through a local helper that asserts (rt_expect, pt_expect_point, ol_expect_label), the same spelling every batch before them used.

docs/corpus-notes.md records what the real-world corpus actually uses, which is what the ordering above is based on; docs/build-plan.md is the plan.