# pdf-odin — architecture and contributor contract A self-contained PDF reader and renderer in pure Odin. No C dependencies: everything outside of `core:` is written here. ## Pipeline ``` bytes │ lexer.odin, parser.odin COS syntax → Object │ xref.odin, objstm.odin file structure, object streams │ crypt.odin decryption (transparent, at parse time) ▼ Document ── document.odin, pagetree.odin ─────────────────────────────────┐ │ filters.odin + filter_*.odin stream decoding │ ▼ │ Page ── page.odin │ │ content.odin content-stream operators │ │ gstate.odin graphics state stack │ │ colorspace.odin, function.odin, shading.odin, pattern.odin │ │ font*.odin, cmap.odin glyph outlines and widths │ ▼ │ Device (vtable) ── device.odin ───────────────────────────────────────────┘ ├── device_raster.odin → Bitmap (RGBA8) path.odin, raster.odin, ├── device_text.odin → extracted text stroke.odin, blend.odin └── device_list.odin → display list image.odin ``` Everything above `Device` is format knowledge; everything below is pixels. The three devices share one interpreter, which is why text extraction agrees with what is drawn instead of being a second, subtly different parse. ## File ownership One concern per file. Do not add procedures to a file you do not own. | File | Contents | |---|---| | `pdf.odin` | package doc, version constants | | `error.odin` | `Error`, `Repair`, `error_string` | | `object.odin` | `Object`, `Dict`, `Array`, `Stream`, accessors | | `geom.odin` | `Point`, `Matrix`, `Rect` | | `lexer.odin` | COS tokeniser | | `parser.odin` | token stream → `Object` | | `xref.odin` | xref tables, xref streams, trailer chain, rebuild-by-scan | | `objstm.odin` | `/Type /ObjStm` compressed object streams | | `document.odin` | `Document`, `open_*`, `close`, `resolve`, object cache | | `pagetree.odin` | page tree walk, inherited attributes, `Page` | | `crypt.odin` | standard security handler, RC4, AES-CBC, key derivation | | `filters.odin` | filter chain dispatch, `stream_data`, predictors | | `filter_flate.odin` | tolerant inflate | | `filter_lzw.odin` | LZW | | `filter_ascii.odin` | ASCIIHex, ASCII85 | | `filter_runlength.odin` | RunLength | | `filter_ccitt.odin` | CCITT G3/G4 fax | | `filter_dct.odin` | JPEG: baseline + progressive, grey/YCbCr/CMYK/YCCK | | `encoding.odin` | text strings, PDFDocEncoding, dates, name/number trees | | `function.odin` | PDF functions type 0/2/3/4 | | `colorspace.odin` | all colour spaces → RGB | | `path.odin` | `Path`, construction, transform, flatten | | `raster.odin` | anti-aliased scanline fill | | `stroke.odin` | stroke → fill path (joins, caps, dashes) | | `bitmap.odin` | `Bitmap`: allocation, rows, premultiplication, RGBA conversion | | `blend.odin` | blend modes, compositing, soft masks | | `image.odin` | image XObjects → samples, image masks, `/SMask` | | `shading.odin` | shading types 1–7 | | `pattern.odin` | tiling and shading patterns | | `cmap.odin` | CMap parsing, `/ToUnicode`, predefined CMaps | | `font.odin` | font loading and dispatch, widths, encodings | | `font_cff.odin` | CFF / Type2 charstrings | | `font_truetype.odin` | `glyf`/`loca`/`cmap`/`hmtx`/`post` | | `font_type1.odin` | Type1, eexec, Type1 charstrings | | `font_standard.odin` | standard-14 metrics, built-in encodings | | `content.odin` | content-stream lexer and operator dispatch, Type3 glyph execution | | `gstate.odin` | graphics state and its stack | | `device.odin` | the `Device` vtable | | `device_raster.odin` | rasterising device | | `device_text.odin` | text-extraction device | | `device_list.odin` | display-list device | | `page.odin` | `render_page`, `extract_text`, public page API | | `annot.odin` | annotation appearance streams | | `outline.odin` | document outline, destinations | ## Conventions - **4 spaces**, no tabs. Types `Pascal_Snake` (`Xref_Entry`, `Graphics_State`) except the handful of short public nouns already fixed (`Object`, `Dict`, `Page`, `Matrix`). Procedures and variables `snake_case`. Enum members `Pascal_Snake`. - Procedures are prefixed by the noun they act on: `path_transform`, `font_glyph`, `dict_get`. This keeps a flat 40-file package navigable. - `@(private)` anything that is not part of the public API or used across files. The public API is exactly what the package doc comment at the top of `pdf.odin` names, plus the types those procedures take and return: `open_file`, `open_bytes`, `Open_Options`, `close`, `page_count`, `page_get`, `Page`, the object accessors (`resolve`, `get_object`, `get_dict`, `get_stream`, `dict_*`, `array_*`, `to_*`, `stream_data`), `Error`, `Repair`, `error_string`, the encryption accessors, and -- once the device layer exists -- `render_page`, `extract_text` and `Bitmap`. Everything else is `@(private)`, tests included: a test lives in the same package and reaches private names fine, so being called by one is not a reason to export. Widening the public API means editing that doc comment in the same commit, which is what keeps the two in step. - Comments explain *why*, especially why a tolerance exists: "a `/Length` that disagrees with the bytes is common enough that …". Do not narrate what the next line does. - Cite the spec by clause number when the behaviour is non-obvious: `// ISO 32000-2, 7.5.8.2`. ## Memory - A `Document` owns a growing arena (`core:mem/virtual`). Every object, dictionary, stream and decoded buffer produced while reading lives there and is reclaimed in one call by `close`. Allocate with `doc.allocator`. - Anything handed to a caller that is *not* owned by the Document takes an explicit `allocator := context.allocator` parameter and says so in its doc comment. That is `render_page`'s `Bitmap`, `extract_text`'s string, and nothing else. - Per-page rendering scratch goes in a temporary arena that is reset between pages, never in the Document arena — a 900-page render must not grow without bound. - Never call `free` or `delete` on Document-arena memory. ## Robustness — the rule that matters most Malformed PDFs are the normal case, not the exception. A reader that rejects a file every other viewer opens is a broken reader. 1. **Never panic.** No unchecked slicing, no `assert` on file-derived data, no unwrapped `or_else` on parsed values. Bounds-check every index that came from the file. 2. **Never loop forever.** Every loop driven by file data needs a bound: a recursion depth limit (`MAX_DEPTH`), an iteration fuel counter, or a visited-set. Cyclic `/Parent` chains, self-referencing object streams and xref `/Prev` loops all occur in real files. 3. **Degrade, don't fail.** A broken font renders as blanks; a broken image is skipped; an unparsable content fragment is stepped over to the next operator. Record a `Repair` on the Document and carry on. Only return an `Error` when there is genuinely nothing to hand back. 4. **Clamp allocations.** A `/Length`, `/W`, `/Height` or `/Size` from a file is attacker-controlled. Check it against the remaining file size before allocating, and against `MAX_*` limits. 5. **No NaN or infinity below the Device line.** Numbers coming out of a file are checked with `is_finite` at the boundary. ## Tests — write them with the code, not after Every file `foo.odin` gets a `foo_test.odin` in the same package, using `core:testing`. A layer is not finished until its tests pass. - **Unit tests** are self-contained: byte literals in, expected values out. Codec tests use vectors from the format's own specification where one exists. - **Golden PDFs** live in `testdata/` and are tiny, hand-written, and committed. They cover the structural cases: classic xref, xref stream, object stream, incremental update, broken xref needing a rebuild, wrong `/Length`, each encryption revision. - **Corpus tests** run over whatever is in the directory named by the `PDF_ODIN_CORPUS` environment variable and skip silently when it is unset, so they never depend on files that are not in the repository. They assert only invariants: opens without error, page count > 0, every page renders without crashing, no leaks. - Fuzz-shaped tests: truncate and bit-flip the golden files at many offsets and assert the library returns an error rather than crashing or hanging. Run `scripts/check.sh` (type-check + vet) and `scripts/test.sh` before declaring anything done. ## Limits Defined in `document.odin`, referenced everywhere: ``` MAX_DEPTH 32 // nested arrays/dicts, form XObject recursion MAX_OBJECT_NUMBER 8388607 MAX_XREF_CHAIN 64 // /Prev hops MAX_CONTENT_OPERANDS 32 MAX_BITMAP_PIXELS (1 << 28) ``` ## Status: what is written and what remains **The library renders.** Every file in the table above is written, tested and driven over a real corpus. `open_file -> page_get -> render_page` produces a correct picture of a real page, `extract_text` produces its text off the same interpreter, and `page_display_list` records it for a caller with a backend of their own. There is no skeleton left. Complete: `pdf error object geom document lexer parser xref objstm pagetree crypt encoding filters filter_flate filter_lzw filter_ascii filter_runlength filter_ccitt filter_dct path bitmap gstate device raster stroke blend colorspace function cmap font_standard font_cff font_truetype font_type1 font image shading pattern content device_raster device_text device_list page annot outline`. Type3 was planned as `font_type3.odin` and does not need a file: a Type3 glyph is a content stream, so loading it belongs to `font.odin` with the other font dictionaries (`font_load_type3`, `font_type3_proc`) and *running* it belongs to the interpreter that runs every other content stream (`content_run_type3`, `content_type3_d1`, and the `Type3_Glyph` flag that makes `d0`/`d1` legal and stops a glyph that shows itself from recursing). Splitting it out would have meant a file that could only call back into `content.odin`. It is finished and proven end to end: a Type3 font whose glyph is `0 0 750 750 re f` at 40 pt rasterises to exactly the three 30-point squares its `/FontMatrix` and `/Widths` call for. ### What remains Nothing structural. What is left is one real gap, two deliberate approximations, and the review debt from the earlier batches. **Non-embedded fonts draw nothing.** This is the one visible defect in the whole layer and the render sweep below found it. A font dictionary with no `/FontFile*` still loads, still lays out and still extracts text correctly -- that is what the render-layer contract asks for and it works -- but `font_glyph_path` has no outline to return, so the text is invisible. The library ships no font programs of its own and reads none from the system, so there is nothing to substitute *with*: fixing it means either bundling a metric-compatible face for the standard 14 or adding a system-font lookup, and both are new subsystems rather than a repair. Fifty-nine of the corpus's 894 fonts are unembedded, and they are why fifteen corpus pages come back blank or nearly blank. **Two transparency approximations**, both stated at the top of `device_raster.odin` where they are made: a knockout group is rendered as a non-knockout one, and a non-isolated group whose own alpha is not 1 or whose own blend mode is not Normal is rendered isolated. Both are the approximations every renderer without a backdrop-removal pass makes, and `docs/corpus-notes.md` ranks the cases they differ on as rare. **Still owed from the earlier batches**, unchanged: the two adversarial review lenses -- visibly-wrong pixels, and the data tables audited line by line against the AFMs and the AGL. Every table reached by binary search (`STD_TEXT_GLYPHS`, `STD_SYMBOL_GLYPHS`, `STD_ZAPF_GLYPHS`, `AGL_TABLE`, `AGL_BY_RUNE`) has been verified sorted and in range, and spot values across all five width tables and both AGL directions match their source documents, but that is a smoke test, not the audit. ### What the integration checks have covered Three batches were built in parallel and each has since been integrated. The first was the six rendering primitives — `raster stroke blend colorspace function cmap font_standard`; the second was the seven files above them — `font font_cff font_truetype font_type1 image shading pattern`; the third was the render layer itself — `content device_raster device_text device_list page annot outline`. From the first batch: - The layer composes. `path_flatten -> raster_fill -> blend_span` and `stroke_flatten -> raster_fill -> blend_span` were driven end to end into a Bitmap and checked pixel by pixel against the geometry: aligned and half-covered rectangles, both fill rules, butt and round joins and caps, a dashed line, a hollow stroked square, a hairline under a 4x CTM, and a clip not anchored at the origin. All correct, with no fringing and no seams. - The three conventions the primitives meet on hold everywhere: coverage is 0..255 with 255 meaning fully covered; every `f64` colour is straight and 0..1; every `Bitmap` below the Device line is premultiplied. `blend.odin`'s header states this rule and every entry point in it obeys it. - One composition bug, in `path.odin` rather than in the six: `path_flatten` dropped the connecting segment of a `l` that followed an `h`, because its `.Line` arm did not seed the new subpath with the current point the way its `.Curve` arm always has. Fixed, with a regression test that holds the two arms to each other. - Two parallel-authorship seams, both fixed: `object_identity` (the cache key both `colorspace_load` and `function_load` use) had been defined in `function.odin` and reached into from `colorspace.odin`, and now lives in `object.odin` with the rest of the `Object` accessors; `font_standard.odin` carried its own hex-digit decoder alongside `object.odin`'s `hex_value`. - One internal contradiction, fixed: the bare `/Pattern` singleton declared `n = 1` directly under a comment saying a colour in it has no components. `n` is what tells the interpreter how many operands `scn` takes, and `/P1 scn` takes none, so it is 0. From the second batch: - The seven compose with the primitives and with each other. A font dictionary with an embedded TrueType program was driven through `font_load -> font_decode -> font_glyph_path -> path_flatten -> raster_fill -> blend_span` and the ink checked pixel by pixel against where the glyph's em square lands under a real text matrix; an image was driven through `image_load -> image_decode -> blend_span` as a stencil and through `blend_span_rgba8` as colour; an axial shading and the same shading behind a `/PatternType 2` were painted and checked against exact pixel values computed by hand. Every one fitted first time. - One parallel-authorship seam, fixed: `image.odin` carried its own `img_byte` — the 0..1-to-byte quantiser, NaN handling and half-up rounding and all — beside `blend.odin`'s `blend_quantize`, which is the same procedure and which `shading.odin`, written by a different author against the same contract, had correctly reused. `image.odin` now calls `blend_quantize` too. Quantising a colour to a byte is `blend.odin`'s to own: it is the file whose header states the byte conventions the whole device layer meets on. - The per-format byte readers (`cff_u16`, `tt_u16`, `mesh_bits`, `img_sample`) were checked against each other and left alone. They look alike and are not: each font-program reader is required to be self-contained, and the two sample unpackers differ in kind — one is indexed and capped at 16 bits, the other is a sequential bitstream reader up to 32. - `corpus_test.odin` grew `test_corpus_resources`, which loads and drives every font, image, shading and pattern named by every page of every corpus file, recursing into form XObjects and tiling-pattern resources. Over 71 documents it reaches 894 distinct fonts and 1579 images, probes 123 000 glyph outlines and decodes 372 000 character codes, and found no failure of any kind. That is the first time the library met real subsetted font programs at scale, and the invariants it holds them to are the ones the interpreter will depend on: a Font always comes back, an embedded program either has glyphs or sets `program_failed`, every outline is finite, `font_decode` consumes exactly the bytes it is given, and an image either decodes to a bitmap of its own dimensions or degrades to an error with no bitmap at all. - Three things the sweep taught, all recorded here because they will otherwise be rediscovered as bugs: a subsetter leaves `maxp.numGlyphs` at the original count while emptying `loca`, so a font can declare 22 026 glyphs and hold 25 — sampling glyph ids rather than scanning them will find none of them; a Type1C subset is routinely stripped to `.notdef` and `space`, both of which are a single `endchar` byte, so "no glyph in this font draws" is a normal file and not a defect; and a font with no embedded program sets neither a glyph count nor `program_failed`, because nothing failed — the standard-14 metrics are the whole of what the file gave. - 296 of the 976 tests belong to these seven files. None are vacuous: the ones that appear to assert little all go through a local helper that asserts, and the two that end in `expect(t, true)` are the truncation and bit-flip fuzzers, where reaching the last line is the property under test — the same spelling `xref_test.odin` and `objstm_test.odin` have used since the COS layer. From the third batch — the render layer, and the first time anything in this repository produced a picture: - The seven files fitted together with no conflicts to resolve. Nothing had to be moved between files, no procedure was duplicated across two authors, and no frozen type had to change: `check.sh` and the 1294-test suite were both green on the first run after the seven landed. That is what freezing `Device`, `Bitmap`, `Render_Options` and the `Span_Proc` contract ahead of the code bought. - **The render sweep** (`test_corpus_render` in `corpus_test.odin`) is the check this layer exists for and the most informative thing run on the project so far. It renders the first page of every corpus file — every page of every file under twenty — at 72 dpi, holds each render to `page_size`'s dimensions, a 4-component straight-alpha bitmap and a time budget, and then measures the fraction of non-background pixels. Over 71 documents and 315 pages: zero errors, zero leaks, mean ink 18.3% (Downloads) and 7.1% (the ISO corpus), slowest page 1.4 s, no page's dimensions disagreeing with `page_size`. - **Blankness is the invariant that finds real bugs, and it found one.** A page that draws nothing is invisible to every other check: it parses, loads every font, decodes every image, and returns `.None`. So the sweep also asks the *text* device what the interpreter saw on any page under 1% ink, and reports three different things where a bare ink number reports one: a page that draws nothing and extracts nothing is an empty sheet, a page that draws nothing and extracts 3 000 characters is a defect, and a page with a paragraph on it under 1% ink is the partial version of the same defect. Of the 19 light pages in the two corpora, 6 are genuinely blank paper and 14 are the non-embedded font gap. **Not one is an interpreter, clip or base-matrix bug** — which was the thing most likely to be wrong and is the reason the sweep was written. - The diagnosis is unambiguous on `Videcom/VRS XML Service 2020 02FEB2023.pdf` page 1, the clearest case: the embedded Courier New draws its code blocks perfectly, every heading rule and the hyperlink rule land exactly where they belong — and every word set in Arial or Times New Roman, neither of which the file embeds, is missing. The geometry, the clipping, the colour and the stroking are all correct; the outlines simply are not in the file and are not in this library either. - **Text extraction** (`test_corpus_text`) runs the same interpreter into the text device over every page of every file: 3 066 pages, 5.5 MB, 5.37 M runes. The ISO corpus extracts text from 100% of its 1 234 pages with **zero** U+FFFD. The Downloads corpus yields text on 85% of pages at 0.55% U+FFFD, and every one of the 21 pages above 10% U+FFFD is in a single Arabic-language ITU recommendation whose subset fonts name their glyphs by contextual presentation form — `lamwithaleffinal`, `hahwithmeeminitial` — which is in neither the AGL nor the file's own `/ToUnicode`. Those glyphs *render* correctly; there is simply nothing in the file that says what they mean, so U+FFFD is the honest answer and not a defect. The unicode chain is otherwise clean across the whole corpus. - `examples/pdfrender` and `examples/pdftext` were added and are compiled by `check.sh` alongside `pdfinfo`, which is what keeps `render_page`, `extract_text`, `Bitmap`, `bitmap_row` and `scale_matrix` genuinely public rather than accidentally reachable only from inside the package. - 307 of the 1294 tests belong to these seven files. None are vacuous: the 21 that carry no `testing.expect` of their own all go through a local helper that asserts (`rt_expect`, `pt_expect_point`, `ol_expect_label`), the same spelling every batch before them used. `docs/corpus-notes.md` records what the real-world corpus actually uses, which is what the ordering above is based on; `docs/build-plan.md` is the plan.