pdf-odin-llm/ARCHITECTURE.md

# pdf-odin — architecture and contributor contract

A self-contained PDF reader and renderer in pure Odin. No C dependencies:
everything outside of `core:` is written here.

## Pipeline

```
bytes
  │  lexer.odin, parser.odin          COS syntax  →  Object
  │  xref.odin, objstm.odin           file structure, object streams
  │  crypt.odin                       decryption (transparent, at parse time)
  ▼
Document  ── document.odin, pagetree.odin ─────────────────────────────────┐
  │  filters.odin + filter_*.odin     stream decoding                      │
  ▼                                                                        │
Page      ── page.odin                                                     │
  │  content.odin                     content-stream operators             │
  │  gstate.odin                      graphics state stack                 │
  │  colorspace.odin, function.odin, shading.odin, pattern.odin            │
  │  font*.odin, cmap.odin            glyph outlines and widths            │
  ▼                                                                        │
Device (vtable)  ── device.odin ───────────────────────────────────────────┘
  ├── device_raster.odin   → Bitmap (RGBA8)      path.odin, raster.odin,
  ├── device_text.odin     → extracted text      stroke.odin, blend.odin
  └── device_list.odin     → display list         image.odin
```

Everything above `Device` is format knowledge; everything below is pixels. The
three devices share one interpreter, which is why text extraction agrees with
what is drawn instead of being a second, subtly different parse.

## File ownership

One concern per file. Do not add procedures to a file you do not own.

| File | Contents |
|---|---|
| `pdf.odin` | package doc, version constants |
| `error.odin` | `Error`, `Repair`, `error_string` |
| `object.odin` | `Object`, `Dict`, `Array`, `Stream`, accessors |
| `geom.odin` | `Point`, `Matrix`, `Rect` |
| `lexer.odin` | COS tokeniser |
| `parser.odin` | token stream → `Object` |
| `xref.odin` | xref tables, xref streams, trailer chain, rebuild-by-scan |
| `objstm.odin` | `/Type /ObjStm` compressed object streams |
| `document.odin` | `Document`, `open_*`, `close`, `resolve`, object cache |
| `pagetree.odin` | page tree walk, inherited attributes, `Page` |
| `crypt.odin` | standard security handler, RC4, AES-CBC, key derivation |
| `filters.odin` | filter chain dispatch, `stream_data`, predictors |
| `filter_flate.odin` | tolerant inflate |
| `filter_lzw.odin` | LZW |
| `filter_ascii.odin` | ASCIIHex, ASCII85 |
| `filter_runlength.odin` | RunLength |
| `filter_ccitt.odin` | CCITT G3/G4 fax |
| `filter_dct.odin` | JPEG: baseline + progressive, grey/YCbCr/CMYK/YCCK |
| `encoding.odin` | text strings, PDFDocEncoding, dates, name/number trees |
| `function.odin` | PDF functions type 0/2/3/4 |
| `colorspace.odin` | all colour spaces → RGB |
| `path.odin` | `Path`, construction, transform, flatten |
| `raster.odin` | anti-aliased scanline fill |
| `stroke.odin` | stroke → fill path (joins, caps, dashes) |
| `bitmap.odin` | `Bitmap`: allocation, rows, premultiplication, RGBA conversion |
| `blend.odin` | blend modes, compositing, soft masks |
| `image.odin` | image XObjects → samples, image masks, `/SMask` |
| `shading.odin` | shading types 1–7 |
| `pattern.odin` | tiling and shading patterns |
| `cmap.odin` | CMap parsing, `/ToUnicode`, predefined CMaps |
| `font.odin` | font loading and dispatch, widths, encodings |
| `font_cff.odin` | CFF / Type2 charstrings |
| `font_truetype.odin` | `glyf`/`loca`/`cmap`/`hmtx`/`post` |
| `font_type1.odin` | Type1, eexec, Type1 charstrings |
| `font_standard.odin` | standard-14 metrics, built-in encodings |
| `content.odin` | content-stream lexer and operator dispatch, Type3 glyph execution |
| `gstate.odin` | graphics state and its stack |
| `device.odin` | the `Device` vtable |
| `device_raster.odin` | rasterising device |
| `device_text.odin` | text-extraction device |
| `device_list.odin` | display-list device |
| `page.odin` | `render_page`, `extract_text`, public page API |
| `annot.odin` | annotation appearance streams |
| `outline.odin` | document outline, destinations |

## Conventions

- **4 spaces**, no tabs. Types `Pascal_Snake` (`Xref_Entry`, `Graphics_State`)
  except the handful of short public nouns already fixed (`Object`, `Dict`,
  `Page`, `Matrix`). Procedures and variables `snake_case`. Enum members
  `Pascal_Snake`.
- Procedures are prefixed by the noun they act on: `path_transform`,
  `font_glyph`, `dict_get`. This keeps a flat 40-file package navigable.
- `@(private)` anything that is not part of the public API or used across
  files. The public API is exactly what the package doc comment at the top of
  `pdf.odin` names, plus the types those procedures take and return: `open_file`,
  `open_bytes`, `Open_Options`, `close`, `page_count`, `page_get`, `Page`, the
  object accessors (`resolve`, `get_object`, `get_dict`, `get_stream`, `dict_*`,
  `array_*`, `to_*`, `stream_data`), `Error`, `Repair`, `error_string`, the
  encryption accessors, and -- once the device layer exists -- `render_page`,
  `extract_text` and `Bitmap`. Everything else is `@(private)`, tests included:
  a test lives in the same package and reaches private names fine, so being
  called by one is not a reason to export. Widening the public API means editing
  that doc comment in the same commit, which is what keeps the two in step.
- Comments explain *why*, especially why a tolerance exists: "a `/Length`
  that disagrees with the bytes is common enough that …". Do not narrate what
  the next line does.
- Cite the spec by clause number when the behaviour is non-obvious:
  `// ISO 32000-2, 7.5.8.2`.

## Memory

- A `Document` owns a growing arena (`core:mem/virtual`). Every object,
  dictionary, stream and decoded buffer produced while reading lives there and
  is reclaimed in one call by `close`. Allocate with `doc.allocator`.
- Anything handed to a caller that is *not* owned by the Document takes an
  explicit `allocator := context.allocator` parameter and says so in its doc
  comment. That is `render_page`'s `Bitmap`, `extract_text`'s string, and
  nothing else.
- Per-page rendering scratch goes in a temporary arena that is reset between
  pages, never in the Document arena — a 900-page render must not grow without
  bound.
- Never call `free` or `delete` on Document-arena memory.

## Robustness — the rule that matters most

Malformed PDFs are the normal case, not the exception. A reader that rejects a
file every other viewer opens is a broken reader.

1. **Never panic.** No unchecked slicing, no `assert` on file-derived data, no
   unwrapped `or_else` on parsed values. Bounds-check every index that came
   from the file.
2. **Never loop forever.** Every loop driven by file data needs a bound: a
   recursion depth limit (`MAX_DEPTH`), an iteration fuel counter, or a
   visited-set. Cyclic `/Parent` chains, self-referencing object streams and
   xref `/Prev` loops all occur in real files.
3. **Degrade, don't fail.** A broken font renders as blanks; a broken image is
   skipped; an unparsable content fragment is stepped over to the next
   operator. Record a `Repair` on the Document and carry on. Only return an
   `Error` when there is genuinely nothing to hand back.
4. **Clamp allocations.** A `/Length`, `/W`, `/Height` or `/Size` from a file is
   attacker-controlled. Check it against the remaining file size before
   allocating, and against `MAX_*` limits.
5. **No NaN or infinity below the Device line.** Numbers coming out of a file
   are checked with `is_finite` at the boundary.

## Tests — write them with the code, not after

Every file `foo.odin` gets a `foo_test.odin` in the same package, using
`core:testing`. A layer is not finished until its tests pass.

- **Unit tests** are self-contained: byte literals in, expected values out.
  Codec tests use vectors from the format's own specification where one exists.
- **Golden PDFs** live in `testdata/` and are tiny, hand-written, and
  committed. They cover the structural cases: classic xref, xref stream, object
  stream, incremental update, broken xref needing a rebuild, wrong `/Length`,
  each encryption revision.
- **Corpus tests** run over whatever is in the directory named by the
  `PDF_ODIN_CORPUS` environment variable and skip silently when it is unset, so
  they never depend on files that are not in the repository. They assert only
  invariants: opens without error, page count > 0, every page renders without
  crashing, no leaks.
- Fuzz-shaped tests: truncate and bit-flip the golden files at many offsets and
  assert the library returns an error rather than crashing or hanging.

Run `scripts/check.sh` (type-check + vet) and `scripts/test.sh` before
declaring anything done.

## Limits

Defined in `document.odin`, referenced everywhere:

```
MAX_DEPTH            32     // nested arrays/dicts, form XObject recursion
MAX_OBJECT_NUMBER    8388607
MAX_XREF_CHAIN       64     // /Prev hops
MAX_CONTENT_OPERANDS 32
MAX_BITMAP_PIXELS    (1 << 28)
```


## Status: what is written and what remains

**The library renders.** Every file in the table above is written, tested and
driven over a real corpus. `open_file -> page_get -> render_page` produces a
correct picture of a real page, `extract_text` produces its text off the same
interpreter, and `page_display_list` records it for a caller with a backend of
their own. There is no skeleton left.

Complete: `pdf error object geom document lexer parser xref objstm pagetree
crypt encoding filters filter_flate filter_lzw filter_ascii filter_runlength
filter_ccitt filter_dct path bitmap gstate device raster stroke blend
colorspace function cmap font_standard font_cff font_truetype font_type1 font
image shading pattern content device_raster device_text device_list page annot
outline`.

Type3 was planned as `font_type3.odin` and does not need a file: a Type3 glyph
is a content stream, so loading it belongs to `font.odin` with the other font
dictionaries (`font_load_type3`, `font_type3_proc`) and *running* it belongs to
the interpreter that runs every other content stream (`content_run_type3`,
`content_type3_d1`, and the `Type3_Glyph` flag that makes `d0`/`d1` legal and
stops a glyph that shows itself from recursing). Splitting it out would have
meant a file that could only call back into `content.odin`. It is finished and
proven end to end: a Type3 font whose glyph is `0 0 750 750 re f` at 40 pt
rasterises to exactly the three 30-point squares its `/FontMatrix` and
`/Widths` call for.

### What remains

Nothing structural. What is left is one real gap, two deliberate
approximations, and the review debt from the earlier batches.

**Non-embedded fonts draw nothing.** This is the one visible defect in the
whole layer and the render sweep below found it. A font dictionary with no
`/FontFile*` still loads, still lays out and still extracts text correctly --
that is what the render-layer contract asks for and it works -- but
`font_glyph_path` has no outline to return, so the text is invisible. The
library ships no font programs of its own and reads none from the system, so
there is nothing to substitute *with*: fixing it means either bundling a
metric-compatible face for the standard 14 or adding a system-font lookup, and
both are new subsystems rather than a repair. Fifty-nine of the corpus's 894
fonts are unembedded, and they are why fifteen corpus pages come back blank or
nearly blank.

**Two transparency approximations**, both stated at the top of
`device_raster.odin` where they are made: a knockout group is rendered as a
non-knockout one, and a non-isolated group whose own alpha is not 1 or whose
own blend mode is not Normal is rendered isolated. Both are the approximations
every renderer without a backdrop-removal pass makes, and `docs/corpus-notes.md`
ranks the cases they differ on as rare.

**Still owed from the earlier batches**, unchanged: the two adversarial review
lenses -- visibly-wrong pixels, and the data tables audited line by line
against the AFMs and the AGL. Every table reached by binary search
(`STD_TEXT_GLYPHS`, `STD_SYMBOL_GLYPHS`, `STD_ZAPF_GLYPHS`, `AGL_TABLE`,
`AGL_BY_RUNE`) has been verified sorted and in range, and spot values across
all five width tables and both AGL directions match their source documents, but
that is a smoke test, not the audit.

### What the integration checks have covered

Three batches were built in parallel and each has since been integrated. The
first was the six rendering primitives — `raster stroke blend colorspace
function cmap font_standard`; the second was the seven files above them —
`font font_cff font_truetype font_type1 image shading pattern`; the third was
the render layer itself — `content device_raster device_text device_list page
annot outline`.

From the first batch:

- The layer composes. `path_flatten -> raster_fill -> blend_span` and
  `stroke_flatten -> raster_fill -> blend_span` were driven end to end into a
  Bitmap and checked pixel by pixel against the geometry: aligned and
  half-covered rectangles, both fill rules, butt and round joins and caps, a
  dashed line, a hollow stroked square, a hairline under a 4x CTM, and a clip
  not anchored at the origin. All correct, with no fringing and no seams.
- The three conventions the primitives meet on hold everywhere: coverage is
  0..255 with 255 meaning fully covered; every `f64` colour is straight and
  0..1; every `Bitmap` below the Device line is premultiplied. `blend.odin`'s
  header states this rule and every entry point in it obeys it.
- One composition bug, in `path.odin` rather than in the six: `path_flatten`
  dropped the connecting segment of a `l` that followed an `h`, because its
  `.Line` arm did not seed the new subpath with the current point the way its
  `.Curve` arm always has. Fixed, with a regression test that holds the two arms
  to each other.
- Two parallel-authorship seams, both fixed: `object_identity` (the cache key
  both `colorspace_load` and `function_load` use) had been defined in
  `function.odin` and reached into from `colorspace.odin`, and now lives in
  `object.odin` with the rest of the `Object` accessors; `font_standard.odin`
  carried its own hex-digit decoder alongside `object.odin`'s `hex_value`.
- One internal contradiction, fixed: the bare `/Pattern` singleton declared
  `n = 1` directly under a comment saying a colour in it has no components.
  `n` is what tells the interpreter how many operands `scn` takes, and `/P1 scn`
  takes none, so it is 0.

From the second batch:

- The seven compose with the primitives and with each other. A font dictionary
  with an embedded TrueType program was driven through `font_load ->
  font_decode -> font_glyph_path -> path_flatten -> raster_fill -> blend_span`
  and the ink checked pixel by pixel against where the glyph's em square lands
  under a real text matrix; an image was driven through `image_load ->
  image_decode -> blend_span` as a stencil and through `blend_span_rgba8` as
  colour; an axial shading and the same shading behind a `/PatternType 2` were
  painted and checked against exact pixel values computed by hand. Every one
  fitted first time.
- One parallel-authorship seam, fixed: `image.odin` carried its own
  `img_byte` — the 0..1-to-byte quantiser, NaN handling and half-up rounding
  and all — beside `blend.odin`'s `blend_quantize`, which is the same procedure
  and which `shading.odin`, written by a different author against the same
  contract, had correctly reused. `image.odin` now calls `blend_quantize` too.
  Quantising a colour to a byte is `blend.odin`'s to own: it is the file whose
  header states the byte conventions the whole device layer meets on.
- The per-format byte readers (`cff_u16`, `tt_u16`, `mesh_bits`, `img_sample`)
  were checked against each other and left alone. They look alike and are not:
  each font-program reader is required to be self-contained, and the two
  sample unpackers differ in kind — one is indexed and capped at 16 bits, the
  other is a sequential bitstream reader up to 32.
- `corpus_test.odin` grew `test_corpus_resources`, which loads and drives every
  font, image, shading and pattern named by every page of every corpus file,
  recursing into form XObjects and tiling-pattern resources. Over 71 documents
  it reaches 894 distinct fonts and 1579 images, probes 123 000 glyph outlines
  and decodes 372 000 character codes, and found no failure of any kind. That
  is the first time the library met real subsetted font programs at scale, and
  the invariants it holds them to are the ones the interpreter will depend on:
  a Font always comes back, an embedded program either has glyphs or sets
  `program_failed`, every outline is finite, `font_decode` consumes exactly the
  bytes it is given, and an image either decodes to a bitmap of its own
  dimensions or degrades to an error with no bitmap at all.
- Three things the sweep taught, all recorded here because they will otherwise
  be rediscovered as bugs: a subsetter leaves `maxp.numGlyphs` at the original
  count while emptying `loca`, so a font can declare 22 026 glyphs and hold 25
  — sampling glyph ids rather than scanning them will find none of them; a
  Type1C subset is routinely stripped to `.notdef` and `space`, both of which
  are a single `endchar` byte, so "no glyph in this font draws" is a normal
  file and not a defect; and a font with no embedded program sets neither a
  glyph count nor `program_failed`, because nothing failed — the standard-14
  metrics are the whole of what the file gave.
- 296 of the 976 tests belong to these seven files. None are vacuous: the ones
  that appear to assert little all go through a local helper that asserts, and
  the two that end in `expect(t, true)` are the truncation and bit-flip fuzzers,
  where reaching the last line is the property under test — the same spelling
  `xref_test.odin` and `objstm_test.odin` have used since the COS layer.

From the third batch — the render layer, and the first time anything in this
repository produced a picture:

- The seven files fitted together with no conflicts to resolve. Nothing had to
  be moved between files, no procedure was duplicated across two authors, and
  no frozen type had to change: `check.sh` and the 1294-test suite were both
  green on the first run after the seven landed. That is what freezing
  `Device`, `Bitmap`, `Render_Options` and the `Span_Proc` contract ahead of
  the code bought.
- **The render sweep** (`test_corpus_render` in `corpus_test.odin`) is the
  check this layer exists for and the most informative thing run on the project
  so far. It renders the first page of every corpus file — every page of every
  file under twenty — at 72 dpi, holds each render to `page_size`'s dimensions,
  a 4-component straight-alpha bitmap and a time budget, and then measures the
  fraction of non-background pixels. Over 71 documents and 315 pages: zero
  errors, zero leaks, mean ink 18.3% (Downloads) and 7.1% (the ISO corpus),
  slowest page 1.4 s, no page's dimensions disagreeing with `page_size`.
- **Blankness is the invariant that finds real bugs, and it found one.** A page
  that draws nothing is invisible to every other check: it parses, loads every
  font, decodes every image, and returns `.None`. So the sweep also asks the
  *text* device what the interpreter saw on any page under 1% ink, and reports
  three different things where a bare ink number reports one: a page that draws
  nothing and extracts nothing is an empty sheet, a page that draws nothing and
  extracts 3 000 characters is a defect, and a page with a paragraph on it
  under 1% ink is the partial version of the same defect. Of the 19 light pages
  in the two corpora, 6 are genuinely blank paper and 14 are the non-embedded
  font gap. **Not one is an interpreter, clip or base-matrix bug** — which was
  the thing most likely to be wrong and is the reason the sweep was written.
- The diagnosis is unambiguous on
  `Videcom/VRS XML Service 2020 02FEB2023.pdf` page 1, the clearest case: the
  embedded Courier New draws its code blocks perfectly, every heading rule and
  the hyperlink rule land exactly where they belong — and every word set in
  Arial or Times New Roman, neither of which the file embeds, is missing. The
  geometry, the clipping, the colour and the stroking are all correct; the
  outlines simply are not in the file and are not in this library either.
- **Text extraction** (`test_corpus_text`) runs the same interpreter into the
  text device over every page of every file: 3 066 pages, 5.5 MB, 5.37 M runes.
  The ISO corpus extracts text from 100% of its 1 234 pages with **zero**
  U+FFFD. The Downloads corpus yields text on 85% of pages at 0.55% U+FFFD, and
  every one of the 21 pages above 10% U+FFFD is in a single Arabic-language ITU
  recommendation whose subset fonts name their glyphs by contextual
  presentation form — `lamwithaleffinal`, `hahwithmeeminitial` — which is in
  neither the AGL nor the file's own `/ToUnicode`. Those glyphs *render*
  correctly; there is simply nothing in the file that says what they mean, so
  U+FFFD is the honest answer and not a defect. The unicode chain is otherwise
  clean across the whole corpus.
- `examples/pdfrender` and `examples/pdftext` were added and are compiled by
  `check.sh` alongside `pdfinfo`, which is what keeps `render_page`,
  `extract_text`, `Bitmap`, `bitmap_row` and `scale_matrix` genuinely public
  rather than accidentally reachable only from inside the package.
- 307 of the 1294 tests belong to these seven files. None are vacuous: the 21
  that carry no `testing.expect` of their own all go through a local helper
  that asserts (`rt_expect`, `pt_expect_point`, `ol_expect_label`), the same
  spelling every batch before them used.

`docs/corpus-notes.md` records what the real-world corpus actually uses, which
is what the ordering above is based on; `docs/build-plan.md` is the plan.