4 commits

Author	SHA1	Message	Date
andodeki	1220f89fc6	feat(pdf): close the last three Phase 4 items — reconciliation, CFF, cmap ... The three items the previous commit's audit found unimplemented while the status line said "complete". All three are done and externally verified. **1. Field-value reconciliation** (`reconcile.rs`). A field carries its value in /V and its rendered look in /AP, and nothing in the format keeps them in step. Files arrive with them disagreeing all the time: a producer writes /V and leaves appearances to the viewer, or something edits /V without touching /AP. Until now this crate simply believed /V and regenerated appearances only for fields it had itself edited — right for a field we changed, wrong for a field that arrived inconsistent. The module deliberately does **not** pick a winner. PDF 32000-1 §12.7.3.3 settles exactly one case — /NeedAppearances true means /V is authoritative — and is silent on the other, where a conforming viewer renders /AP and never looks at /V. So it classifies the disagreement and resolves it against a caller-declared `Intent`, because the right answer genuinely differs: a viewer must show /AP to match other viewers, an extractor must read /V, an editor must regenerate so the saved file agrees with itself. Silently choosing one would be ADR 0017's failure in a new place — every answer plausible, none checkable, the caller unaware a decision was made for it. Two cases are not judgement calls and are handled outright. A missing or dangling appearance renders *blank*, and blank is never what the producer meant, so even Display regenerates. An unselected radio member showing /Off while the group's /V names another member is correct, not a conflict — reporting it would flag every well-built radio group there is. **2. Type1/CFF embedding** (`embed_opentype_whole`). The spec says "if feasible". Subsetting CFF is not — it means rebuilding the CFF INDEX, charset and charstrings, a second font format inside the first — and `subset_truetype` rightly keeps refusing it by name. Embedding the program *whole* is feasible, and that is what this does: /FontFile3 with /Subtype /OpenType under a CIDFontType0 descendant, per Table 126. Each of those keys matters and none is guessable from the others. A CFF program in /FontFile2, or under a CIDFontType2 descendant, still produces a file qpdf accepts and a font that loads as the wrong type or not at all. /CIDToGIDMap is omitted because it is defined for CIDFontType2 only. The trade is made visible rather than buried: `EmbeddedFont::is_subsetted` is false here, so a caller with a size budget — or a licence that forbids shipping a whole face — can refuse instead of discovering it from the output size. **3. `repair-cmap`** (`glyph_index`). A symbol font declares no Unicode subtable: it maps glyphs into the private-use area at 0xF000 + the low byte under platform 3, encoding 0. Asking it for 'A' found nothing and the character silently vanished from the output — the font "missing" a glyph it plainly has. Now the (3,0) subtable is kept as a fallback and retried at 0xF000 + low byte, after the proper lookup fails so a font with both subtables is still read through the Unicode one. Format 0 is read too; omitting it left legacy and symbol fonts mapping nothing while appearing to have a usable cmap. The repair must not manufacture glyphs, which is its own test: a character the font genuinely lacks still returns None, because turning a missing character into a wrong one is worse. **Fixtures.** No CFF or symbol font ships on the CI image, and neither can be tested honestly against a hand-built stub — the point is that the bytes are a font program a third-party reader accepts. Both are generated from DejaVu by checked-in fontTools scripts: `cff_sample.otf` (1.6 KB, real OTTO/CFF outlines) and `symbol_sample.ttf` (664 B, a single (3,0) subtable so the repair path is the only route to its glyphs). Both generators pin `head.created`/`head.modified` to zero. fontTools stamps the current time, so the output differed on every run and CI's "fixtures match their generator" check failed against a file nobody had edited. Caught by running that check rather than assuming it passed. A fixture that cannot be regenerated byte-for-byte is not reviewable: you cannot tell a deliberate change from a rebuild. **Verified by mutation**, seven injected defects, each confirmed red: NeedAppearances ignored 1 fail dangling /AS not detected 1 fail blank rendering shown faithfully 1 fail CFF written to /FontFile2 1 fail CFF given a CIDFontType2 descendant 1 fail whole font claims to be subset 1 fail cmap 0xF000 retry removed 3 fail **Verified externally.** The sample now carries a third page set in the whole-embedded CFF font, and `check-pdf-external-readers.sh` gained `pdffonts` — the only check that inspects a font *program* rather than the file structure, which is exactly where a wrong /FontFile key shows up. poppler reports both fonts embedded and distinguishes them correctly: ETXLDI+DejaVuSans CID TrueType Identity-H emb yes sub yes NigigTestCFF CID Type 0C (OT) Identity-H emb yes sub no and extracts "Hello CFF 123", which only works if the CFF program loaded, /Identity-H addressed its glyphs and /ToUnicode mapped them back. That check also caught its own page-count assertion going stale when the third page landed — a gate that notices its own fixture changing is working. Engine suite 953 -> 985. Coverage 87.27%, all floors met. Phase 4 is complete but for the ui.rs interaction tests, which are written and blocked on the Makepad fork's missing headless backend.	2026-08-17 09:31:41 +00:00
andodeki	89ca5186c6	docs(pdf): Phase 4 is not 100% — audit it, and verify the half that is ... Asked whether Phase 4 was complete, I checked the tree instead of my own commit message, and the commit message was wrong. Three items named in the Phase 4 spec are **not** implemented, and the status line said "complete" over them: - **Field-value reconciliation** (`form_reconcile_test.dart`). Setting a value writes /V, marks the field dirty and regenerates /AP — that all works. What is missing is the reconciliation case: a file opened with /V and /AP *already disagreeing*, where the right answer depends on /NeedAppearances. Nothing decides that today. - **Type1/CFF embedding.** The spec hedges with "if feasible", so this is a legitimate deferral rather than an oversight — but "complete" did not say so. `sfnt.rs` detects CFF outlines and `font.rs` reads an existing /FontFile3; nothing writes one. Creation is TrueType-only. - **`repair-cmap`.** No equivalent exists. `text_box_appearance_test.dart` *is* covered, by appearance.rs:235 — it just does not carry that filename, which is why a grep for the dart test names is a starting point and not an answer. The other half of the exit criterion — "generated PDFs open cleanly in external viewers" — had never been checked at all. The sample generator's own doc comment admits no test in this repository can assert it. So I ran it through implementations we share no code with, and **it passes**: qpdf --check no syntax or stream encoding errors pdfinfo title, author, subject, keywords, 2 pages, Form: AcroForm pdftotext all text, including the embedded DejaVu subset and its em-dash qpdf --list-attachments readme.txt, extracted by name with description catalogue /Outlines /Names /EmbeddedFiles /PageLabels /Dests /PageMode /ViewerPreferences /AcroForm `tools/check-pdf-external-readers.sh` makes that repeatable, and pdf.yml runs it. It treats a qpdf *warning* as failure, not just an error: qpdf warns where it had to reconstruct, and reconstructing is exactly what a stricter viewer will refuse to do. Negative-tested twice — removing the attachment fails 3 checks, and corrupting the startxref offset makes qpdf report "file is damaged". Two defects that audit found: - **The sample never exercised XMP**, so the Phase 4 feature most likely to be silently missing was also the one nothing looked at. Probed separately: `set_xmp_metadata` works, pdfinfo reports `Metadata Stream: yes`. - **A `Banner` naming an unregistered font produces a structurally valid PDF that renders no text.** qpdf --check passes; poppler says `Unknown font tag 'F1'` and draws nothing. `stamp.rs` cannot register the font itself — fonts belong to the document, and a banner does not know which document it will be drawn into — so this is now documented on `Banner` with a worked example, and pinned by `a_banner_font_must_be_registered_or_the_page_lacks_the_resource`, which asserts on the page's /Font resources because that is the thing actually missing and the thing a caller can check. The plan now records that it was wrong once, rather than quietly correcting itself. A status line that has been overstated should show its working. Engine suite 952 -> 953. Phase 4's engine half is verified end to end against third-party readers; the ui.rs interaction half is written and still blocked on the Makepad headless backend.	2026-08-17 09:11:03 +00:00
andodeki	8701f5df51	feat(pdf): image embedding and header/footer stamping — Phase 4 complete ... The last gap in Phase 4: dart-pdf's header_footer_test, image_stamp_test and image_pdf_test had no counterpart here. What was missing is worth stating precisely, because it is the shape of bug ADR 0017 exists to catch. ContentWriter::draw_image has emitted `q w 0 0 h x y cm /Name Do Q` since Phase 2, and was tested. But nothing in the stack could *create* the image XObject that /Name resolves to. So every Do operator ever written named a resource that did not exist, no document could contain a raster image, and nothing anywhere returned an error. The writing half was present, the reading half faithfully reported the content stream, and the image was simply never there. stamp.rs adds: image XObject embedding, header/footer banners with left/centre/right alignment, image stamp content, and stream composition. A JPEG is embedded as-is with /DCTDecode — PDF's image model is the same DCT data the file already holds, so re-encoding would lose quality for nothing — and its geometry is read from its own SOF marker rather than trusted from the caller, because a /Width that disagrees with the codestream renders as diagonal garbage in every viewer. Raw samples embed as Flate. Embedding an image then adding the page that draws it exposed a live defect in PdfDocBuilder. add_object derived its number from `3 + 2 * pages.len()`, so every add_page after an add_object silently shifted a number already handed out. Embedding an image and then adding its page — the natural order, since the page's content stream has to name the image — produced a page whose /XObject entry pointed at the page object itself: 3 0 obj <</Type /Page ... /XObject <</Im0 3 0 R>>>> The file parsed. The reference resolved. The resource was the page. This is the same positional-numbering defect already fixed once for fonts, one layer out — the comment above first_extra_object_number describes the font version, where /ToUnicode pointed at the descriptor and /FontFile2 at the Type0 wrapper. Both come from deriving object numbers from collections that are still growing. Fixed at the root: the page count is frozen when the first extra number is issued, and pages added afterwards are allocated past the fixed block instead of colliding with it. Non-contiguous page numbers are legal — /Kids is an explicit array — and 952 tests confirm nothing depended on the order. The integration tests parse the generated file back with PdfDocument and assert the image appears in `page.xobjects` with subtype Image, that its /Width and /Height match the SOF marker, and that the header and footer baselines are at opposite ends of the page. Reading the resource back is the assertion that matters: a substring check for "/Im0 Do" passed throughout the entire period when no image could be embedded at all. Verified by mutation, five injected defects, each confirmed red: numbering fix reverted 4 fail JPEG width/height transposed 5 fail header positioned from bottom 3 fail sample-count check removed 1 fail attach_image_to_page a no-op 5 fail One test needed correcting rather than the code: three assertions grepped the output for operators, which are Flate-compressed by default, so they were asserting against compressed bytes. They now disable compression explicitly — the structure is identical either way, and the alternative was a test of miniz_oxide. Engine suite 920 -> 952. Coverage 86.16% -> 86.40%; stamp.rs at 94.64% with a floor at 90. Phase 4 is complete and the plan records it, including the numbering defect, since a status table that lists only features would not have told the next reader why the object numbers look the way they do.	2026-08-16 22:27:32 +00:00
andodeki	7d6fc4cbbe	feat(pdf): document creation — outlines, forms, attachments, font subsetting ... Phase 4 of NIGIG_PDF_FEATURE_PARITY_PLAN.md. ADR 0019. Almost none of it existed: Outlines, PageLabels, EmbeddedFiles and ViewerPreferences appeared nowhere in the workspace, in any crate. What did exist was a builder whose central method was pub fn add_page_with_content(&mut self, _width: f64, _height: f64, ...) which accepted a page size and discarded it. Asking for 200x400 and 300x500 gave two US Letter pages, because no /MediaBox was written at all. The test asserted the output contained the string "/Type /Page", which it did. Two more defects sat in the object writer, both producing files our own parser rejects: dictionary keys were written unescaped (a key with a space reparses as "expected number"), and f64::NAN was emitted as the literal token NaN, so one non-finite value anywhere made the document unreadable. Added: outline trees with the open/closed state in the sign of /Count, /PageLabels as a number tree with real roman and A..Z/AA..ZZ numbering, named destinations, attachments with file specs, /Info, XMP, viewer preferences, page mode and layout; AcroForm creation for text, checkbox, radio, choice and signature fields with generated appearances; and TrueType subsetting - DejaVu Sans goes from 759,720 bytes to 4,348 for twelve characters. cmap is deliberately not rebuilt: the subset is embedded as a CID font with Identity-H, so the content stream addresses glyphs by id and /ToUnicode serves extraction. A cmap disagreeing with the content stream is worse than none. CFF is refused by name rather than emitting a font with no glyphs. Nine real bugs, every one found by running the output through an independent tool rather than by reading the code: 1 page size discarded reading a generated file back 2 dict keys unescaped probing the writer 3 NaN written as a keyword probing the writer 4 subset zeroed the lsb fontTools outline compare 5 hmtx indexed by new gid fontTools outline compare 6 name table format read as count BaseFont came out "Embedded" 7 add_font shifted numbers already handed out 8 trees allocated over font numbers - object 29 written twice 9 widgets missing /F Print, /P and appearance /Resources 7 and 8 are the instructive pair: every reference resolved and every object existed, each simply named the wrong thing. pypdf reported correct field values from a file PDFium rendered blank. 9 is the one only a renderer could find - /F defaults to non-printable, and a form XObject naming a font its /Resources does not declare is discarded whole. Verified by three independent implementations: fontTools (0 outline mismatches of 12 against the source font), pypdf (metadata, page sizes, outline with resolved page numbers, all five fields, attachment byte-for-byte, labels ['i','1']) and PDFium, which renders both pages correctly. cargo run -p nigig-pdf-graphics --example generate_sample regenerates the sample. Fourteen mutations. Three survived and each exposed a weak test: the key test used an attachment name (written as a string, never a key), nothing read the outline open state, and /P could not be witnessed because page_index is supplied by the reader, which already knows the page. All three now killed. pdf: 789 passed (was 730). pdf-ui: 775. Coverage 85.17%.	2026-08-16 21:02:19 +00:00