12 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| a994213e8b |
feat(pdf): tagged structure tree, and the BMC bug that turned red green
Phase 8 #7 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("PDF/UA structure tree"). Design and merge criteria in REVIEWS/adr/0011-pdf-structure-tree.md. Investigating the accessibility gap surfaced four defects in the marked-content operators the structure tree depends on. The first is not an accessibility problem at all. 1. BMC never parsed, and corrupted the colour of everything after it. The dispatcher matched b'B'+b'M' only when the third byte was a delimiter; BMC's third byte is 'C', so the arm never fired. Because the operator was never recognised it never CLEARED ITS OPERAND, and the leftover /Tag shifted the operands of whatever came next: 1 0 0 rg -> RgbFill(1.0, 0.0, 0.0) red /Span BMC 1 0 0 rg -> RgbFill(0.0, 1.0, 0.0) green Tagged documents are precisely the ones containing BMC, so the documents that tried hardest to be accessible rendered wrong colours. This is the fifth instance of the operator-shadowing family already fixed for cm, rg, gs, b/b* and end-of-stream text operators. ADR 0009's every_multi_char_operator_parses_as_itself test exists to stop exactly this - and would have caught it, except BMC was one of two operators excluded from its table as "genuinely unimplemented". Excluding a known-broken operator from the test whose job is finding broken operators is how it survived. The exclusion list is gone. 2. BDC discarded its property list, which carries /MCID - the only link between a run of page content and the structure element describing it. Without it a tree can be parsed but never attached to anything. 3. The op produced by BDC was named MarkContentBmc, and BMC produced nothing. The names were the wrong way round, which is how the missing arm survived review: the enum looked like it had a BMC case. 4. Found while fixing 2: the content lexer had no dictionary support at all. It read `<` as a hex string without checking for a second `<`, so `<</MCID 0>>` parsed as the string "0C0D0". Content streams now parse direct objects properly, bounded at 16 levels; a single `<` is still a hex string and a test pins that. On top of that, new pdf-document/src/structure.rs: /StructTreeRoot, /StructElem trees, /RoleMap resolution, depth-first reading order, /Alt, /ActualText, /E, /Lang, and MCID-to-element lookup. Cycles in /K are cut at the first repeat and reported by object number rather than expanded to the depth bound. Accessibility findings are mechanical checks reported as findings, NOT a conformance verdict: there is no is_pdf_ua and no ComplianceReport, and a mutation-checked tripwire test fails if either appears. Real PDF/UA conformance needs human judgement - whether /Alt text is accurate is not mechanically decidable - so claiming it from six checks would be exactly the overclaim this codebase keeps removing. 7 corpus fixtures, 14 acceptance tests, 22 unit tests, and a parse_content_dict fuzz target for the new object parser. TEST_TARGET=pdf 536 -> 575, TEST_TARGET=pdf-ui 580 -> 619. rustfmt and clippy -D warnings clean. |
|||
| d8d29c226d |
feat(pdf): transparency, and four operator-parsing bugs it exposed
Phase 8 #5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("Advanced transparency"). Design and merge criteria in REVIEWS/adr/0009-pdf-transparency.md. The headline defect was not that transparency was missing. `gs` was MIS-PARSED as CloseStroke: /GS0 gs 1 0 0 rg 100 100 200 200 re f -> [CloseStroke, RgbFill, Rectangle, FillWinding] so every page setting a graphics state gained a stroked path the document never asked for, and lost its alpha, blend mode and soft mask silently. Downstream everything was dead: StrokeExtGState/FillExtGState were never constructed, set_fill_opacity was never called and emitted no command when it was, and PdfPage::ext_gstate was read by no code at all. New pdf-graphics/src/transparency.rs: full /ExtGState (ca, CA, BM, SMask, LW, LC, LJ, ML, D, AIS, TK), all sixteen blend modes including the four non-separable ones, soft masks with /S, /G, /BC and a /TR evaluated through ADR 0006's PdfFunction, and /Group parsing. Wired end to end: parser -> PdfOp::SetExtGState -> device -> RenderCommand -> Makepad renderer. Compositing is NOT claimed. Backdrop blending needs render-to-texture, which an engine-neutral crate has no framebuffer for. Constant alpha is applied because it needs no backdrop; blend modes and soft masks are reported through TransparencyError::Unsupported rather than dropped, because a silently ignored /Multiply looks exactly like a correct /Normal. The ADR required a test enumerating every multi-character operator, on the grounds that fixing the third instance of a shadowing bug (after cm and rg) without preventing the fourth is not a fix. It immediately found three more live bugs, none of them transparency-related: - `b` and `b*` dropped their close-path, mapping to FillStroke* instead of CloseFillStroke*, so every closed-and-stroked path drew with a gap. - Text operators within three bytes of the end of a content stream were mis-parsed: the b'T' arm guarded `*i + 3 < len` while reading only two bytes, so a stream ending in `/F1 12 Tf` parsed as FillWinding. And the transparency-group fixture found a fourth: - PdfPage::xobjects was empty for every page of every document. extract_xobjects called doc.resolve(), which follows the reference, then asked the resolved object for as_ref() - always None - and only accepted a bare dict when every XObject is a stream. No `Do` operator could be resolved through the page model. Same defect as the one that once destroyed annotation object references; it now has its own regression test. T* and BMC are genuinely unimplemented and are deliberately excluded from the guard's table rather than papered over. 7 corpus fixtures, 12 acceptance tests, 32 unit tests asserting the §11.3.5 formulas (not our own output), and a parse_ext_gstate fuzz target. TEST_TARGET=pdf 451 -> 497, TEST_TARGET=pdf-ui 495 -> 541. rustfmt and clippy -D warnings clean. |
|||
| 7d21532ebf |
feat(pdf): real colour spaces, ICC profiles and PDF functions
Phase 8 #4 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("Advanced colour"). Design and merge criteria in REVIEWS/adr/0006-pdf-advanced-color.md. The interpreter tracked only the *name* of the active colour space and then passed sc/scn operands to the device as if they were already RGBA. Every non-device space therefore rendered a confident wrong colour with no error: /Spot cs 1.0 scn full tint of a spot ink -> pure red /Idx cs 3 scn palette entry 3 -> near-black /Lab cs 50 0 0 scn mid gray -> white (clamped) /DevN cs (5 inks) five colorants -> inks 5+ discarded /ICCBased profile-defined colour -> profile discarded DeviceCMYK also used the additive 1-c-k conversion, which crushes any colour printed over black. Three new modules in pdf-graphics: - function.rs PDF functions, all four types. Type 4 runs on a bounded interpreter: depth 32, 32768 tokens, stack 100, 100000 steps, and an unknown operator is an error rather than a no-op that would leave a plausible wrong colour. - icc.rs ICC matrix/TRC and gray kTRC profiles, applied exactly. LUT-class profiles are reported as such and the caller falls back to /Alternate; they are never pretended to be matrix profiles. - colorspace.rs All eleven families, converting through XYZ with Bradford adaptation and a real sRGB transfer function. Wiring: - PdfDevice gains set_stroke_components/set_fill_components, so SC/SCN reach the device as components of the active space instead of being read positionally as RGBA. - cs/CS now resets to the space's initial colour (table 74), which is why golden/colors.txt gains a line. - PdfPage::color_spaces carries /Resources /ColorSpace fully dereferenced with streams decoded; a half-resolved space would make every ICC profile, palette and type 0/4 transform silently fall back. - A space that cannot be resolved keeps the previous colour and records a typed ColorError. No colour is invented, and no error is swallowed. Tests: 12 corpus fixtures under tests/corpus/color/, 14 acceptance tests in pdf-document/tests/color.rs asserting numeric RGB (the broken code produced a colour for every one of these; only the value was wrong), plus unit tests per function type and per curve type. Two fuzz targets added: eval_function and parse_colorspace. TEST_TARGET=pdf 387 -> 447 passing, TEST_TARGET=pdf-ui 431 -> 491. rustfmt and clippy -D warnings clean. |
|||
| 1c7dc9e9c0 |
feat(pdf): complete Phase 7 performance work
Phase 7 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md, taken only now because the review is explicit that it comes after correctness is proven. The caching and threading live in pdf-graphics rather than the Makepad crate because none of it needs a GPU. That is what lets the staleness, eviction and cancellation rules be tested without a window; the widget keeps only the parts that genuinely need a Cx. Step 7.1, off-thread parsing (new pdf-graphics/src/worker.rs): - RenderWorker interprets content streams on a background thread and returns results tagged with the Generation they were requested for. - PendingPages tracks in-flight pages so the widget can draw a placeholder and never queues the same page twice. - Drop joins the thread rather than detaching it: a detached thread writing into a dropped channel is the kind of shutdown race that surfaces as a flaky test months later. - A content stream that fails to parse yields an empty page, so one broken page cannot take down the document. Step 7.2, texture and memory management (new pdf-graphics/src/cache.rs): - PageCache is an LRU keyed by page index with a byte budget, not an entry count: one image-heavy page can outweigh fifty text pages, so counting entries would evict the wrong things. - retain_around() releases pages that scrolled out of view, keeping a margin so a small scroll does not immediately re-render. - Decoding produces DecodedImage bytes off-thread; GPU upload stays on the UI thread. - A page larger than the whole budget is still stored, since refusing it would mean re-rendering it every frame. Step 7.3, render command cache: - CachedPage holds the interpreted Vec<RenderCommand>, so replaying a page skips re-parsing its content stream. - invalidate_appearance() marks a form edit dirty without discarding the commands, because a form edit changes what is drawn over the page, not the page content stream. Widget wiring: PdfPageWidget carries a generation, refuses PageContent from a superseded document, and draws a placeholder while a page is still rendering. Exit criterion (new pdf-document/tests/phase7_exit_criterion.rs, 9 tests) against a new 60-page corpus fixture. Measured here: first page 3ms against the 200ms budget, and a warm cache read 389x faster than re-parsing (1us vs 389us). The budget is deliberately loose because a shared CI runner is unpredictable and a flaky performance test gets muted, and a muted test is worse than none; it still catches the order-of-magnitude regression the review is guarding against. Memory is asserted bounded across 30 document switches, and every page of an abandoned document is refused. Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh (270 tests) TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (307 tests) Both rustfmt and clippy -D warnings clean. |
|||
| eacc86077e |
feat(pdf): complete Phase 5 font engine and text metrics
Phase 5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md. Step 5.1, real metrics (new pdf-graphics/src/sfnt.rs): - Reads head, hhea, OS/2 and hmtx from embedded TrueType/OpenType programs, which the review asks for by name. sTypoAscender/Descender are preferred over hhea when non-zero, since subset fonts often zero them; sCapHeight and sxHeight are only read from OS/2 version 2 and later, because reading them from a v1 table returns whatever bytes follow it. - Every read is bounds-checked. A table pointing outside the file, a truncated directory or a zero unitsPerEm yields None rather than a panic or a divide by zero. These are untrusted embedded programs. - resolve_font() now reads FontFile/FontFile2/FontFile3 and falls back to the descriptors declared Ascent/Descent only when no program is embedded. - renderer.rs looks up a per-font ascent instead of leaving ascent_em permanently None, which had left the fallback ratio always in effect. Step 5.2, accumulated advances (new pdf-graphics/src/advance.rs): - measure_advance() implements PDF 32000-1 9.4.4 properly: (w0/1000 * Tfs + Tc + Tw) * Th/100, with Tw restricted to single-byte code 32. Applying Tw to a two-byte code whose low byte is 32 is a classic composite-font bug and is now covered by a test. - glyph_advances() gives per-glyph widths so a caret or a partial selection rectangle no longer assumes even spacing. - GlyphAdvanceCache is keyed by font name and size as the review specifies, and carries a generation so it rebuilds when page data changes rather than every frame. Redefining a font drops its stale measurements. Step 5.3, composite fonts: - The old build_cid_widths truncated every CID to u8, so any glyph above 255 silently took the default width. It is replaced by CidWidths keyed by the real CID, with the CMap resolved from a predefined name or an embedded stream. - ResolvedFont::decode_text returns None for a composite font with no ToUnicode map instead of guessing Latin-1 and producing plausible nonsense. Tests: new tests/phase5_exit_criterion.rs asserts the exit criterion. Selection across a proportional and a monospaced run verifies the second run starts at the first ones true end; per-glyph advances sum to the run advance and H measures wider than l, which an even-spacing estimate would not. Copy returns correct Unicode through ToUnicode including a non-ASCII scalar, and through WinAnsi for simple fonts. Search finds both hits across two lines with rectangles inside their own runs and the right vertical order. Cache reuse is asserted through hit and miss counts. Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh (203 tests) TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (238 tests) Both rustfmt and clippy -D warnings clean. |
|||
| 66c7590d66 |
feat(pdf): start Phases 4 and 5, and make pdf-makepad compile
Some checks failed
Payment domain and storage / isolated-payment-tests (push) Has been cancelled
Phases 4 and 5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md. pdf-makepad had never compiled. It was missing eight PdfDevice methods, had a non-exhaustive RenderCommand match, called a function through the wrong path and held a borrow conflict. Everything previously claimed about the Makepad rendering path was therefore unverified. It now builds and is tested under a new pdf-ui target; the makepad build needs system GUI libraries, so that target checks for them with pkg-config and names the missing packages rather than failing in the linker. Phase 4 (new pdf-makepad/src/interaction.rs): - Viewport maps between PDF space (y up) and screen space (y down) in one place, so hit testing and rendering cannot disagree. Inverted /Rect corners are normalised. - Click routing follows the review order: form fields first, then annotation hit testing. A field overlapping a link takes the click. - Clicking a link emits PdfAction::OpenUri to the host. The viewer never opens a URL itself (rule 5). - Text editing is buffered: typing changes a working copy and only Enter, Tab or blur commits it, so Escape abandons an edit with the document untouched. Caret arithmetic is in characters, not bytes, so a non-ASCII value cannot panic. - Checkboxes toggle on click using the state the widget declares. - Hover reporting for cursor feedback; read-only fields are not targets. - render_fields() returns placement data so a focused field shows its uncommitted buffer while drawing stays trivial. Phase 5: - New pdf-graphics/src/cid.rs: composite font support. Codespace ranges give variable-width code decoding (the previous CMap was u8 to char, so every two-byte CID font decoded as garbage), plus cidchar/cidrange, ToUnicode bfchar/bfrange including array destinations and multi-character values such as ligatures, /W and /DW CID widths, and Identity-H/V. An unknown predefined CMap returns None instead of silently substituting Identity; unmapped codes decode to U+FFFD instead of vanishing. Width ranges are bounded so a malformed /W cannot exhaust memory. - New pdf-graphics/src/text.rs: the PageText extraction API. Segments carry origin, advance and font size; find() returns matches with rectangles; selection spans runs and works in either drag direction. Runs sharing a baseline stay on one line even with mixed font sizes. - renderer.rs: the two unexplained `fs * 0.75` constants are replaced by a named DEFAULT_EM_TO_CAP_RATIO used only as a fallback, with the real ascent preferred when the descriptor supplies one. Ts (text rise) was stored but never applied, so superscripts drew on the baseline. Also fixes clippy across pdf-makepad, which had never been linted. Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh (167 tests) TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (191 tests) Both rustfmt and clippy -D warnings clean. |
|||
| 586826feed |
fix(pdf): add the Phase 2 interpreter sources omitted from b7d23f2
|
|||
| b7d23f26bc |
feat(pdf): complete Phase 2 render pipeline with golden tests
Phase 2 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md. The exit criterion is that a real content stream renders through the device path and is verifiable; previously nothing could fail, so nothing was proven. Step 2.3, the operations the review calls "the lies", all reached a dead end in the interpreter. Each now reaches the device: - Inline images: BI/ID/EI were emitted as two empty marker ops and the payload was discarded, so an inline image could never be drawn. They are now parsed into a single InlineImage op carrying the dictionary and bytes. Abbreviated keys (/W /H /BPC /CS /F) and colour-space and filter abbreviations are expanded. The EI scan requires delimiters on both sides so binary data containing the bytes "EI" does not truncate the image, and an unterminated image yields no image rather than invented pixels. - Do (XObject) was an empty match arm. The interpreter cannot resolve a resource name, so it now reports it through PdfDevice::paint_x_object and the device performs the lookup. - set_dash and set_miter_limit only mutated interpreter-local state and emitted no command, so dashes never reached any renderer. Three further defects surfaced while reviewing the generated goldens, all of which silently corrupted output rather than failing: - `cm` was never parsed at all. The single-character `m` arm matched first and consumed it as a moveto, so every CTM change in every document was lost and content drew at the wrong position. Two-character operators are now tested before their one-character prefixes. - `rg` and `RG` were shadowed by the `r` and `R` arms, so an RGB fill was read as a single-component grey: `0.1 0.2 0.3 rg` produced 0.1 0.1 0.1. - ImageInfo defaulted a missing /Filter to FlateDecode. An absent /Filter means the data is stored raw, so every uncompressed image was undecodable. Testing: adds format_commands(), a deterministic one-line-per-command text form of a RenderCommand list, and eight golden files covering vector paths, text, kerning and spacing, dash and stroke parameters, fill rules, inline images, XObjects and colour operators. Floats are fixed-precision and negative zero is normalised so no spurious diffs appear. UPDATE_GOLDEN=1 regenerates them for review. Also fixes .gitignore: blanket *.txt and *.pdf rules were silently excluding the golden expectations and the AcroForm fixture added in the previous commit, which would have left both test suites unable to run on a fresh clone. Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh 132 tests pass; rustfmt and clippy -D warnings clean. |
|||
| af4ef6f170 |
fix(pdf): restore a compiling, warning-free PDF baseline
Some checks failed
Payment domain and storage / isolated-payment-tests (push) Has been cancelled
Phase 0 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md requires a clean baseline before any feature work. The four pdf crates did not compile at all, so every test claim about them was unverified. Compile fixes: - decode_lzw/decode_run_length returned Vec<u8> where callers expected PdfResult, so decode_stream_with_params did not type-check. - interpret_ops called a current_state_mut() method that PdfDevice does not have; colour-space tracking now goes through explicit device hooks. - image.rs used miniz_oxide without depending on it; PNG inflate now reuses the COS crate through a new pdf_cos::filter::inflate_zlib. Correctness fixes found while making the code build: - LZW and RunLength silently truncated malformed input and indexed unchecked; both now return typed errors (rule 6: no silent degradation). - Tw/Tc/Tz were parsed and thrown away, and the " operator dropped its word and character spacing, so every advance after them drifted. - TJ attached each kern to the preceding string instead of the following one and discarded a trailing kern entirely. - GlyphWidths::width() fell back to default_width for out-of-range codes; PDF 32000-1 9.6.2.1 requires /MissingWidth. - Text advances silently substituted a guessed font_size * 0.6 when no width table was present; ShowTextWithMetrics now carries advance_is_measured so callers can distinguish a measurement from an unknown. Tests: two tests had never compiled and were wrong once they ran (WinAnsi 0x99 is U+2122 not U+2019; q/cm/l/Q records four commands not three). The document test asserted only that the writer emits a %PDF header; replaced with real page-tree, out-of-range and malformed-input coverage. Adds a pdf target to tools/test-rust-clean.sh that tests the three UI-independent crates bottom-up under rustfmt and clippy -D warnings. Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh 72 tests pass; rustfmt and clippy -D warnings clean. |
|||
| 99f73f587d | added updates | |||
| a2ea0ffc7c | updated map | |||
| cc05abdc71 | Initial commit |
