Phase 8 #7 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("PDF/UA
structure tree"). Design and merge criteria in
REVIEWS/adr/0011-pdf-structure-tree.md.
Investigating the accessibility gap surfaced four defects in the
marked-content operators the structure tree depends on. The first is not
an accessibility problem at all.
1. BMC never parsed, and corrupted the colour of everything after it.
The dispatcher matched b'B'+b'M' only when the third byte was a
delimiter; BMC's third byte is 'C', so the arm never fired. Because
the operator was never recognised it never CLEARED ITS OPERAND, and
the leftover /Tag shifted the operands of whatever came next:
1 0 0 rg -> RgbFill(1.0, 0.0, 0.0) red
/Span BMC 1 0 0 rg -> RgbFill(0.0, 1.0, 0.0) green
Tagged documents are precisely the ones containing BMC, so the
documents that tried hardest to be accessible rendered wrong colours.
This is the fifth instance of the operator-shadowing family already
fixed for cm, rg, gs, b/b* and end-of-stream text operators. ADR
0009's every_multi_char_operator_parses_as_itself test exists to stop
exactly this - and would have caught it, except BMC was one of two
operators excluded from its table as "genuinely unimplemented".
Excluding a known-broken operator from the test whose job is finding
broken operators is how it survived. The exclusion list is gone.
2. BDC discarded its property list, which carries /MCID - the only link
between a run of page content and the structure element describing
it. Without it a tree can be parsed but never attached to anything.
3. The op produced by BDC was named MarkContentBmc, and BMC produced
nothing. The names were the wrong way round, which is how the missing
arm survived review: the enum looked like it had a BMC case.
4. Found while fixing 2: the content lexer had no dictionary support at
all. It read `<` as a hex string without checking for a second `<`,
so `<</MCID 0>>` parsed as the string "0C0D0". Content streams now
parse direct objects properly, bounded at 16 levels; a single `<` is
still a hex string and a test pins that.
On top of that, new pdf-document/src/structure.rs: /StructTreeRoot,
/StructElem trees, /RoleMap resolution, depth-first reading order,
/Alt, /ActualText, /E, /Lang, and MCID-to-element lookup. Cycles in /K
are cut at the first repeat and reported by object number rather than
expanded to the depth bound.
Accessibility findings are mechanical checks reported as findings, NOT a
conformance verdict: there is no is_pdf_ua and no ComplianceReport, and
a mutation-checked tripwire test fails if either appears. Real PDF/UA
conformance needs human judgement - whether /Alt text is accurate is not
mechanically decidable - so claiming it from six checks would be exactly
the overclaim this codebase keeps removing.
7 corpus fixtures, 14 acceptance tests, 22 unit tests, and a
parse_content_dict fuzz target for the new object parser.
TEST_TARGET=pdf 536 -> 575, TEST_TARGET=pdf-ui 580 -> 619.
rustfmt and clippy -D warnings clean.
Phase 8 #6 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Signatures"). Design and merge criteria in
REVIEWS/adr/0010-pdf-signatures.md.
Two defects, and the first is not about signatures at all.
1. acroform() dereferenced /AcroForm with self.resolve(), which recurses.
That replaced every /Fields [5 0 R] entry with an inline dictionary,
so AcroForm::walk saw node.as_ref() == None, decided the field had no
identity to key an edit on, and dropped it. The form came back EMPTY
rather than wrong, which is indistinguishable from a document with no
fields, so nothing announced the loss.
This is the third appearance of one mistake: page_annotations once
destroyed every annotation's obj_ref the same way, and
extract_xobjects resolved a reference then asked the resolved object
for as_ref(), leaving every page's XObject map empty.
It was invisible because both existing fixtures declare /AcroForm as
an INDIRECT reference, where only one level is dereferenced and the
refs survive. A direct /AcroForm dictionary - equally legal - hits it.
The new fixture uses one deliberately; reverting the one-line fix
fails 9 of the 12 new acceptance tests.
2. Nothing read signatures. FieldType::Signature was classified and then
ignored; /ByteRange, /Contents, /SubFilter and /DocMDP appear nowhere
in the codebase. A signed contract was presented exactly like an
unsigned one.
New pdf-document/src/signature.rs reads the signature dictionary and
checks BYTE-RANGE INTEGRITY, which needs no cryptography and catches the
common real-world tampering: whether the signed range reaches the end of
the file. A signature that stops short leaves appended bytes uncovered,
which is exactly how an incremental-update attack hides content behind a
signature that still verifies.
Cryptographic verification is NOT implemented and cannot be faked:
VerificationStatus has no Valid variant. That is enforced by the type,
not by convention, because the failure mode for a signature feature is
not "it doesn't work" - it is a green tick beside a document nobody
checked. A test asserts the capability's absence so adding Valid without
the cryptography breaks the build rather than shipping a false tick.
Signing is out of scope entirely: no private keys in this crate.
Also verified ADR 0003's claim that an append-only save preserves a
signed byte range, byte for byte, rather than leaving it asserted.
Implementation bug worth recording: /Contents was initially hex-decoded,
but the COS lexer already decodes <...>. Running it twice on the common
128-zero-byte placeholder - which contains no hex digits - produced an
EMPTY vector, silently discarding the signature blob while every other
field looked right. Caught by asserting the blob is non-empty rather
than asserting the parse returned Ok.
6 corpus fixtures, 12 acceptance tests, 28 unit tests, and a
parse_signature fuzz target because /ByteRange is four
attacker-controlled integers used to index the file.
TEST_TARGET=pdf 497 -> 536 passing. rustfmt and clippy -D warnings clean.
Phase 8 #5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Advanced transparency"). Design and merge criteria in
REVIEWS/adr/0009-pdf-transparency.md.
The headline defect was not that transparency was missing. `gs` was
MIS-PARSED as CloseStroke:
/GS0 gs 1 0 0 rg 100 100 200 200 re f
-> [CloseStroke, RgbFill, Rectangle, FillWinding]
so every page setting a graphics state gained a stroked path the
document never asked for, and lost its alpha, blend mode and soft mask
silently. Downstream everything was dead: StrokeExtGState/FillExtGState
were never constructed, set_fill_opacity was never called and emitted no
command when it was, and PdfPage::ext_gstate was read by no code at all.
New pdf-graphics/src/transparency.rs: full /ExtGState (ca, CA, BM,
SMask, LW, LC, LJ, ML, D, AIS, TK), all sixteen blend modes including
the four non-separable ones, soft masks with /S, /G, /BC and a /TR
evaluated through ADR 0006's PdfFunction, and /Group parsing. Wired end
to end: parser -> PdfOp::SetExtGState -> device -> RenderCommand ->
Makepad renderer.
Compositing is NOT claimed. Backdrop blending needs render-to-texture,
which an engine-neutral crate has no framebuffer for. Constant alpha is
applied because it needs no backdrop; blend modes and soft masks are
reported through TransparencyError::Unsupported rather than dropped,
because a silently ignored /Multiply looks exactly like a correct
/Normal.
The ADR required a test enumerating every multi-character operator, on
the grounds that fixing the third instance of a shadowing bug (after cm
and rg) without preventing the fourth is not a fix. It immediately found
three more live bugs, none of them transparency-related:
- `b` and `b*` dropped their close-path, mapping to FillStroke* instead
of CloseFillStroke*, so every closed-and-stroked path drew with a gap.
- Text operators within three bytes of the end of a content stream were
mis-parsed: the b'T' arm guarded `*i + 3 < len` while reading only two
bytes, so a stream ending in `/F1 12 Tf` parsed as FillWinding.
And the transparency-group fixture found a fourth:
- PdfPage::xobjects was empty for every page of every document.
extract_xobjects called doc.resolve(), which follows the reference,
then asked the resolved object for as_ref() - always None - and only
accepted a bare dict when every XObject is a stream. No `Do` operator
could be resolved through the page model. Same defect as the one that
once destroyed annotation object references; it now has its own
regression test.
T* and BMC are genuinely unimplemented and are deliberately excluded
from the guard's table rather than papered over.
7 corpus fixtures, 12 acceptance tests, 32 unit tests asserting the
§11.3.5 formulas (not our own output), and a parse_ext_gstate fuzz
target.
TEST_TARGET=pdf 451 -> 497, TEST_TARGET=pdf-ui 495 -> 541.
rustfmt and clippy -D warnings clean.
Phase 8 #4 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Advanced colour"). Design and merge criteria in
REVIEWS/adr/0006-pdf-advanced-color.md.
The interpreter tracked only the *name* of the active colour space and
then passed sc/scn operands to the device as if they were already RGBA.
Every non-device space therefore rendered a confident wrong colour with
no error:
/Spot cs 1.0 scn full tint of a spot ink -> pure red
/Idx cs 3 scn palette entry 3 -> near-black
/Lab cs 50 0 0 scn mid gray -> white (clamped)
/DevN cs (5 inks) five colorants -> inks 5+ discarded
/ICCBased profile-defined colour -> profile discarded
DeviceCMYK also used the additive 1-c-k conversion, which crushes any
colour printed over black.
Three new modules in pdf-graphics:
- function.rs PDF functions, all four types. Type 4 runs on a bounded
interpreter: depth 32, 32768 tokens, stack 100, 100000
steps, and an unknown operator is an error rather than a
no-op that would leave a plausible wrong colour.
- icc.rs ICC matrix/TRC and gray kTRC profiles, applied exactly.
LUT-class profiles are reported as such and the caller
falls back to /Alternate; they are never pretended to be
matrix profiles.
- colorspace.rs All eleven families, converting through XYZ with
Bradford adaptation and a real sRGB transfer function.
Wiring:
- PdfDevice gains set_stroke_components/set_fill_components, so SC/SCN
reach the device as components of the active space instead of being
read positionally as RGBA.
- cs/CS now resets to the space's initial colour (table 74), which is
why golden/colors.txt gains a line.
- PdfPage::color_spaces carries /Resources /ColorSpace fully
dereferenced with streams decoded; a half-resolved space would make
every ICC profile, palette and type 0/4 transform silently fall back.
- A space that cannot be resolved keeps the previous colour and records
a typed ColorError. No colour is invented, and no error is swallowed.
Tests: 12 corpus fixtures under tests/corpus/color/, 14 acceptance
tests in pdf-document/tests/color.rs asserting numeric RGB (the broken
code produced a colour for every one of these; only the value was
wrong), plus unit tests per function type and per curve type. Two fuzz
targets added: eval_function and parse_colorspace.
TEST_TARGET=pdf 387 -> 447 passing, TEST_TARGET=pdf-ui 431 -> 491.
rustfmt and clippy -D warnings clean.
Phase 8 feature 3 of 10, designed in REVIEWS/adr/0005-pdf-encryption.md.
An encrypted PDF did something worse than fail: it succeeded. Probing a
structurally valid RC4-encrypted file through the parser gave
parsed OK: pages=1
page 0 content bytes=44
content parsed into 0 ops
No error and no warning. The document reported a page, the page reported
content, and the content interpreted to nothing because it was ciphertext.
The user saw a blank page and was told the file was fine. That is the defect
class Phase 0 existed to remove, and it was the worst one left in the PDF
stack because it was silent.
New pdf-cos/src/encrypt.rs implements the standard security handler for
reading:
- V1/R2 RC4 40-bit, V2/R3 RC4 40 to 128-bit, V4/R4 crypt filters selecting
RC4 or AES-128, and V5/R6 AES-256 with the SHA-256 based revision 6 hash.
- The empty user password, which is the common case for a document
encrypted only to set permissions, and explicit user or owner passwords.
The owner path recovers the user password from /O and re-derives.
- Per-object keys, as the spec requires. Reusing one keystream across
objects would be a real cryptographic break, so the object and generation
numbers are mixed in by construction and a test asserts the keys differ.
Every primitive comes from audited RustCrypto crates: aes, cbc, rc4, md-5
and sha2, all MIT OR Apache-2.0, which deny.toml already permits. Phase 0
deleted a hand-rolled MD5/SHA/AES/RC4 implementation from this codebase and
called it a CVE factory; ADR 0005 keeps that rule.
Refusals rather than half-open documents: a public-key or otherwise
unsupported handler is refused and named, an unsupported V/R combination is
refused, and a wrong password returns a distinct error so a caller can
prompt again rather than reporting a damaged file.
Permissions are parsed and exposed but deliberately not enforced, and the
code says why: once content is decrypted a caller can read it regardless, so
enforcing here would imply a guarantee that does not exist.
Saving an encrypted document stays refused, as ADR 0003 established.
Decrypting and then writing plaintext would silently strip the protection
the author applied, which is not a decision a library should make.
Fixtures: tests/corpus/encrypted/ gains RC4 40-bit, RC4 128-bit, AES-128 and
an unsupported-handler document. The generator implements the handler's
algorithms independently from the specification, so a fixture that decrypts
shows the reader agrees with the spec rather than merely with itself. Each
plaintext contains a marker the tests assert on, and one test additionally
asserts the decrypted content interprets to real render commands, because
asserting Ok from the parser is exactly what the old broken behaviour did.
Fuzzing: adds a decrypt target covering key derivation, which consumes
attacker-controlled /O, /U, /P, /Length, filter names and file id. Run for
real rather than compile-checked: 1,953,940 executions, no crashes.
Validation:
TEST_TARGET=pdf ./tools/test-rust-clean.sh (383 tests)
TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (427 tests, 6 ignored)
Both rustfmt and clippy -D warnings clean.
Incremental save made /Prev chain walking load-bearing for every document,
not only saved ones: XRefTable::parse now follows offsets taken straight from
the file on every parse. Phase 6 established that code consuming untrusted
input needs corpus and fuzz coverage. That path had neither, so this adds it
and fixes what it found.
Corpus (7 new fixtures, generated by the checked-in script as usual):
- revisions/two.pdf, three.pdf: chained revisions that override a form
value. These are direct regression tests for the two bugs the previous
commit fixed. Before it, two.pdf read back as "first" rather than
"second", because find_xref_start never matched its own keyword and fell
through to the oldest section in the file.
- revisions/added_page.pdf: a revision that rewrites /Pages, so the newer
definition must win for structure as well as for values.
- malformed/prev_loop.pdf, prev_out_of_range.pdf, prev_negative.pdf and
prev_chain_bomb.pdf: the hostile shapes.
Two robustness defects found by those fixtures:
- A broken /Prev orphaned every object the unreachable sections defined,
even though the bytes were still in the file, so a document with one bad
offset failed to open at all. The chain now sets a recovered flag and
sweeps the file for object headers, filling only genuine gaps: entries a
parsed section supplied always win, because those reflect the document's
own view of which revision is current, and scanning cannot tell newer
from older.
- A negative /Prev was filtered to None, which silently ended the chain as
though the file had no history. It is now treated as a broken link and
triggers the same recovery.
Also caps the chain at 64 revisions. A legitimate document has a handful; a
file with thousands is an attack, not a history. prev_chain_bomb.pdf asserts
the cap holds and that parsing stays fast.
The recovered flag is public so a caller can distinguish a cleanly parsed
document from a salvaged one rather than being handed a guess silently. A
test asserts it stays false for healthy files, or it would mean nothing.
Fuzzing: adds parse_revision_chain, which splices fuzzer input onto a valid
base document so the fuzzer spends its time on chain shapes rather than on
rediscovering PDF syntax. Run for real, not merely compile-checked:
parse_revision_chain 1,926,164 runs
parse_xref 1,387,713 runs
parse_document 1,279,328 runs
No crashes. The two re-run targets cover the file this commit changes.
One fixture-generator bug fixed on the way: the helper that reads a file's
startxref took the first token after rfind without skipping the keyword,
producing a startxref that pointed at its own text.
Validation:
TEST_TARGET=pdf ./tools/test-rust-clean.sh (318 tests)
TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (362 tests, 6 ignored)
Both rustfmt and clippy -D warnings clean.
Phase 6 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md: "Real PDFs. Real
regressions. No test theater."
Step 6.1, corpus. 27 fixtures across basic, fonts, forms, annotations,
images, edge and malformed, in the layout the review specifies. They are
produced by tests/corpus/generate.py rather than committed as opaque blobs,
because a corpus you cannot read is a corpus you cannot trust; CI regenerates
them and fails if they differ. Hand-rolled rather than library-produced,
since fixtures for a parser must contain constructs a library refuses to
emit.
Step 6.2, corpus tests (pdf-document/tests/corpus.rs, 27 tests). Text,
vectors, Flate, multipage, CID fonts, every form field type, inherited field
keys, link actions, hidden annotations, XObjects, inline images with
embedded EI bytes, rotation, crop boxes, nested CTMs and content arrays.
Step 6.3, robustness (pdf-document/tests/robustness.rs, 3 tests) plus five
cargo-fuzz targets. cargo-fuzz needs nightly and libFuzzer so it cannot gate
a stable CI run; the harness covers the same ground deterministically by
mutating the real corpus with a fixed-seed PRNG, so a failure is reproducible
from the seed rather than only from a saved artefact. The fuzz targets remain
the deeper coverage-guided search and run on a schedule.
Three crashes on untrusted input, all found by this work and all previously
reachable from a malformed file:
- collect_pages_ref recursed forever on a /Kids cycle. Stack overflow aborts
the process; it cannot be caught. Now tracks visited nodes and bounds depth.
- PdfDocument::resolve and the COS lexer recursed once per nesting level, so
a file of 5000 open brackets overflowed the stack. Both are now bounded.
- decode_85_group multiplied an accumulator that a malformed group can
overflow, and subtracted below zero on a digit outside the valid range.
Both panic in a debug build. Now saturating.
Step 6.4, CI (.forgejo/workflows/pdf.yml). An engine job that runs the
corpus and robustness suites under rustfmt and clippy -D warnings; a separate
makepad-integration job so a missing system library is not reported as a PDF
regression; and a scheduled fuzz job. The engine job also enforces the two
architectural rules mechanically rather than in prose: no Makepad dependency
or import in the engine crates, and no process or URL launching anywhere in
them.
Also fixes .gitignore: the blanket *.pdf rule silently excluded all 27
fixtures, which would have left CI unable to run them on a fresh clone.
Validation:
TEST_TARGET=pdf ./tools/test-rust-clean.sh (233 tests)
TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh (268 tests)
Both rustfmt and clippy -D warnings clean; all five fuzz targets compile.
