Commit graph

31 commits

Author SHA1 Message Date
c1d1e67f3a feat(pdf): shadings — the sh operator was parsed and thrown away
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
ADR 0028, the first of Phase 7's eight bullets.

content.rs contained `PdfOp::Shading(_name) => {}`. The operator was lexed,
given its own variant, matched during interpretation, and discarded. A page
whose background is a gradient rendered as nothing.

Nothing caught it for the usual reason: a blank region is a legal thing for
a page to contain, so "drew nothing" and "drew what was asked" are
indistinguishable without an assertion naming the expected colour. The
golden corpus had no shading page, so there was nothing to be wrong.

Two of the three pieces already existed — function.rs evaluates the colour
function and colorspace.rs converts it to RGB. What was missing was the
geometry between them.

Sampling rather than a gradient primitive: a PDF shading is defined by an
arbitrary function, possibly a sampled table or a PostScript program, and
neither reduces to a stop list without loss. A device with a native
gradient can still recognise the two-stop case from the samples.

"No colour here" is None, not black. Black is a colour a shading can
legitimately produce, so returning it for "outside an unextended shading"
would paint a rectangle the author never asked for and the caller could not
tell the two apart.

Types 1-5 exact. Coons and tensor patches are flattened to their corners,
which loses the curvature, and is_approximate says so rather than leaving a
caller to assume fidelity. An unknown type is refused by number: a mesh
drawn as a flat fill is a plausible-looking wrong answer.

paint_shading is a new trait method, so the compiler found every
implementor. The Makepad renderer records the request in pending_shadings,
mirroring pending_xobjects — it cannot resolve a /Shading resource because
it does not own the page dictionary, and recording the request is what
stops the operator vanishing a second time. That holds even for types we
refuse, so a host can warn the user.

Four mutations, all killed. The first — discarding sh again — fails three
tests.

Stated plainly and left unticked: the mesh path is written but NOT
exercised by any real stream. shading.rs is at 68% and the uncovered part
is exactly parse_mesh and triangulate. Mesh support should be treated as
unproven, not working: the code runs and produces triangles, and nothing
yet demonstrates they are the right triangles. That is the position
image.rs was in before ADR 0016 found the JPEG decoder was a stub.

The Phase 7 status line is a table from the start this time — one row per
spec bullet, seven of them saying "not started". Per ADR 0021, written
before the work rather than after it.

pdf: 1321 passed (was 1291). pdf-ui: 1366. Coverage 87.60%, floors met.
2026-08-18 17:30:39 +00:00
cb8912f762 test(pdf): close the two real coverage gaps in the signing module
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
repo hygiene / hygiene (push) Has been cancelled
Asked to verify Phase 6 was complete *with test coverage*, I measured
sign.rs per function rather than trusting the file-level 82%. Most of the
apparent gap is error arms inside covered functions — llvm-cov attributes
each `map_err` closure separately — but two things were genuinely untested,
and one of them was not code that should exist.

  algorithm_name() was dead. It returned a &'static str describing the
  algorithm and nothing called it: `algorithm()` supersedes it, returns a
  type rather than a string, and is what the CMS writer actually uses.
  Deleted rather than tested, because a test would have preserved code
  whose only caller was the test.

  SigningError's Display impl was never exercised. These strings reach a
  user through a host application. ContentsTooSmall in particular must
  carry both numbers — a caller cannot raise the reservation without
  knowing by how much — and that is now driven through the real signing
  path with a 32-byte reservation rather than by constructing the error.

sign.rs 82.07% -> 83.79%. pdf: 1291 passed. Coverage 88.03%, floors met.
2026-08-18 17:08:11 +00:00
374af5ccad feat(pdf): the five Phase 6 bullets the status line omitted
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
ADR 0027. Asked whether Phase 6 was 100% complete, I checked the plan's
bullets against the code instead of answering from the status line. Five
were not implemented and the status line named none of them:

  detached/ATTACHED signatures   /SubFilter hardcoded to adbe.pkcs7.detached
  PAdES basics                   ETSI.CAdES.detached was a string in a match
  external_signing_test.dart     absent; SigningIdentity needs an in-memory key
  OCSP/CRL lookup                CRL only; OCSP counted, never parsed
  Fulcio identity                absent (optional in the plan)

This is the second time. ADR 0021 recorded the same failure in Phase 4 and
wrote the rule meant to prevent it — enumerate criteria from the plan text
first, then mark each done or explicitly deferred. I wrote that rule and
then produced another prose summary of what I had built. A summary written
from the work cannot show what the work omitted.

PAdES is a real profile, not a label. CAdES signs a set of signed
attributes, one carrying the document digest, and the signature is over
those attributes re-tagged as a SET (RFC 5652 5.4) rather than over the
[0] IMPLICIT SEQUENCE they are carried in. Verification checks the
messageDigest attribute against the document as well as verifying the
attribute signature; without that, a signature over somebody else's digest
would be accepted. /SubFilter now comes from the profile, so a document
cannot claim CAdES while carrying plain PKCS#7.

ExternalSigner is a trait: bytes in, signature out. A smartcard or KMS
never hands out its key, so SigningIdentity could not represent one.
SigningIdentity implements the trait rather than sitting beside it, so
there is one signing path — a second path for hardware keys would be a
second place the byte range could be computed differently.

OCSP is decoded with the der crate already present rather than adding the
ocsp crate for two fields. Revoked from any response beats Good from any
other.

Attached signatures are REFUSED, not deferred. Both attached profiles
(adbe.pkcs7.sha1, adbe.x509.rsa_sha1) are SHA-1 based, and SHA-1 is broken
for signatures. They are parsed so such documents can be read; they cannot
be written, enforced by the absence of a SignatureProfile variant. Same
decision as RC4 in ADR 0024. Recorded as refused rather than not-done,
because "not done" invites someone to finish it.

Four mutations, all killed first attempt: messageDigest not compared,
CAdES verified against the wrong bytes, /SubFilter hardcoded again, OCSP
revoked read as good.

The status line is now the plan's own bullets in a table, one row per spec
item, not prose. Two wrong status lines in the same direction is a pattern,
and the fix is structural: a missing row is visible, a missing sentence is
not. Four rows are left unticked — Fulcio, independent review, Acrobat
interoperability, and signing a document that already has an AcroForm.

qpdf accepts documents under both profiles. pdf: 1289 passed (was 1276).
Coverage 87.96%.
2026-08-18 12:06:59 +00:00
99aebc202a fix(pdf): security review of the signing code — a forgery verified as valid
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
ADR 0026. Both ADR 0024 and ADR 0025 said this code needed a security
review before shipping. This is that review, done adversarially: for each
way a signature could be defeated, a test that attempts it. It found a
critical vulnerability in the code as shipped last turn.

FINDING 1, critical, exploitable with no special access.

Verification recovered the certificate and the signature by *scanning* the
blob for DER-shaped bytes rather than decoding it. The signature was checked
against certificates[0]; trust was checked against ANY certificate present.
Two questions, two different certificates. So:

  the attacker signs a forgery with their own key
  the attacker appends the victim's trusted certificate to the blob
  signature_valid = true   (their signature over their own content is real)
  chain_trusted   = true   (the victim's certificate is present)
  is_valid()      = true

Demonstrated before the fix, with the message "I hereby transfer everything
to the attacker" verifying as valid.

Fixed by decoding the ContentInfo/SignedData structure and finding the
certificate the SignerInfo actually names, by issuer AND serial, then
evaluating both the signature and the trust path against that one
certificate. Trailing data now fails the decode instead of being ignored.
The scanning functions are deleted, not left unused: dead code that once
returned the wrong answer is an invitation to call it again.

FINDING 2, moderate. signer_certificate() returned chain[0] unconditionally,
so a chain whose first entry was not the signing key's certificate made the
SignerInfo name the wrong one. Not a forgery route — the signature fails —
but a UI showing "signed by <somebody trustworthy>" beside a failed check is
its own kind of dangerous. Now it finds the entry whose public key matches
the key doing the signing.

FINDING 3, informational. digest_matches was hardcoded true under a comment
claiming it was computed. Not exploitable, because is_valid() also requires
signature_valid and the signature covers the bytes — but a field asserting
an unperformed check is ADR 0017's pattern exactly.

The four items ADR 0025 left unticked are closed:

  PKIX chain building, with each link's issuer signature verified. A name
  match alone is not a chain; anyone can put any name in a certificate.
  Pinning still short-circuits first.

  Stapled revocation from /DSS, offline only. Unknown is the default and a
  first-class answer: treating "no information" as "not revoked" is a claim
  a verifier cannot support.

  Signature appearances, with the claimed time labelled "Time claimed"
  because a self-declared /M carries no authority.

  One-call sign_document. Three things were wrong first: the /ByteRange
  placeholder was too narrow for real offsets so patching them moved every
  later byte; /Contents must be a hex string because a literal full of NULs
  needs escaping and changes length; and a signature dictionary nothing
  points at is invisible — the first version wrote one and the reader
  reported zero signatures over a correctly signed document.

Four mutations, all killed — two only after strengthening the tests. My
first smuggling test put the attacker's certificate first, where
certificates[0] finds it anyway, so it passed with or without the
issuer/serial match. Putting the TRUSTED certificate first is what
distinguishes them, and writing that test is what exposed Finding 2.

qpdf --check accepts the signed documents. pdf: 1276 passed. Coverage 87.98%.

Left unticked, deliberately: an independent review by someone who did not
write the code. This is a self-review; it found two real vulnerabilities,
which is evidence the method works and not evidence that nothing remains.
Also untested against Acrobat, which is stricter than the spec, and
sign_document replaces rather than merges an existing AcroForm.
2026-08-18 10:39:20 +00:00
9a5ce9c0e6 feat(pdf): signing and verification — Valid becomes reachable, with a policy
Some checks failed
nigig-build (CAD) / supply-chain (push) Has been cancelled
nigig-build (CAD) / cad-module (push) Has been cancelled
nigig-build (CAD) / full-crate-check (push) Has been cancelled
nigig-build (CAD) / cad-engine-coverage (push) Has been cancelled
nigig-build (CAD) / doc-workspace-coverage (push) Has been cancelled
repo hygiene / hygiene (push) Has been cancelled
email / gates (push) Has been cancelled
email / email-domain (push) Has been cancelled
email / nigig-email (push) Has been cancelled
email / supply-chain (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
doc-engine / engine (push) Has been cancelled
doc-engine / coverage (push) Has been cancelled
doc-engine / consumer (push) Has been cancelled
nigig-map / test (push) Has been cancelled
sms / gates (push) Has been cancelled
sms / robius-sms (push) Has been cancelled
sms / android (push) Has been cancelled
sms / nigig-sms (push) Has been cancelled
sms / supply-chain (push) Has been cancelled
ADR 0025, completing Phase 6's functional core. This partially reverses
ADR 0010, which refused signing and cryptographic verification outright,
and it reverses only the half whose justification expired.

ADR 0010 gave two reasons. The first — a parsing library has no business
signing — stopped being true when Phase 4 began creating documents and
Phase 5 editing them. The second is still true and is preserved intact:

  deciding which certificate authorities to trust is a policy decision
  that belongs to the host, not to a parsing library

So VerificationStatus::Valid is still not reachable by default. Verification
returns three independent booleans and is_valid() needs all three; the
third, chain_trusted, can only become true through a caller-supplied
TrustAnchors. There is no TrustAnchors::system(), no bundled root store, no
Default that trusts anything. A caller with no policy is told
"cryptographically intact, signed by somebody you have not said you trust"
— a different fact from "forged", and a host that cannot tell them apart
shows the wrong thing to a user.

RSA PKCS#1 v1.5, ECDSA P-256 and Ed25519, all with SHA-256. PSS is stronger
and not universally accepted by PDF verifiers, so v1.5 is what is written.
Ed25519 carries an interoperability caveat in the doc comment on the
variant itself, because that is where someone choosing it will read it:
ISO 32000-2 does not list it and most desktop viewers will reject it.

No network. Revocation is not implemented rather than smuggled in: the
engine crates are CI-gated against reaching outward, and that gate is a
rule about layering, not an obstacle to work around.

Every test generates a real key and a real certificate at run time. Nothing
asserts against a checked-in blob — a fixed expectation only proves the
code still does what it did, which is the wrong question for a signature.
The tampering tests assert the signature verifies FIRST, then flip a bit;
without that half they could pass by never verifying anything.

Four mutations, all killed. The one that matters is the first: making an
empty anchor set confer trust is exactly the regression that would turn
this back into the thing ADR 0010 refused, and it fails immediately.

Two bugs the tests found:

  UTCTime cannot encode a year past 2049 (RFC 5280 4.1.2.5.1). The first
  fixture used a 2096 expiry and every certificate failed to encode.

  The certificate scanner assumed a two-byte DER length. RSA certificates
  are large enough to use that form, so RSA and P-256 passed while Ed25519
  found no certificate at all — its certificate is small enough for the
  short form. A scanner tested only against the largest input fails
  silently on the smallest.

72 dependency packages pulled in, zero non-compliant licences, no C.

Stated plainly and left unticked in the ADR: chain_trusted is anchor
identity matching, not PKIX path building. Correct for certificate pinning,
a false negative for a real CA hierarchy. Also outstanding: revocation,
signature appearance generation, and one-call incremental signing.

pdf: 1247 passed. Coverage 88.08%, floors met.
2026-08-18 09:45:01 +00:00
b870dc4c69 feat(pdf): outline, page label and struct-tree editing — Phase 5 complete
Some checks failed
email.yml / feat(pdf): outline, page label and struct-tree editing — Phase 5 complete (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
The last functional item. `catalog.rs` read all three; nothing could write
them into an existing document. `PdfDocBuilder` can emit an outline when
*creating* a file, but a document already on disk could not have its
bookmarks changed.

An outline is a doubly-linked tree — `/First`, `/Last`, `/Next`, `/Prev`,
`/Parent` and a signed `/Count` — and every pointer has to agree. A viewer
walking `/Next` and one walking `/First`..`/Last` must see the same list,
or bookmarks vanish in one reader and not another with no error anywhere.
Object numbers are reserved before any dictionary is built, because each
item names its parent, its siblings and its children.

`/Count` is signed and that matters: positive means open and counts
*visible* descendants, negative means closed. A closed child contributes
itself but hides its own children. Writing the total unconditionally makes
every node render expanded.

Page labels are a number tree, so the keys are sorted before writing and
two rules starting on the same page are refused — that page's label would
be undefined, and picking one arbitrarily is worse than saying so.

Struct-tree editing is deliberately **removal only**. Editing the tree in
place means rewriting `/K` arrays whose entries are marked-content ids
inside page content streams; the tree and the content must stay in step,
and changing one without the other produces a document whose accessibility
information describes content that is no longer there. Removal is honest —
the document stops claiming to be tagged — and `/MarkInfo` goes with it,
because `/Marked true` with no tree tells a screen reader there is
structure to find.

Verified by mutation, seven defects, all caught:

  /Prev never written              1 fail
  /Next never written              5 fail
  /Count always positive           1 fail
  page validation skipped          2 fail
  /MarkInfo left behind            1 fail
  children not linked via /First   2 fail
  label rules not sorted           1 fail

The last one needed a new test. Our reader walks `/Nums` linearly, so it
tolerates any order and the round-trip passed unsorted — but a conforming
reader binary-searches it and would label pages arbitrarily. Only reading
the raw array catches that, which is the same lesson as the stale `/Count`
in the page-ops tranche: our parser's tolerance hides defects that harm
other readers.

Engine suite 1158 -> 1187. External readers still pass.

**Phase 5 is complete** but for the `ui.rs` interaction tests, blocked on
the same missing Makepad headless backend as Phase 4's. The plan records
the item-by-item status and, separately, the six defects the round-trip
tests found in code that already existed — an unordered dictionary writer
that made every generated PDF differ run to run, a short /Length that
silently truncated streams, two readers disagreeing by a byte, a nested
paren that truncated a string and desynchronised the stream, undecoded #
escapes in names, and unknown operators being dropped outright.
2026-08-17 12:26:29 +00:00
41c43df0e5 feat(pdf): redaction and object compaction — and three reader defects
Some checks failed
email.yml / feat(pdf): redaction and object compaction — and three reader defects (push) Failing after 0s
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
repo hygiene / hygiene (push) Has been cancelled
Phase 5's last two functional items. Together they are what makes a
redaction real, which is why they are one commit: redaction removes the
content, compaction removes the revision that still holds it.

**Redaction removes operators; it does not draw rectangles.**

The famous failure is painting black over text and shipping it — the text
is still there, and `pdftotext` prints it. This module draws nothing. It
removes the text-showing operators whose position falls inside a
rectangle, and the test that matters is that the text can no longer be
extracted.

Positioning needs the text matrix, so the module tracks `Tm`/`Td`/`TD`/`T*`
and the CTM through `q`/`Q`/`cm`. It cannot reach the graphics layer — the
crate boundary again — so it treats a showing operator's origin as its
position and removes the whole run. That is coarse in the *safe*
direction: removing more than asked loses content the user can see is
missing; removing less leaves the secret in the file.

What it refuses to claim is as important. Images are removed entirely
rather than cropped. Metadata and attachments are untouched. And an
incremental redaction leaves the original text in the earlier revision —
the report says so via `earlier_revisions_retain_content` rather than
implying the job is done.

**Compaction finishes it.** The output is built from the object graph
reachable from `/Root`, so dead objects, superseded revisions and the
bytes behind a redaction are not copied — they are simply never written.
A signed document is refused unless `allow_signed` is set, because
compaction destroys the revision a signature covers and would leave every
signature unverifiable with no warning.

The end-to-end test is the point: redact, compact, then search the output
bytes for the secret. It is gone.

**Three reader defects, all found by writing the tests.**

- **`PdfWriter` wrote dictionary keys unordered.** `PdfDict` is a HashMap
  and Rust seeds its hasher per process, so *every generated PDF differed
  run to run*. Found by compaction's idempotence test — compacting an
  already-compact file produced the same objects at the same offsets with
  their keys shuffled. Verified fixed by running four separate processes
  and getting a byte-identical file. Same defect as the one fixed in
  `content_edit::write_dict`; this one affected every file this codebase
  has ever written.

- **A short `/Length` silently truncated a stream.** The reader guarded
  against a `/Length` running past the buffer but trusted one that was too
  small, cutting the stream at the wrong place and losing the rest with no
  error. Short lengths are common in hand-edited files. `endstream` is now
  the authority when the two disagree — but only when it is *further* on,
  so binary data containing the word `endstream` is still bounded by its
  declared length.

- **Two stream readers disagreed by one byte.** `read_object_at` did not
  trim the EOL before `endstream` while `find_endstream` did, so a
  write-read-write cycle grew every stream by a newline. A test fixture
  had encoded the bug: it declared `/Length 9` for eight bytes of content
  and asserted the newline came back as data. Both corrected — the
  newline is syntax (§7.3.8.1), not content.

Verified by mutation. Ten defects across the two modules, all caught:

  redaction covers instead of removes    16 fail
  CTM ignored                             1 fail
  Q does not restore the CTM              1 fail
  operands kept when operator removed    12 fail
  revision warning always false           1 fail
  signature guard removed                 1 fail
  reachability keeps everything           2 fail
  dropped reference left dangling         1 fail
  unresolvable object kept as reachable   1 fail
  writer dictionary order unsorted        1 fail
  short-/Length fix reverted              1 fail
  /Length not rewritten on compaction     1 fail

One mutation survived and deleted code rather than adding a test: a
`continue` skipping `/Length` in the compaction loop was dead, because the
`set` after the loop overwrites it either way. Removed rather than left as
untested defence with a reassuring comment — the same call ADR 0017 made
about the visited-set guard.

A second mutation moved a test rather than a fixture: a stale `/Length`
can no longer reach `renumber` through a file, because the reader now
repairs it first, so that branch is tested directly instead.

Engine suite 1108 -> 1158. External readers still pass.

Phase 5 remaining: outline, page label and struct-tree editing.
2026-08-17 12:20:25 +00:00
e0d86274d8 feat(pdf): flatten annotations and form fields into page content
Some checks failed
email.yml / feat(pdf): flatten annotations and form fields into page content (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 5's `flatten_test.dart`. An annotation draws from its /AP /N stream,
which lives beside the page rather than in it; flattening moves that
appearance into the page's own content and removes the annotation, so what
is drawn is part of the page and cannot be turned off, edited or extracted
as a field. That is what "finalise this form" and "make these comments
permanent" mean, and it is irreversible by design.

The placement transform is the whole problem, and it is §12.5.5: transform
the /BBox by /Matrix, take the bounding box of the *result*, then fit that
onto /Rect. Skip a step and the stamp lands at the origin, or in the right
place at the wrong size, and the page still renders. A rotated appearance
is the case that exposes it — rotation swaps the transformed box's width
and height, so fitting the untransformed box squashes it.

What is refused matters as much as what is done:

- **No appearance stream**: left in place and reported. Dropping it loses
  it; inventing an appearance draws something the producer never specified.
- **Hidden or /NoView**: not drawn on screen, so burning it in would *add*
  ink the user never saw.
- **A /Popup**: the pop-up window of another annotation, never drawn on the
  page itself.

Appearances are painted as XObjects rather than having their operators
spliced in. Splicing needs the stream's resources merged into the page's
with every name collision renamed, and it loses the /BBox clip an XObject
applies for free.

`pdf-document` cannot depend on `pdf-graphics` — the crate boundary is
cos -> document -> graphics and inverting it to reuse `write_ops` would be
a far worse trade than emitting the four operators (`q`, `cm`, `Do`, `Q`)
directly. The number formatting follows the same shortest-exact rule as
`content_edit::write_real`, and for the same reason.

Verified by mutation, five defects, each confirmed red:

  placement matrix ignored          1 fail
  /BBox not transformed first       2 fail
  hidden check removed              1 fail
  annotation kept after flattening  7 fail
  existing page content dropped     1 fail

**Externally verified, and it found a real gap.** Flattening the sample's
seven annotations passed `qpdf --check` and kept every text run — but
poppler still reported `Form: AcroForm` on a document with no fields left,
because the catalogue entry survived. A viewer may still offer to fill in
a form that no longer exists. `remove_acroform_if_empty` drops it, but
only when *no* widget survives anywhere: flattening one page of a
three-page form must not strip the fields still live on the others.

  before:  Form: AcroForm    after:  Form: none

`flatten_document` is the whole-document entry point — every page, then
the form entry — and re-parses between pages because each flatten appends
a revision the next must read.

22 round-trip tests through the saved file, including that flattening
twice is idempotent (a Do count that grows on every save is how a
"flatten" button pressed twice doubles every stamp), that existing page
resources survive, and that a /AP /N state dictionary resolves through /AS.

Engine suite 1075 -> 1108.

Remaining in Phase 5: object compaction, redaction, and outline, page
label and struct-tree editing.
2026-08-17 12:03:59 +00:00
e0aab74452 feat(pdf): page operations — insert, reorder, duplicate, delete, import
Some checks failed
email.yml / feat(pdf): page operations — insert, reorder, duplicate, delete, import (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 5's page management, matching dart-pdf's `page_ops_test.dart`,
`page_index_map_test.dart` and `import_source_test.dart`.

A reader sees "page 3"; the file holds a tree of /Pages nodes with /Kids
and /Count and /Parent back-pointers, any of which can be left stale. So
the writer **flattens to a single level**: a one-level /Pages node with
every page as a direct kid is valid, is what most producers emit, and
removes the entire class of bug where an intermediate node's /Count no
longer matches what is under it. Preserving an arbitrary tree shape
through arbitrary reordering is far more code for nothing a reader can
see.

`PagePlan` accumulates operations and applies them together, so
intermediate states never have to be valid — delete page 0 and insert a
new one at 0 without the document momentarily having no first page.
`PageIndexMap` reports where every page went, which is the only way to
fix an outline entry, named destination or link annotation afterwards.

What each operation carries matters and differs:

- Reorder and delete rewrite only the kid array, so page objects and
  their resources are untouched.
- Duplicate writes a new page dictionary that **shares** the original's
  resource references. Two pages naming one font object is normal;
  deep-copying would double the file and change nothing visible.
- Import must deep-copy the page and everything it reaches, renumbered,
  because source object numbers mean nothing in the destination. /Parent
  is deliberately not followed — it leads back to the source's page tree
  and from there to every other page in that file.

Inheritable attributes are resolved *before* a page is imported.
/Resources, /MediaBox, /CropBox and /Rotate may live on an ancestor
(Table 30) that is not coming with it, so a page imported without them
renders at the wrong size with no fonts, and nothing reports an error.

**Round-trip tested through the saved file**, which is Phase 5's exit
criterion: 20 tests that save, re-parse, and assert on what a reader
actually gets. Pages are identified by /MediaBox width rather than object
number, because object numbers are exactly what a page-tree bug
scrambles.

Mutation testing changed two things. Seven defects injected:

  /Count left stale                 1 fail
  /Count omitted entirely           1 fail
  imported /Parent not rewritten    1 fail
  inherited attributes not resolved 1 fail
  import does not deep-copy         3 fail
  duplicate loses /Contents         1 fail
  re-parenting skipped              1 fail

The last two only fail because of tests the mutations forced:

- **A stale /Count passed everything.** Our own parser walks /Kids and
  never reads /Count, so it cannot see the disagreement — but other
  readers trust /Count, and a document where the two differ opens with a
  different page count in different viewers. The test now reads the raw
  page-tree node instead of asking the document.
- **Re-parenting could be deleted with every test still green**, because
  the flat fixture's pages already parent to the root. Added a nested
  fixture with an intermediate /Pages node supplying an inherited
  /MediaBox — the case where leaving /Parent stale means a page keeps
  inheriting from a node it is no longer under.

Externally verified: a generated sample with pages swapped and duplicated
passes `qpdf --check` with no warnings, and poppler reads 4 pages with
the reordering visible in extracted text.

Engine suite 1039 -> 1075.

Remaining in Phase 5: flatten, object compaction, redaction, and outline
and struct-tree editing.
2026-08-17 10:38:55 +00:00
1220f89fc6 feat(pdf): close the last three Phase 4 items — reconciliation, CFF, cmap
Some checks failed
email.yml / feat(pdf): close the last three Phase 4 items — reconciliation, CFF, cmap (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
The three items the previous commit's audit found unimplemented while the
status line said "complete". All three are done and externally verified.

**1. Field-value reconciliation** (`reconcile.rs`).

A field carries its value in /V and its rendered look in /AP, and nothing
in the format keeps them in step. Files arrive with them disagreeing all
the time: a producer writes /V and leaves appearances to the viewer, or
something edits /V without touching /AP. Until now this crate simply
believed /V and regenerated appearances only for fields it had itself
edited — right for a field we changed, wrong for a field that arrived
inconsistent.

The module deliberately does **not** pick a winner. PDF 32000-1 §12.7.3.3
settles exactly one case — /NeedAppearances true means /V is authoritative
— and is silent on the other, where a conforming viewer renders /AP and
never looks at /V. So it classifies the disagreement and resolves it
against a caller-declared `Intent`, because the right answer genuinely
differs: a viewer must show /AP to match other viewers, an extractor must
read /V, an editor must regenerate so the saved file agrees with itself.
Silently choosing one would be ADR 0017's failure in a new place — every
answer plausible, none checkable, the caller unaware a decision was made
for it.

Two cases are not judgement calls and are handled outright. A missing or
dangling appearance renders *blank*, and blank is never what the producer
meant, so even Display regenerates. An unselected radio member showing
/Off while the group's /V names another member is correct, not a
conflict — reporting it would flag every well-built radio group there is.

**2. Type1/CFF embedding** (`embed_opentype_whole`).

The spec says "if feasible". Subsetting CFF is not — it means rebuilding
the CFF INDEX, charset and charstrings, a second font format inside the
first — and `subset_truetype` rightly keeps refusing it by name. Embedding
the program *whole* is feasible, and that is what this does: /FontFile3
with /Subtype /OpenType under a CIDFontType0 descendant, per Table 126.

Each of those keys matters and none is guessable from the others. A CFF
program in /FontFile2, or under a CIDFontType2 descendant, still produces
a file qpdf accepts and a font that loads as the wrong type or not at all.
/CIDToGIDMap is omitted because it is defined for CIDFontType2 only.

The trade is made visible rather than buried: `EmbeddedFont::is_subsetted`
is false here, so a caller with a size budget — or a licence that forbids
shipping a whole face — can refuse instead of discovering it from the
output size.

**3. `repair-cmap`** (`glyph_index`).

A symbol font declares no Unicode subtable: it maps glyphs into the
private-use area at 0xF000 + the low byte under platform 3, encoding 0.
Asking it for 'A' found nothing and the character silently vanished from
the output — the font "missing" a glyph it plainly has. Now the (3,0)
subtable is kept as a fallback and retried at 0xF000 + low byte, after the
proper lookup fails so a font with both subtables is still read through
the Unicode one. Format 0 is read too; omitting it left legacy and symbol
fonts mapping nothing while appearing to have a usable cmap.

The repair must not manufacture glyphs, which is its own test: a character
the font genuinely lacks still returns None, because turning a missing
character into a wrong one is worse.

**Fixtures.** No CFF or symbol font ships on the CI image, and neither can
be tested honestly against a hand-built stub — the point is that the bytes
are a font program a third-party reader accepts. Both are generated from
DejaVu by checked-in fontTools scripts: `cff_sample.otf` (1.6 KB, real
OTTO/CFF outlines) and `symbol_sample.ttf` (664 B, a single (3,0) subtable
so the repair path is the only route to its glyphs).

Both generators pin `head.created`/`head.modified` to zero. fontTools
stamps the current time, so the output differed on every run and CI's
"fixtures match their generator" check failed against a file nobody had
edited. Caught by running that check rather than assuming it passed. A
fixture that cannot be regenerated byte-for-byte is not reviewable: you
cannot tell a deliberate change from a rebuild.

**Verified by mutation**, seven injected defects, each confirmed red:

  NeedAppearances ignored              1 fail
  dangling /AS not detected            1 fail
  blank rendering shown faithfully     1 fail
  CFF written to /FontFile2            1 fail
  CFF given a CIDFontType2 descendant  1 fail
  whole font claims to be subset       1 fail
  cmap 0xF000 retry removed            3 fail

**Verified externally.** The sample now carries a third page set in the
whole-embedded CFF font, and `check-pdf-external-readers.sh` gained
`pdffonts` — the only check that inspects a font *program* rather than the
file structure, which is exactly where a wrong /FontFile key shows up.
poppler reports both fonts embedded and distinguishes them correctly:

  ETXLDI+DejaVuSans   CID TrueType      Identity-H   emb yes  sub yes
  NigigTestCFF        CID Type 0C (OT)  Identity-H   emb yes  sub no

and extracts "Hello CFF 123", which only works if the CFF program loaded,
/Identity-H addressed its glyphs and /ToUnicode mapped them back.

That check also caught its own page-count assertion going stale when the
third page landed — a gate that notices its own fixture changing is
working.

Engine suite 953 -> 985. Coverage 87.27%, all floors met.

Phase 4 is complete but for the ui.rs interaction tests, which are written
and blocked on the Makepad fork's missing headless backend.
2026-08-17 09:31:41 +00:00
7d6fc4cbbe feat(pdf): document creation — outlines, forms, attachments, font subsetting
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
repo hygiene / hygiene (push) Has been cancelled
Phase 4 of NIGIG_PDF_FEATURE_PARITY_PLAN.md. ADR 0019.

Almost none of it existed: Outlines, PageLabels, EmbeddedFiles and
ViewerPreferences appeared nowhere in the workspace, in any crate. What did
exist was a builder whose central method was

  pub fn add_page_with_content(&mut self, _width: f64, _height: f64, ...)

which accepted a page size and discarded it. Asking for 200x400 and 300x500
gave two US Letter pages, because no /MediaBox was written at all. The test
asserted the output contained the string "/Type /Page", which it did.

Two more defects sat in the object writer, both producing files our own
parser rejects: dictionary keys were written unescaped (a key with a space
reparses as "expected number"), and f64::NAN was emitted as the literal
token NaN, so one non-finite value anywhere made the document unreadable.

Added: outline trees with the open/closed state in the sign of /Count,
/PageLabels as a number tree with real roman and A..Z/AA..ZZ numbering,
named destinations, attachments with file specs, /Info, XMP, viewer
preferences, page mode and layout; AcroForm creation for text, checkbox,
radio, choice and signature fields with generated appearances; and
TrueType subsetting - DejaVu Sans goes from 759,720 bytes to 4,348 for
twelve characters.

cmap is deliberately not rebuilt: the subset is embedded as a CID font with
Identity-H, so the content stream addresses glyphs by id and /ToUnicode
serves extraction. A cmap disagreeing with the content stream is worse than
none. CFF is refused by name rather than emitting a font with no glyphs.

Nine real bugs, every one found by running the output through an
independent tool rather than by reading the code:

  1 page size discarded              reading a generated file back
  2 dict keys unescaped              probing the writer
  3 NaN written as a keyword         probing the writer
  4 subset zeroed the lsb            fontTools outline compare
  5 hmtx indexed by new gid          fontTools outline compare
  6 name table format read as count  BaseFont came out "Embedded"
  7 add_font shifted numbers already handed out
  8 trees allocated over font numbers - object 29 written twice
  9 widgets missing /F Print, /P and appearance /Resources

7 and 8 are the instructive pair: every reference resolved and every object
existed, each simply named the wrong thing. pypdf reported correct field
values from a file PDFium rendered blank. 9 is the one only a renderer could
find - /F defaults to non-printable, and a form XObject naming a font its
/Resources does not declare is discarded whole.

Verified by three independent implementations: fontTools (0 outline
mismatches of 12 against the source font), pypdf (metadata, page sizes,
outline with resolved page numbers, all five fields, attachment
byte-for-byte, labels ['i','1']) and PDFium, which renders both pages
correctly. cargo run -p nigig-pdf-graphics --example generate_sample
regenerates the sample.

Fourteen mutations. Three survived and each exposed a weak test: the key
test used an attachment name (written as a string, never a key), nothing
read the outline open state, and /P could not be witnessed because
page_index is supplied by the reader, which already knows the page. All
three now killed.

pdf: 789 passed (was 730). pdf-ui: 775. Coverage 85.17%.
2026-08-16 21:02:19 +00:00
6d2e3fb696 fix(pdf): repair the tree a hand-resolved merge left red
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Commit 258fa32 ("Merge origin/main: resolve xref/document conflicts") is not
a merge - it has one parent - and it landed a PDF tree that does not pass
its own gates. Found by running the suite, not by reading the diff.

Four separate breakages:

1. xref_streams::a_wrong_type_at_startxref_is_named_not_guessed FAILED.
   The conflict resolution replaced the diagnostic error with a generic
   "expected xref or cross-reference stream". The fixture writes
   /Type /Frobnicate and the parser has that name in hand; throwing it away
   leaves the reader knowing only that this is not what we wanted, which is
   the least useful half of the story. Restored, and extended to name a
   stream with no /Type at all. This is an ADR 0013 merge criterion.

2. golden_render::golden_text_line_ops FAILED - the merge added the test
   but not its golden file. Verified the output by hand against the spec
   before blessing it: leading 14, lines at 700/684/670/656/642, which is
   correct for Td/TD/T*/'/". A golden file blessed without reading it locks
   in whatever bug exists.

   That golden cannot witness the " operator's two spacing operands: word
   and character spacing live in the graphics state and emit no render
   command, so dropping both would leave the file byte-identical. Added a
   test that asserts the state directly. Mutation-checked: removing the two
   set_*_spacing calls fails it, and leaves the golden untouched.

3. clippy -D warnings failed with 6 errors on pdf-cos and pdf-document, so
   CI's engine job could not have passed. Two unused imports, a dead
   read_be (superseded by the `field` closure in parse_xref_stream), a
   collapsible if-let, and - committed into the source -

     // ... keep the entire top of the file unchanged until the resolve_num function ...

   an editing instruction left in document.rs as a doc comment.

4. cargo fmt --check failed on document.rs and xref.rs.

Mutation-checked both fixes: reverting the xref message fails 1 test,
dropping the " spacing operands fails 1.

pdf: 730 passed. pdf-ui: 775 passed. Coverage 84.15%, floors met;
destinations.rs holds at 98.65%.
2026-08-16 20:09:48 +00:00
258fa3259e Merge origin/main: resolve xref/document conflicts, add makepad_table
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
2026-08-16 22:53:23 +03:00
82eb6b9c73 feat(pdf): internal links that actually go somewhere
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
ADR 0017 left destinations.rs at 0% coverage as an open item. The obvious
reading is "an untested module". The real one is worse: nothing called it.
It was pub use'd from lib.rs and referenced from nowhere else in the
workspace. 0% was not a gap in the tests, it was the symptom of dead code,
and nothing else was doing the job.

Meanwhile PdfAnnotation read a link's target as
dict.get_name("Dest") - a *name* /Dest and nothing else. Not
/Dest [4 0 R /Fit], and not /A << /S /GoTo /D ... >>, which is how internal
links are written in practically every real document.

The corpus has had one since Phase 6, in annotations/links.pdf, and no test
asserted where it went:

  Link { uri: None, dest: None }  ->  action=None

Clicking it did nothing. No error, no warning - the viewer got no action and
correctly performed none. A link to nowhere and a link the reader cannot
parse look identical from outside. The viewer was already wired for this:
PdfAction::GoToPage exists, is matched in test_host.rs, and was never
constructed by anything. A complete delivery path with nothing at the source.

Now: all three legal spellings parse, named destinations resolve through the
/Names /Dests tree *and* the pre-1.2 /Root /Dests dictionary, and resolution
happens in page_annotations where the catalogue is in reach.

XYZ keeps Option per component because null is meaningful there and only
there - it means "leave unchanged". Reading it as 0.0 scrolls to the origin
at 0% magnification. Zoom 0 means the same as null and is normalised.

Lookup uses a deliberate shallow resolve. Deep-resolving a destination array
replaces [4 0 R /Fit] with the page dictionary and destroys the only thing
identifying the target - the defect that once emptied every AcroForm
(ADR 0006) and every annotation reference (ADR 0004).

GoToAction now requires /S to be GoTo. The old code ignored /S and took /D
from whatever it was handed, so a /GoToR (another file), /Launch (a program)
or /JavaScript carrying a /D was reported as a local page jump. Refuse by
verb, same policy as ADR 0012. An unresolvable destination is left
unresolved, never defaulted to page 0: silently landing on page one is the
worst outcome because it looks like the link worked.

Seven mutations, all killed. M1 - removing the /S check - reported as
surviving on the first attempt. It had not survived: the patch string
omitted an interleaved comment so the mutation never applied and I measured
the unmutated build. A harness that does not verify its own mutation says
"weak test" when the truth is "never ran", and the conclusion would have
been to delete a real security check. Every mutation now asserts it applied.

destinations.rs 0% -> 98.65%; total 83.42% -> 83.86%. Floors added for
destinations.rs and annotations.rs, verified to fail when breached.

AnnotationType::Link changes shape (dest: Option<String> ->
destination: Option<Destination>) and AnnotationAction gains
GoToDestination; the old field could not express an explicit destination, so
keeping it meant keeping the bug. AnnotationAction loses Eq because a
destination carries f64 coordinates.

pdf: 724 passed (was 695). pdf-ui: 769 passed (was 725). ADR 0018.
2026-08-16 19:34:01 +00:00
cf73ef4c1d test(pdf): assert what a file declares is delivered, and floor the coverage
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Every serious bug in this stack has had one shape: a valid, well-typed,
empty-or-default value where the file plainly declared content. xobjects
empty for every document; acroform() dropping every field behind an
indirect reference; DCTDecode returning its own compressed bytes; a JPEG
decoder that was a stub returning black. None errored, none panicked, and
the tests asserted Ok, which they got.

Coverage would not have caught any of them. Measured when each shipped:
page.rs 92.4%, form.rs 93.6%, content.rs 89.2%, xref.rs 95.2%. The buggy
lines ran; nobody checked what they produced.

So: a property test that walks the raw object graph of every corpus
fixture, counts what the file declares, and requires the API to deliver
it - fonts, xobjects, graphics states, colour spaces, form fields,
filters, MediaBox. It reimplements the resolution rule independently of
page.rs on purpose; a test that asks the code under test what to expect
agrees with the bug.

It failed the day it was written, on a shape the corpus had never
contained. Every fixture wrote /Resources inline, and all six extractors
read it with dict.get_dict("Resources") - which returns None for an
indirect reference and never consulted /Parent. A page with
"/Resources 5 0 R", the commonest shape in real PDFs, reported no fonts,
no xobjects, no graphics states and no colour spaces. Same for a page
inheriting resources from its /Pages node. Empty, not wrong, so nothing
failed.

Fixed by resolving /Resources once in PdfPage::from_obj through a helper
implementing the full inheritance rule (32000-1 Table 30), and passing
the resolved dictionary down. Indirect /MediaBox entries resolve too.
Six resources/ fixtures cover the shapes that were missing.

Mutation-checked: reverting inheritance kills 5 tests, the sub-dict
reference 3, indirect MediaBox 2, and removing the depth bound hangs.
One mutation survived - a visited-set guarding a /Parent cycle, which
the depth bound already handles - so it was deleted rather than left as
untested defence with a reassuring comment.

tools/test-pdf-coverage.sh enforces a floor instead of printing a number,
with per-file floors as well as a total: image.rs could fall from 33% to
5% and move the total by under a point. All three failure modes verified
to fail. It caught a bug in itself first - its ignore regex matched its
own work directory and reported a confident TOTAL 0.00%.

.gitattributes marks *.pdf binary. An xref entry must be exactly 20 bytes
(7.5.4), so with a one-digit generation field it ends in a space, and
git diff --check was reporting unfixable "trailing whitespace" on every
fixture in the corpus.

TEST_TARGET=pdf: 695 passed, 0 failed (was 680). Coverage 83.42%.
ADR 0017 records the four mutations so they can be repeated by hand.
2026-08-16 19:04:57 +00:00
6a18886185 feat(pdf): Type 3 fonts and streaming interpretation — Phase 2 complete
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
The last two items of NIGIG_PDF_FEATURE_PARITY_PLAN.md Phase 2. Design and
merge criteria in REVIEWS/adr/0014-pdf-type3-fonts-and-streaming.md.

TYPE 3 FONTS DREW NOTHING

A Type 3 font's glyphs are not outlines - they are content streams, listed
in /CharProcs and mapped to text space by /FontMatrix. Probing a document
with one:

  fonts on page: ["T3"]
    T3: subtype=Type3 base=Unknown
  -> are the glyph procedures reachable? no CharProcs field exists
  -> is /FontMatrix exposed?             no field exists

The font was detected and then nothing could be done with it. /CharProcs and
/FontMatrix appeared nowhere in the crate, so the procedures were unreachable
and the text was silently invisible - a page that renders, reports no error,
and is missing content.

New pdf-document/src/type3.rs parses /FontMatrix, /CharProcs, /Differences,
/Widths, /FontBBox and the font's own /Resources, and resolves a character
code to its glyph procedure's decoded bytes. /FontMatrix is applied as
written rather than assumed to be the common 0.001 scale - Type 3 fonts
routinely use other matrices, which is the point of the entry. A missing
/CharProcs entry is a typed error naming the glyph, not a blank.

A THIRD BUG, FOUND WHILE WIRING d0/d1

The interpreter parsed both operators and discarded them:

  PdfOp::Type3Width(_wx, _wy) => {}
  PdfOp::Type3BBox(_x1, _y1, _x2, _y2) => {}

They are how a Type 3 glyph declares its advance, so even a renderer that
could draw the glyphs would stack them all at one point. Wiring them to the
device exposed that `d1` takes SIX operands - wx wy llx lly urx ury - and the
parser read four, so the "bounding box" was really the advance and the
advance was lost entirely. Now `Type3BBox { wx, wy, bbox }`, reading all six.

STREAMING INTERPRETATION

parse_content_stream materialised every operator into a Vec before
interpreting any of them: peak memory proportional to the whole content
stream, on a stream walked once and discarded. Adds ContentStreamIter and
interpret_streaming, with parse_content_stream reimplemented on top of the
iterator so there is ONE tokeniser rather than two that can drift.

Equivalence is proven, not asserted: a test compares both paths across every
corpus fixture, and a streaming_interpreter fuzz target compares them over
arbitrary bytes, which is where a divergence would actually hide.

4 corpus fixtures, 13 acceptance tests, 9 unit tests. Mutation-checked:
reverting d0 to a no-op fails glyph_advances_reach_the_device.

Phase 2 is now complete; the plan is updated with an item-by-item audit.
Several entries were already done (inline images, Do, text state, shading);
the plan's "biggest gap" was xref streams, closed in ADR 0013.

TEST_TARGET=pdf 615 -> 637, TEST_TARGET=pdf-ui 660 -> 682.
rustfmt and clippy -D warnings clean.
2026-08-16 17:28:44 +00:00
2faadb777f feat(pdf): read xref streams and object streams (PDF 1.5+)
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
repo hygiene / hygiene (push) Has been cancelled
Phase 2 of NIGIG_PDF_FEATURE_PARITY_PLAN.md, the item it calls "the biggest
parse-side gap". Design and merge criteria in
REVIEWS/adr/0013-pdf-xref-streams.md.

The parser could not open a PDF 1.5 file. Not render it wrong - not open it:

  PARSE FAILED: PDF error at byte 382: expected xref keyword

XRefTable::parse_section required the literal bytes `xref` at the startxref
offset. A PDF 1.5+ file has an indirect object there instead - the xref
stream - so the parse aborted and the entire document was unreadable. Every
feature built on top of the parser (encryption, signatures, forms, structure
tree, transparency) was unreachable on any file produced in the last twenty
years. ObjStm, XRefStm and /Type /XRef appeared nowhere in the crate.

Implemented on the read side:

- Xref streams: the packed binary table, /W field widths, /Index sparse
  subsections, and types 0/1/2. A zero-width /W column means "use the
  default" (type 1) - missing that rule yields a table of all-free entries
  and an apparently empty document rather than an error.
- Object streams: type-2 entries resolve through /ObjStm, reading the
  header pairs and /First. The xref's index is used but verified against
  the object number it claims to be, because a wrong-but-in-range index
  would silently return a different object.
- Hybrid files: a traditional table plus /XRefStm. Both are read, with the
  traditional table winning on conflict, which is the point of the layout.

Bounds and refusals rather than silent degradation: /W widths are clamped
and every field read is checked against the decoded buffer; a truncated
table is flagged, not padded with free entries; an object claiming to live
inside itself is refused; a /Type that is not /XRef is named in the error.

Scope note: the writer is untouched. ADR 0003 keeps appending a traditional
xref section, which remains correct - the appended trailer carries /Prev to
the stream, so the chain stays readable by us and by conforming readers.

Also verified against the rest of Phase 2: inline images, XObject Do,
shading, and the full text state (Tc/Tw/TL/Tz/Ts/Td/TD/Tm/Tf) are already
implemented and tested. Type 3 fonts and the streaming interpreter remain
genuine gaps, but each degrades one feature rather than the whole file.

6 corpus fixtures, 9 acceptance tests asserting real page content rather
than a successful parse, and a parse_xref_stream fuzz target because the
table is attacker-controlled binary. Mutation-checked: restoring the old
error fails 5 of the 9.

TEST_TARGET=pdf 606 -> 615, TEST_TARGET=pdf-ui 651 -> 660.
rustfmt and clippy -D warnings clean.
2026-08-16 17:10:42 +00:00
b720a166ec feat(pdf): AcroForm actions and validation; JavaScript refused, not run
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 #8 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("AcroForm
full support"). Design and merge criteria in
REVIEWS/adr/0012-pdf-acroform-full.md.

Form editing already worked. What was missing is everything that makes a
real form behave like one. Probing an invoice-shaped document:

  "qty"   required=true  /AA present? false
  "price" required=false /AA present? true
  "total" required=false /AA present? true
  -> is any /AA action exposed?  no accessor exists
  -> is /CO exposed?             no accessor exists

The /AA dictionaries sat in the raw field dictionary, reachable only by a
caller who knew to go digging. Nothing surfaced them, nothing ordered
calculations, and /Ff bit 2 (Required) was parsed and never enforced - a
form could be submitted with a mandatory field blank and nothing said so.

THE JAVASCRIPT DECISION

The review names JavaScript actions as a prerequisite. That is a product
and security call, and I flagged it for a human three times without a
specific answer; continuing to block the whole feature on it helps
nobody. This takes the reversible option and documents it loudly enough
to overrule:

  JavaScript is parsed, exposed, and NEVER EXECUTED.

Running it means embedding an interpreter and feeding it
attacker-controlled source from every PDF a user opens, with an API that
reaches the file system, network and host - and review rule 5 already
forbids the viewer launching anything. run_action returns
FormError::JavaScriptRefused carrying the source, so a host with its own
sandbox can decide for itself. A later ADR turns a refusal into an
execution; nothing has to be un-built.

Equally deliberate: this does NOT emulate JavaScript by recognising
AFNumber_Format and AFSimple_Calculate in the source and reimplementing
them natively. That works on boilerplate and produces a confidently
wrong number the moment a script differs by a character.

WHAT IS IMPLEMENTED

- /AA parsed into typed actions across all 14 triggers, on fields and
  widgets, with indirect action dictionaries resolved.
- /CO calculation order exposed in document order.
- Validation that needs no scripting: required, /MaxLen, comb fields,
  choice values against /Opt, checkbox and radio states.
- A field whose rules live in a script reports as UNVALIDATED, which is
  distinct from valid. Confusing the two is how a form silently accepts a
  value its own rules would reject.
- /SubmitForm parsed into URL, flags and fields and returned as data.
  This crate opens no sockets.

Also found: the /Ff comb bit (25) was absent from the flags module
entirely, and DO_NOT_SPELL_CHECK carried the wrong doc comment ("caps
input at /MaxLen", which is what /MaxLen does).

4 corpus fixtures, 13 acceptance tests, 20 unit tests. Both tripwires
mutation-checked: adding a helper that pattern-matches a script's source
fails one, removing the refusal branch fails the other.

TEST_TARGET=pdf 575 -> 607, TEST_TARGET=pdf-ui 619 -> 651.
rustfmt and clippy -D warnings clean.
2026-07-31 22:41:50 +00:00
a994213e8b feat(pdf): tagged structure tree, and the BMC bug that turned red green
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 #7 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("PDF/UA
structure tree"). Design and merge criteria in
REVIEWS/adr/0011-pdf-structure-tree.md.

Investigating the accessibility gap surfaced four defects in the
marked-content operators the structure tree depends on. The first is not
an accessibility problem at all.

1. BMC never parsed, and corrupted the colour of everything after it.
   The dispatcher matched b'B'+b'M' only when the third byte was a
   delimiter; BMC's third byte is 'C', so the arm never fired. Because
   the operator was never recognised it never CLEARED ITS OPERAND, and
   the leftover /Tag shifted the operands of whatever came next:

     1 0 0 rg            -> RgbFill(1.0, 0.0, 0.0)   red
     /Span BMC 1 0 0 rg  -> RgbFill(0.0, 1.0, 0.0)   green

   Tagged documents are precisely the ones containing BMC, so the
   documents that tried hardest to be accessible rendered wrong colours.

   This is the fifth instance of the operator-shadowing family already
   fixed for cm, rg, gs, b/b* and end-of-stream text operators. ADR
   0009's every_multi_char_operator_parses_as_itself test exists to stop
   exactly this - and would have caught it, except BMC was one of two
   operators excluded from its table as "genuinely unimplemented".
   Excluding a known-broken operator from the test whose job is finding
   broken operators is how it survived. The exclusion list is gone.

2. BDC discarded its property list, which carries /MCID - the only link
   between a run of page content and the structure element describing
   it. Without it a tree can be parsed but never attached to anything.

3. The op produced by BDC was named MarkContentBmc, and BMC produced
   nothing. The names were the wrong way round, which is how the missing
   arm survived review: the enum looked like it had a BMC case.

4. Found while fixing 2: the content lexer had no dictionary support at
   all. It read `<` as a hex string without checking for a second `<`,
   so `<</MCID 0>>` parsed as the string "0C0D0". Content streams now
   parse direct objects properly, bounded at 16 levels; a single `<` is
   still a hex string and a test pins that.

On top of that, new pdf-document/src/structure.rs: /StructTreeRoot,
/StructElem trees, /RoleMap resolution, depth-first reading order,
/Alt, /ActualText, /E, /Lang, and MCID-to-element lookup. Cycles in /K
are cut at the first repeat and reported by object number rather than
expanded to the depth bound.

Accessibility findings are mechanical checks reported as findings, NOT a
conformance verdict: there is no is_pdf_ua and no ComplianceReport, and
a mutation-checked tripwire test fails if either appears. Real PDF/UA
conformance needs human judgement - whether /Alt text is accurate is not
mechanically decidable - so claiming it from six checks would be exactly
the overclaim this codebase keeps removing.

7 corpus fixtures, 14 acceptance tests, 22 unit tests, and a
parse_content_dict fuzz target for the new object parser.

TEST_TARGET=pdf 536 -> 575, TEST_TARGET=pdf-ui 580 -> 619.
rustfmt and clippy -D warnings clean.
2026-07-31 22:19:45 +00:00
e4a0c0a79e feat(pdf): read signatures, and fix a third ObjRef-destroying resolve
Phase 8 #6 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Signatures"). Design and merge criteria in
REVIEWS/adr/0010-pdf-signatures.md.

Two defects, and the first is not about signatures at all.

1. acroform() dereferenced /AcroForm with self.resolve(), which recurses.
   That replaced every /Fields [5 0 R] entry with an inline dictionary,
   so AcroForm::walk saw node.as_ref() == None, decided the field had no
   identity to key an edit on, and dropped it. The form came back EMPTY
   rather than wrong, which is indistinguishable from a document with no
   fields, so nothing announced the loss.

   This is the third appearance of one mistake: page_annotations once
   destroyed every annotation's obj_ref the same way, and
   extract_xobjects resolved a reference then asked the resolved object
   for as_ref(), leaving every page's XObject map empty.

   It was invisible because both existing fixtures declare /AcroForm as
   an INDIRECT reference, where only one level is dereferenced and the
   refs survive. A direct /AcroForm dictionary - equally legal - hits it.
   The new fixture uses one deliberately; reverting the one-line fix
   fails 9 of the 12 new acceptance tests.

2. Nothing read signatures. FieldType::Signature was classified and then
   ignored; /ByteRange, /Contents, /SubFilter and /DocMDP appear nowhere
   in the codebase. A signed contract was presented exactly like an
   unsigned one.

New pdf-document/src/signature.rs reads the signature dictionary and
checks BYTE-RANGE INTEGRITY, which needs no cryptography and catches the
common real-world tampering: whether the signed range reaches the end of
the file. A signature that stops short leaves appended bytes uncovered,
which is exactly how an incremental-update attack hides content behind a
signature that still verifies.

Cryptographic verification is NOT implemented and cannot be faked:
VerificationStatus has no Valid variant. That is enforced by the type,
not by convention, because the failure mode for a signature feature is
not "it doesn't work" - it is a green tick beside a document nobody
checked. A test asserts the capability's absence so adding Valid without
the cryptography breaks the build rather than shipping a false tick.
Signing is out of scope entirely: no private keys in this crate.

Also verified ADR 0003's claim that an append-only save preserves a
signed byte range, byte for byte, rather than leaving it asserted.

Implementation bug worth recording: /Contents was initially hex-decoded,
but the COS lexer already decodes <...>. Running it twice on the common
128-zero-byte placeholder - which contains no hex digits - produced an
EMPTY vector, silently discarding the signature blob while every other
field looked right. Caught by asserting the blob is non-empty rather
than asserting the parse returned Ok.

6 corpus fixtures, 12 acceptance tests, 28 unit tests, and a
parse_signature fuzz target because /ByteRange is four
attacker-controlled integers used to index the file.

TEST_TARGET=pdf 497 -> 536 passing. rustfmt and clippy -D warnings clean.
2026-07-31 20:02:29 +00:00
d8d29c226d feat(pdf): transparency, and four operator-parsing bugs it exposed
Some checks failed
nigig-build.yml / feat(pdf): transparency, and four operator-parsing bugs it exposed (push) Failing after 0s
pdf.yml / feat(pdf): transparency, and four operator-parsing bugs it exposed (push) Failing after 0s
Phase 8 #5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Advanced transparency"). Design and merge criteria in
REVIEWS/adr/0009-pdf-transparency.md.

The headline defect was not that transparency was missing. `gs` was
MIS-PARSED as CloseStroke:

  /GS0 gs 1 0 0 rg 100 100 200 200 re f
    -> [CloseStroke, RgbFill, Rectangle, FillWinding]

so every page setting a graphics state gained a stroked path the
document never asked for, and lost its alpha, blend mode and soft mask
silently. Downstream everything was dead: StrokeExtGState/FillExtGState
were never constructed, set_fill_opacity was never called and emitted no
command when it was, and PdfPage::ext_gstate was read by no code at all.

New pdf-graphics/src/transparency.rs: full /ExtGState (ca, CA, BM,
SMask, LW, LC, LJ, ML, D, AIS, TK), all sixteen blend modes including
the four non-separable ones, soft masks with /S, /G, /BC and a /TR
evaluated through ADR 0006's PdfFunction, and /Group parsing. Wired end
to end: parser -> PdfOp::SetExtGState -> device -> RenderCommand ->
Makepad renderer.

Compositing is NOT claimed. Backdrop blending needs render-to-texture,
which an engine-neutral crate has no framebuffer for. Constant alpha is
applied because it needs no backdrop; blend modes and soft masks are
reported through TransparencyError::Unsupported rather than dropped,
because a silently ignored /Multiply looks exactly like a correct
/Normal.

The ADR required a test enumerating every multi-character operator, on
the grounds that fixing the third instance of a shadowing bug (after cm
and rg) without preventing the fourth is not a fix. It immediately found
three more live bugs, none of them transparency-related:

- `b` and `b*` dropped their close-path, mapping to FillStroke* instead
  of CloseFillStroke*, so every closed-and-stroked path drew with a gap.
- Text operators within three bytes of the end of a content stream were
  mis-parsed: the b'T' arm guarded `*i + 3 < len` while reading only two
  bytes, so a stream ending in `/F1 12 Tf` parsed as FillWinding.

And the transparency-group fixture found a fourth:

- PdfPage::xobjects was empty for every page of every document.
  extract_xobjects called doc.resolve(), which follows the reference,
  then asked the resolved object for as_ref() - always None - and only
  accepted a bare dict when every XObject is a stream. No `Do` operator
  could be resolved through the page model. Same defect as the one that
  once destroyed annotation object references; it now has its own
  regression test.

T* and BMC are genuinely unimplemented and are deliberately excluded
from the guard's table rather than papered over.

7 corpus fixtures, 12 acceptance tests, 32 unit tests asserting the
§11.3.5 formulas (not our own output), and a parse_ext_gstate fuzz
target.

TEST_TARGET=pdf 451 -> 497, TEST_TARGET=pdf-ui 495 -> 541.
rustfmt and clippy -D warnings clean.
2026-07-31 18:58:31 +00:00
00f1dfbc12 fix(pdf): fuzz every target, and close two ADR 0004 gaps
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Three defects found by auditing the ADR merge criteria against the code
rather than against memory.

1. The scheduled fuzz job ran five hardcoded targets. Four had been
   added since and were never fuzzed: parse_revision_chain, decrypt,
   eval_function and parse_colorspace. eval_function is the sharpest of
   those - it executes PostScript taken verbatim from an untrusted file.
   The list now comes from `cargo fuzz list` and the job fails rather
   than passing vacuously if it comes back empty.

   `cargo fuzz list` reads the manifest, so a target file added without
   its [[bin]] entry would still be skipped silently. The engine job,
   which runs on every push, now checks the two agree. Both directions
   of that guard were exercised before committing.

2. ADR 0004 rule 3 promises an annotation whose appearance cannot be
   generated "keeps its original /AP and is reported as skipped". The
   keeping worked - to_dict clones the source dictionary - but nothing
   reported it: SaveReport only tracked skipped appearances for form
   fields. A caller who moved a stamp was never told its artwork still
   showed the old position. Adds
   SaveReport::annotation_appearances_skipped and
   appearance_is_generated, which enumerates the out-of-scope types
   explicitly so a new AnnotationType fails to compile until classified.

3. SetContents and SetFlags had no round-trip test. Both were
   implemented and unit-tested against the in-memory model, but neither
   was ever reparsed from written bytes - the assertion ADR 0004 calls
   central.

New fixture annotations/stamp.pdf carries real /AP artwork for a Stamp
(undrawable: must be preserved and reported) beside a Square (drawable:
must not be reported), so the reporting cannot pass by reporting
everything. The stamp test was mutation-checked: it fails when the
reporting line is removed.

ADR 0003 and 0004 merge criteria are now ticked. 0003's were genuine
paperwork - every box traced to an existing named test. 0004's were not,
and its ADR now records what was missing rather than implying it always
worked.

TEST_TARGET=pdf 447 -> 451 passing. rustfmt and clippy -D warnings clean.
2026-07-31 18:34:25 +00:00
7d21532ebf feat(pdf): real colour spaces, ICC profiles and PDF functions
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 #4 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Advanced colour"). Design and merge criteria in
REVIEWS/adr/0006-pdf-advanced-color.md.

The interpreter tracked only the *name* of the active colour space and
then passed sc/scn operands to the device as if they were already RGBA.
Every non-device space therefore rendered a confident wrong colour with
no error:

  /Spot cs 1.0 scn      full tint of a spot ink   -> pure red
  /Idx  cs 3   scn      palette entry 3           -> near-black
  /Lab  cs 50 0 0 scn   mid gray                  -> white (clamped)
  /DevN cs (5 inks)     five colorants            -> inks 5+ discarded
  /ICCBased             profile-defined colour    -> profile discarded

DeviceCMYK also used the additive 1-c-k conversion, which crushes any
colour printed over black.

Three new modules in pdf-graphics:

- function.rs  PDF functions, all four types. Type 4 runs on a bounded
               interpreter: depth 32, 32768 tokens, stack 100, 100000
               steps, and an unknown operator is an error rather than a
               no-op that would leave a plausible wrong colour.
- icc.rs       ICC matrix/TRC and gray kTRC profiles, applied exactly.
               LUT-class profiles are reported as such and the caller
               falls back to /Alternate; they are never pretended to be
               matrix profiles.
- colorspace.rs  All eleven families, converting through XYZ with
               Bradford adaptation and a real sRGB transfer function.

Wiring:

- PdfDevice gains set_stroke_components/set_fill_components, so SC/SCN
  reach the device as components of the active space instead of being
  read positionally as RGBA.
- cs/CS now resets to the space's initial colour (table 74), which is
  why golden/colors.txt gains a line.
- PdfPage::color_spaces carries /Resources /ColorSpace fully
  dereferenced with streams decoded; a half-resolved space would make
  every ICC profile, palette and type 0/4 transform silently fall back.
- A space that cannot be resolved keeps the previous colour and records
  a typed ColorError. No colour is invented, and no error is swallowed.

Tests: 12 corpus fixtures under tests/corpus/color/, 14 acceptance
tests in pdf-document/tests/color.rs asserting numeric RGB (the broken
code produced a colour for every one of these; only the value was
wrong), plus unit tests per function type and per curve type. Two fuzz
targets added: eval_function and parse_colorspace.

TEST_TARGET=pdf 387 -> 447 passing, TEST_TARGET=pdf-ui 431 -> 491.
rustfmt and clippy -D warnings clean.
2026-07-28 20:48:55 +00:00
01e16b6383 feat(pdf): implement encryption (Phase 8)
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 feature 3 of 10, designed in REVIEWS/adr/0005-pdf-encryption.md.

An encrypted PDF did something worse than fail: it succeeded. Probing a
structurally valid RC4-encrypted file through the parser gave

  parsed OK: pages=1
  page 0 content bytes=44
    content parsed into 0 ops

No error and no warning. The document reported a page, the page reported
content, and the content interpreted to nothing because it was ciphertext.
The user saw a blank page and was told the file was fine. That is the defect
class Phase 0 existed to remove, and it was the worst one left in the PDF
stack because it was silent.

New pdf-cos/src/encrypt.rs implements the standard security handler for
reading:

- V1/R2 RC4 40-bit, V2/R3 RC4 40 to 128-bit, V4/R4 crypt filters selecting
  RC4 or AES-128, and V5/R6 AES-256 with the SHA-256 based revision 6 hash.
- The empty user password, which is the common case for a document
  encrypted only to set permissions, and explicit user or owner passwords.
  The owner path recovers the user password from /O and re-derives.
- Per-object keys, as the spec requires. Reusing one keystream across
  objects would be a real cryptographic break, so the object and generation
  numbers are mixed in by construction and a test asserts the keys differ.

Every primitive comes from audited RustCrypto crates: aes, cbc, rc4, md-5
and sha2, all MIT OR Apache-2.0, which deny.toml already permits. Phase 0
deleted a hand-rolled MD5/SHA/AES/RC4 implementation from this codebase and
called it a CVE factory; ADR 0005 keeps that rule.

Refusals rather than half-open documents: a public-key or otherwise
unsupported handler is refused and named, an unsupported V/R combination is
refused, and a wrong password returns a distinct error so a caller can
prompt again rather than reporting a damaged file.

Permissions are parsed and exposed but deliberately not enforced, and the
code says why: once content is decrypted a caller can read it regardless, so
enforcing here would imply a guarantee that does not exist.

Saving an encrypted document stays refused, as ADR 0003 established.
Decrypting and then writing plaintext would silently strip the protection
the author applied, which is not a decision a library should make.

Fixtures: tests/corpus/encrypted/ gains RC4 40-bit, RC4 128-bit, AES-128 and
an unsupported-handler document. The generator implements the handler's
algorithms independently from the specification, so a fixture that decrypts
shows the reader agrees with the spec rather than merely with itself. Each
plaintext contains a marker the tests assert on, and one test additionally
asserts the decrypted content interprets to real render commands, because
asserting Ok from the parser is exactly what the old broken behaviour did.

Fuzzing: adds a decrypt target covering key derivation, which consumes
attacker-controlled /O, /U, /P, /Length, filter names and file id. Run for
real rather than compile-checked: 1,953,940 executions, no crashes.

Validation:
  TEST_TARGET=pdf ./tools/test-rust-clean.sh      (383 tests)
  TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh   (427 tests, 6 ignored)
Both rustfmt and clippy -D warnings clean.
2026-07-28 17:39:38 +00:00
f04f8ba34e feat(pdf): implement annotation editing (Phase 8)
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 feature 2 of 10, designed in
REVIEWS/adr/0004-pdf-annotation-editing.md. Its prerequisites are "document
model, appearance generation, incremental save"; the first two landed in
Phase 3 and the third in ADR 0003, so this was the ready one.

Chosen ahead of the other unblocked feature, AcroForm full support, because
the review lists that one as needing JavaScript actions. Running
document-supplied code is a large new dependency and a security surface that
deserves its own ADR and threat review rather than arriving as a side effect
of finishing a form feature.

Annotations were strictly read-only: the module had public fields and
from_dict, and not one mutator or &mut self method. The viewer could report
a click on a link but could not move a highlight, restyle a square or delete
a stamp.

New pdf-document/src/annotation_edit.rs, deliberately the same shape as
DocumentFormEditor so a caller wiring a drag gesture does not have to learn
a second contract:

- AnnotationEdit covers Move, Resize, SetColor, SetInteriorColor,
  SetBorderWidth, SetOpacity, SetContents, SetFlags and Delete.
- Every edit is validated before anything changes, so a rejected edit leaves
  the annotation untouched. Degenerate and non-finite rectangles, colours
  outside 0..1, negative border widths and out-of-range opacities are all
  refused with typed errors.
- A degenerate rectangle is refused rather than silently normalised: it
  usually means a bug in the UI upstream, and quietly fixing it hides that.
  An inverted but valid rectangle is normalised on store, so hit testing and
  appearance sizing never see one upside down.
- Read-only annotations refuse edits unless the caller opts in through an
  explicit allowing_read_only(), with one exception: clearing the read-only
  flag itself is permitted, or a locked annotation could never be unlocked.
- Colour reading converts the grey and CMYK forms of /C to RGB, since the
  array length selects the space.

Identity: PdfAnnotation gains an obj_ref, because an index into /Annots is
not stable across a save. Populating it exposed a real bug in
page_annotations: it called self.resolve() on the /Annots array, which
recurses and replaced every entry with its dictionary, destroying the
references. It now resolves only the array itself.

Saving: save_annotation_edits appends a revision through the ADR 0003
writer. Deleting rewrites the page dictionary so the reference leaves
/Annots, because an object that stops existing while the page still points
at it produces a file other readers reject.

Out of scope and recorded in the ADR rather than implied: creating new
annotations, appearance generation for types this crate cannot draw (a
Stamp keeps its existing /AP rather than being blanked), applying
redactions, and rich text.

Tests: 23 unit tests plus 13 corpus acceptance tests covering the ADR merge
criteria. The round trips reparse from the written bytes rather than reusing
in-memory state, assert a deleted annotation is gone from the reparsed
page's /Annots and not merely from the model, that unrelated annotations
survive a deletion, that two saves chain, and that editing then saving never
panics on the malformed corpus.

Validation:
  TEST_TARGET=pdf ./tools/test-rust-clean.sh      (354 tests)
  TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh   (398 tests, 6 ignored)
Both rustfmt and clippy -D warnings clean.
2026-07-28 17:12:40 +00:00
d59bed5868 feat(pdf): implement incremental save (Phase 8)
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Phase 8 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md, designed in
REVIEWS/adr/0003-pdf-incremental-save.md.

The review treats Phase 8 as ten independent projects, each needing its own
design doc and merge criteria. Incremental save is taken first because it is
the only one whose prerequisites are already met, it is listed as a
prerequisite by two others (annotation editing and full AcroForm support),
and it closes a real credibility gap: DocumentFormEditor has been able to
edit form fields since Phase 3, and there was no way to save the result.
A grep for a public save API across all four crates returned nothing.

Design decision: append a revision, never rewrite. The original bytes are
copied verbatim and changed objects are appended with a new xref chained
through /Prev. A full rewrite would be easier and wrong: it would silently
discard everything this parser does not yet model (structure trees, optional
content, embedded files), and it would invalidate any signature, foreclosing
a feature listed later in the same phase. ADR 0003 records this in full.

Two latent bugs surfaced while building it, both pre-existing:

- find_xref_start searched with windows(10) for the 9-byte keyword
  "startxref", so it never matched. Every parse silently fell through to a
  forward scan for the first "xref" in the file. On a single-revision
  document that happens to be correct; on an incrementally saved one it is
  the *oldest* revision, so a saved edit read back as its pre-edit value.
  This had no visible effect before because nothing produced multi-revision
  files.
- XRefTable::parse read one section and ignored /Prev entirely, so a
  multi-revision document lost every object the earlier revisions defined.
  It now walks the chain newest-first, keeping the first definition of each
  object, with a visited set against /Prev loops and bounds checks on the
  offsets, which come from the file and cannot be trusted. A bad link ends
  the chain instead of indexing out of bounds.

The xref unit fixture claimed startxref 408 in a 191-byte file and only ever
passed because of the windows(10) defect; it is corrected rather than
adjusted to keep passing.

Implementation:
- pdf-cos/src/incremental.rs: IncrementalUpdate builds one revision.
  Recomputes stream /Length so a caller cannot write an inconsistent one,
  emits xref subsections for contiguous runs, sizes /Size over the whole
  chain, and is byte-reproducible for a given set of edits.
- pdf-document/src/save.rs: turns dirty AcroForm fields into a revision,
  writing the new /V and a regenerated appearance stream referenced from
  /AP, keyed by state name for checkboxes and radios.

Refusals rather than partial saves: an encrypted document returns
SaveError::Encrypted, because writing plaintext objects into it would
corrupt the file; a source with no startxref or no /Root is refused; and a
save with no pending edits returns the input unchanged rather than growing
the file and churning its timestamp.

Tests: 10 acceptance tests in tests/save_roundtrip.rs covering the ADR merge
criteria. The central one reparses from the written bytes rather than
reusing in-memory state, so it tests the file rather than the writer against
itself. Also asserts three chained revisions still reparse with the newest
value winning, that pages and annotations survive a save, and that saving
never panics on the malformed corpus.

Known limitations, recorded in the ADR rather than glossed: cross-reference
streams and object streams are not written, and superseded objects are not
compacted.

Validation:
  TEST_TARGET=pdf ./tools/test-rust-clean.sh      (307 tests)
  TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh   (351 tests, 6 ignored)
Both rustfmt and clippy -D warnings clean.
2026-07-28 16:35:22 +00:00
b23df6a5cb feat(pdf): complete Phase 6 testing infrastructure
Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
Payment domain and storage / isolated-payment-tests (push) Has been cancelled
Phase 6 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md: "Real PDFs. Real
regressions. No test theater."

Step 6.1, corpus. 27 fixtures across basic, fonts, forms, annotations,
images, edge and malformed, in the layout the review specifies. They are
produced by tests/corpus/generate.py rather than committed as opaque blobs,
because a corpus you cannot read is a corpus you cannot trust; CI regenerates
them and fails if they differ. Hand-rolled rather than library-produced,
since fixtures for a parser must contain constructs a library refuses to
emit.

Step 6.2, corpus tests (pdf-document/tests/corpus.rs, 27 tests). Text,
vectors, Flate, multipage, CID fonts, every form field type, inherited field
keys, link actions, hidden annotations, XObjects, inline images with
embedded EI bytes, rotation, crop boxes, nested CTMs and content arrays.

Step 6.3, robustness (pdf-document/tests/robustness.rs, 3 tests) plus five
cargo-fuzz targets. cargo-fuzz needs nightly and libFuzzer so it cannot gate
a stable CI run; the harness covers the same ground deterministically by
mutating the real corpus with a fixed-seed PRNG, so a failure is reproducible
from the seed rather than only from a saved artefact. The fuzz targets remain
the deeper coverage-guided search and run on a schedule.

Three crashes on untrusted input, all found by this work and all previously
reachable from a malformed file:

- collect_pages_ref recursed forever on a /Kids cycle. Stack overflow aborts
  the process; it cannot be caught. Now tracks visited nodes and bounds depth.
- PdfDocument::resolve and the COS lexer recursed once per nesting level, so
  a file of 5000 open brackets overflowed the stack. Both are now bounded.
- decode_85_group multiplied an accumulator that a malformed group can
  overflow, and subtracted below zero on a digit outside the valid range.
  Both panic in a debug build. Now saturating.

Step 6.4, CI (.forgejo/workflows/pdf.yml). An engine job that runs the
corpus and robustness suites under rustfmt and clippy -D warnings; a separate
makepad-integration job so a missing system library is not reported as a PDF
regression; and a scheduled fuzz job. The engine job also enforces the two
architectural rules mechanically rather than in prose: no Makepad dependency
or import in the engine crates, and no process or URL launching anywhere in
them.

Also fixes .gitignore: the blanket *.pdf rule silently excluded all 27
fixtures, which would have left CI unable to run them on a fresh clone.

Validation:
  TEST_TARGET=pdf ./tools/test-rust-clean.sh      (233 tests)
  TEST_TARGET=pdf-ui ./tools/test-rust-clean.sh   (268 tests)
Both rustfmt and clippy -D warnings clean; all five fuzz targets compile.
2026-07-27 17:08:23 +00:00
3f107bf2a5 feat(pdf): complete Phase 3 document model with real form state
Phase 3 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md. The previous
form.rs was keyed by name strings, had no inheritance and no way to edit
anything, so the review item "delete PdfFormFilling and replace it with a
DocumentFormEditor that returns real errors" had no implementation.

Step 3.1 object identity:
- Fields are keyed by ObjRef, not by name. Widget-to-field identity is
  preserved across parse, and a field defined as a direct object is skipped
  rather than given an invented identity.
- PdfDocument::page_index_of() maps a page reference back to its index and
  returns None for a stranger.

Step 3.2 annotations:
- PdfDocument::page_annotations() reads /Annots and records the real page on
  every annotation; the hardcoded page_index: None is gone.
- Annotations expose a typed action() (OpenUri / GoToNamed / GoToPage). The
  document reports intent only and never opens anything itself (rule 5).
- contains_point() normalises the rectangle: a PDF /Rect is any two opposite
  corners, so an inverted one previously never hit-tested.

Step 3.3 form model:
- /Parent chain inheritance for FT, Ff, V, DV, DA, MaxLen and Opt, with the
  child key overriding the ancestor.
- /Ff resolved into concrete types: checkbox vs radio vs pushbutton, combo
  vs list box.
- DocumentFormEditor takes typed edits and returns FormError. Read-only,
  type-mismatched, over-MaxLen, non-option and unknown-state edits are all
  refused, and a refused edit leaves the field untouched and not dirty.
- MaxLen counts characters, not bytes.
- Checking a box uses the on state the widget declares, not an assumed /Yes.
- The field tree walk is depth-bounded so a cyclic /Kids cannot recurse
  until the stack dies.

Step 3.4 appearance generation (new appearance.rs):
- Document-level, not widget code. Generates text, multiline, choice and
  checkbox appearances as Form XObjects with correct /BBox and /Length.
- /DA parsing resolves font, size and colour, converting gray and CMYK.
- Auto-size (0 Tf) resolves to a size that fits the widget.
- Values are escaped, so a parenthesis in a value cannot terminate the
  string and corrupt the stream; a single-line value cannot break out via
  newlines; content is clipped to the widget box.
- Unsupported kinds (pushbutton, signature) and degenerate rectangles return
  AppearanceError instead of a blank stream that would erase the field.

Wiring: PdfDocument::acroform() parses the form against the real page tree,
which is what makes widget page resolution meaningful.

Tests: adds tests/acroform.pdf, a hand-written two-page AcroForm fixture
with an inherited field, a checkbox with /On and /Off states and a URI link,
all on page index 1 so any code defaulting to page 0 fails. 12 integration
tests assert the Phase 3 exit criterion against that real parsed file.

Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh
118 tests pass; rustfmt and clippy -D warnings clean.
2026-07-27 15:49:46 +00:00
af4ef6f170 fix(pdf): restore a compiling, warning-free PDF baseline
Some checks failed
Payment domain and storage / isolated-payment-tests (push) Has been cancelled
Phase 0 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md requires a clean
baseline before any feature work. The four pdf crates did not compile at all,
so every test claim about them was unverified.

Compile fixes:
- decode_lzw/decode_run_length returned Vec<u8> where callers expected
  PdfResult, so decode_stream_with_params did not type-check.
- interpret_ops called a current_state_mut() method that PdfDevice does not
  have; colour-space tracking now goes through explicit device hooks.
- image.rs used miniz_oxide without depending on it; PNG inflate now reuses
  the COS crate through a new pdf_cos::filter::inflate_zlib.

Correctness fixes found while making the code build:
- LZW and RunLength silently truncated malformed input and indexed unchecked;
  both now return typed errors (rule 6: no silent degradation).
- Tw/Tc/Tz were parsed and thrown away, and the " operator dropped its word
  and character spacing, so every advance after them drifted.
- TJ attached each kern to the preceding string instead of the following one
  and discarded a trailing kern entirely.
- GlyphWidths::width() fell back to default_width for out-of-range codes;
  PDF 32000-1 9.6.2.1 requires /MissingWidth.
- Text advances silently substituted a guessed font_size * 0.6 when no width
  table was present; ShowTextWithMetrics now carries advance_is_measured so
  callers can distinguish a measurement from an unknown.

Tests: two tests had never compiled and were wrong once they ran (WinAnsi
0x99 is U+2122 not U+2019; q/cm/l/Q records four commands not three). The
document test asserted only that the writer emits a %PDF header; replaced
with real page-tree, out-of-range and malformed-input coverage.

Adds a pdf target to tools/test-rust-clean.sh that tests the three
UI-independent crates bottom-up under rustfmt and clippy -D warnings.

Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh
72 tests pass; rustfmt and clippy -D warnings clean.
2026-07-27 15:33:15 +00:00
a2ea0ffc7c updated map 2026-07-26 18:00:48 +03:00
cc05abdc71 Initial commit 2026-07-26 19:38:26 +03:00