Commit 258fa32 ("Merge origin/main: resolve xref/document conflicts") is not
a merge - it has one parent - and it landed a PDF tree that does not pass
its own gates. Found by running the suite, not by reading the diff.
Four separate breakages:
1. xref_streams::a_wrong_type_at_startxref_is_named_not_guessed FAILED.
The conflict resolution replaced the diagnostic error with a generic
"expected xref or cross-reference stream". The fixture writes
/Type /Frobnicate and the parser has that name in hand; throwing it away
leaves the reader knowing only that this is not what we wanted, which is
the least useful half of the story. Restored, and extended to name a
stream with no /Type at all. This is an ADR 0013 merge criterion.
2. golden_render::golden_text_line_ops FAILED - the merge added the test
but not its golden file. Verified the output by hand against the spec
before blessing it: leading 14, lines at 700/684/670/656/642, which is
correct for Td/TD/T*/'/". A golden file blessed without reading it locks
in whatever bug exists.
That golden cannot witness the " operator's two spacing operands: word
and character spacing live in the graphics state and emit no render
command, so dropping both would leave the file byte-identical. Added a
test that asserts the state directly. Mutation-checked: removing the two
set_*_spacing calls fails it, and leaves the golden untouched.
3. clippy -D warnings failed with 6 errors on pdf-cos and pdf-document, so
CI's engine job could not have passed. Two unused imports, a dead
read_be (superseded by the `field` closure in parse_xref_stream), a
collapsible if-let, and - committed into the source -
// ... keep the entire top of the file unchanged until the resolve_num function ...
an editing instruction left in document.rs as a doc comment.
4. cargo fmt --check failed on document.rs and xref.rs.
Mutation-checked both fixes: reverting the xref message fails 1 test,
dropping the " spacing operands fails 1.
pdf: 730 passed. pdf-ui: 775 passed. Coverage 84.15%, floors met;
destinations.rs holds at 98.65%.
The last two items of NIGIG_PDF_FEATURE_PARITY_PLAN.md Phase 2. Design and
merge criteria in REVIEWS/adr/0014-pdf-type3-fonts-and-streaming.md.
TYPE 3 FONTS DREW NOTHING
A Type 3 font's glyphs are not outlines - they are content streams, listed
in /CharProcs and mapped to text space by /FontMatrix. Probing a document
with one:
fonts on page: ["T3"]
T3: subtype=Type3 base=Unknown
-> are the glyph procedures reachable? no CharProcs field exists
-> is /FontMatrix exposed? no field exists
The font was detected and then nothing could be done with it. /CharProcs and
/FontMatrix appeared nowhere in the crate, so the procedures were unreachable
and the text was silently invisible - a page that renders, reports no error,
and is missing content.
New pdf-document/src/type3.rs parses /FontMatrix, /CharProcs, /Differences,
/Widths, /FontBBox and the font's own /Resources, and resolves a character
code to its glyph procedure's decoded bytes. /FontMatrix is applied as
written rather than assumed to be the common 0.001 scale - Type 3 fonts
routinely use other matrices, which is the point of the entry. A missing
/CharProcs entry is a typed error naming the glyph, not a blank.
A THIRD BUG, FOUND WHILE WIRING d0/d1
The interpreter parsed both operators and discarded them:
PdfOp::Type3Width(_wx, _wy) => {}
PdfOp::Type3BBox(_x1, _y1, _x2, _y2) => {}
They are how a Type 3 glyph declares its advance, so even a renderer that
could draw the glyphs would stack them all at one point. Wiring them to the
device exposed that `d1` takes SIX operands - wx wy llx lly urx ury - and the
parser read four, so the "bounding box" was really the advance and the
advance was lost entirely. Now `Type3BBox { wx, wy, bbox }`, reading all six.
STREAMING INTERPRETATION
parse_content_stream materialised every operator into a Vec before
interpreting any of them: peak memory proportional to the whole content
stream, on a stream walked once and discarded. Adds ContentStreamIter and
interpret_streaming, with parse_content_stream reimplemented on top of the
iterator so there is ONE tokeniser rather than two that can drift.
Equivalence is proven, not asserted: a test compares both paths across every
corpus fixture, and a streaming_interpreter fuzz target compares them over
arbitrary bytes, which is where a divergence would actually hide.
4 corpus fixtures, 13 acceptance tests, 9 unit tests. Mutation-checked:
reverting d0 to a no-op fails glyph_advances_reach_the_device.
Phase 2 is now complete; the plan is updated with an item-by-item audit.
Several entries were already done (inline images, Do, text state, shading);
the plan's "biggest gap" was xref streams, closed in ADR 0013.
TEST_TARGET=pdf 615 -> 637, TEST_TARGET=pdf-ui 660 -> 682.
rustfmt and clippy -D warnings clean.
Phase 8 #7 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md ("PDF/UA
structure tree"). Design and merge criteria in
REVIEWS/adr/0011-pdf-structure-tree.md.
Investigating the accessibility gap surfaced four defects in the
marked-content operators the structure tree depends on. The first is not
an accessibility problem at all.
1. BMC never parsed, and corrupted the colour of everything after it.
The dispatcher matched b'B'+b'M' only when the third byte was a
delimiter; BMC's third byte is 'C', so the arm never fired. Because
the operator was never recognised it never CLEARED ITS OPERAND, and
the leftover /Tag shifted the operands of whatever came next:
1 0 0 rg -> RgbFill(1.0, 0.0, 0.0) red
/Span BMC 1 0 0 rg -> RgbFill(0.0, 1.0, 0.0) green
Tagged documents are precisely the ones containing BMC, so the
documents that tried hardest to be accessible rendered wrong colours.
This is the fifth instance of the operator-shadowing family already
fixed for cm, rg, gs, b/b* and end-of-stream text operators. ADR
0009's every_multi_char_operator_parses_as_itself test exists to stop
exactly this - and would have caught it, except BMC was one of two
operators excluded from its table as "genuinely unimplemented".
Excluding a known-broken operator from the test whose job is finding
broken operators is how it survived. The exclusion list is gone.
2. BDC discarded its property list, which carries /MCID - the only link
between a run of page content and the structure element describing
it. Without it a tree can be parsed but never attached to anything.
3. The op produced by BDC was named MarkContentBmc, and BMC produced
nothing. The names were the wrong way round, which is how the missing
arm survived review: the enum looked like it had a BMC case.
4. Found while fixing 2: the content lexer had no dictionary support at
all. It read `<` as a hex string without checking for a second `<`,
so `<</MCID 0>>` parsed as the string "0C0D0". Content streams now
parse direct objects properly, bounded at 16 levels; a single `<` is
still a hex string and a test pins that.
On top of that, new pdf-document/src/structure.rs: /StructTreeRoot,
/StructElem trees, /RoleMap resolution, depth-first reading order,
/Alt, /ActualText, /E, /Lang, and MCID-to-element lookup. Cycles in /K
are cut at the first repeat and reported by object number rather than
expanded to the depth bound.
Accessibility findings are mechanical checks reported as findings, NOT a
conformance verdict: there is no is_pdf_ua and no ComplianceReport, and
a mutation-checked tripwire test fails if either appears. Real PDF/UA
conformance needs human judgement - whether /Alt text is accurate is not
mechanically decidable - so claiming it from six checks would be exactly
the overclaim this codebase keeps removing.
7 corpus fixtures, 14 acceptance tests, 22 unit tests, and a
parse_content_dict fuzz target for the new object parser.
TEST_TARGET=pdf 536 -> 575, TEST_TARGET=pdf-ui 580 -> 619.
rustfmt and clippy -D warnings clean.
Phase 8 #5 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md
("Advanced transparency"). Design and merge criteria in
REVIEWS/adr/0009-pdf-transparency.md.
The headline defect was not that transparency was missing. `gs` was
MIS-PARSED as CloseStroke:
/GS0 gs 1 0 0 rg 100 100 200 200 re f
-> [CloseStroke, RgbFill, Rectangle, FillWinding]
so every page setting a graphics state gained a stroked path the
document never asked for, and lost its alpha, blend mode and soft mask
silently. Downstream everything was dead: StrokeExtGState/FillExtGState
were never constructed, set_fill_opacity was never called and emitted no
command when it was, and PdfPage::ext_gstate was read by no code at all.
New pdf-graphics/src/transparency.rs: full /ExtGState (ca, CA, BM,
SMask, LW, LC, LJ, ML, D, AIS, TK), all sixteen blend modes including
the four non-separable ones, soft masks with /S, /G, /BC and a /TR
evaluated through ADR 0006's PdfFunction, and /Group parsing. Wired end
to end: parser -> PdfOp::SetExtGState -> device -> RenderCommand ->
Makepad renderer.
Compositing is NOT claimed. Backdrop blending needs render-to-texture,
which an engine-neutral crate has no framebuffer for. Constant alpha is
applied because it needs no backdrop; blend modes and soft masks are
reported through TransparencyError::Unsupported rather than dropped,
because a silently ignored /Multiply looks exactly like a correct
/Normal.
The ADR required a test enumerating every multi-character operator, on
the grounds that fixing the third instance of a shadowing bug (after cm
and rg) without preventing the fourth is not a fix. It immediately found
three more live bugs, none of them transparency-related:
- `b` and `b*` dropped their close-path, mapping to FillStroke* instead
of CloseFillStroke*, so every closed-and-stroked path drew with a gap.
- Text operators within three bytes of the end of a content stream were
mis-parsed: the b'T' arm guarded `*i + 3 < len` while reading only two
bytes, so a stream ending in `/F1 12 Tf` parsed as FillWinding.
And the transparency-group fixture found a fourth:
- PdfPage::xobjects was empty for every page of every document.
extract_xobjects called doc.resolve(), which follows the reference,
then asked the resolved object for as_ref() - always None - and only
accepted a bare dict when every XObject is a stream. No `Do` operator
could be resolved through the page model. Same defect as the one that
once destroyed annotation object references; it now has its own
regression test.
T* and BMC are genuinely unimplemented and are deliberately excluded
from the guard's table rather than papered over.
7 corpus fixtures, 12 acceptance tests, 32 unit tests asserting the
§11.3.5 formulas (not our own output), and a parse_ext_gstate fuzz
target.
TEST_TARGET=pdf 451 -> 497, TEST_TARGET=pdf-ui 495 -> 541.
rustfmt and clippy -D warnings clean.
Phase 2 of REVIEWS/DART_PDF_VS_MAKEPAD_PDF_GAP_ANALYSIS.md. The exit
criterion is that a real content stream renders through the device path and
is verifiable; previously nothing could fail, so nothing was proven.
Step 2.3, the operations the review calls "the lies", all reached a dead end
in the interpreter. Each now reaches the device:
- Inline images: BI/ID/EI were emitted as two empty marker ops and the
payload was discarded, so an inline image could never be drawn. They are
now parsed into a single InlineImage op carrying the dictionary and bytes.
Abbreviated keys (/W /H /BPC /CS /F) and colour-space and filter
abbreviations are expanded. The EI scan requires delimiters on both sides
so binary data containing the bytes "EI" does not truncate the image, and
an unterminated image yields no image rather than invented pixels.
- Do (XObject) was an empty match arm. The interpreter cannot resolve a
resource name, so it now reports it through PdfDevice::paint_x_object and
the device performs the lookup.
- set_dash and set_miter_limit only mutated interpreter-local state and
emitted no command, so dashes never reached any renderer.
Three further defects surfaced while reviewing the generated goldens, all
of which silently corrupted output rather than failing:
- `cm` was never parsed at all. The single-character `m` arm matched first
and consumed it as a moveto, so every CTM change in every document was
lost and content drew at the wrong position. Two-character operators are
now tested before their one-character prefixes.
- `rg` and `RG` were shadowed by the `r` and `R` arms, so an RGB fill was
read as a single-component grey: `0.1 0.2 0.3 rg` produced 0.1 0.1 0.1.
- ImageInfo defaulted a missing /Filter to FlateDecode. An absent /Filter
means the data is stored raw, so every uncompressed image was undecodable.
Testing: adds format_commands(), a deterministic one-line-per-command text
form of a RenderCommand list, and eight golden files covering vector paths,
text, kerning and spacing, dash and stroke parameters, fill rules, inline
images, XObjects and colour operators. Floats are fixed-precision and
negative zero is normalised so no spurious diffs appear. UPDATE_GOLDEN=1
regenerates them for review.
Also fixes .gitignore: blanket *.txt and *.pdf rules were silently excluding
the golden expectations and the AcroForm fixture added in the previous
commit, which would have left both test suites unable to run on a fresh
clone.
Validation: TEST_TARGET=pdf ./tools/test-rust-clean.sh
132 tests pass; rustfmt and clippy -D warnings clean.
