Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
email.yml / feat(pdf): JPEG 2000 decoding — Phase 3 complete, all three codecs (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
The last codec ADR 0015 deferred. The plan recorded the blocker as a dependency decision, not an algorithm: openjpeg would add a C dependency that breaks the Android cross-compile. This is pure Rust and adds no dependency at all. It shares the MQ arithmetic decoder with JBIG2 — T.800 and T.88 specify the same coder — so the previous tranche paid for most of this one. Context::with_state moved onto the shared type because JPEG 2000 starts three of its nineteen contexts away from state 0 and JBIG2 starts all of them at 0. Implemented: codestream and JP2 container parsing, packet headers with tag trees and the bit-stuffing rule, EBCOT tier-1 (all three passes, four zero-coding context tables, run-length mode), both 5/3 reversible and 9/7 irreversible wavelets, RCT and ICT, arbitrary decomposition levels, and multiple components. Refused by name: multiple tiles, custom precinct partitions, code-block style options, COC/QCC/RGN/POC overrides, subsampled components. Each error says which feature the file needs. This matters more here than anywhere else in the stack, because a JPEG 2000 decoder that quietly skips something does not fail — it returns a slightly soft or banded image that looks entirely fine. That property also dictates how this is tested. Fixtures are produced by OpenJPEG via Pillow and compared **exactly**, sample for sample: the fixtures are lossless 5/3 so no tolerance is needed, and a tolerance is where a subtly wrong decoder hides. Four images — grayscale raw codestream, the same in a JP2 container, a larger one whose tag trees actually branch, and RGB. A generator script is checked in beside them so CI can prove the fixtures still match what produced them. Verified by mutation. The first round was misleading and is worth recording, because it is the same lesson as ADR 0017: DC level shift dropped 3 fail 5/3 lifting rounding changed 2 fail RCT sign flipped PASSED <- survived RCT components swapped PASSED <- survived cleanup run-length disabled PASSED <- survived sign-context XOR dropped PASSED <- survived Four mutations survived because Pillow writes MCT=0 by default, so the RGB fixture coded its three components independently and never reached the colour transform at all. The RCT branch was completely untested while appearing covered — an untested branch that looks tested is worse than one that looks missing. Added rgb8_mct.j2k with mct=1; all four now fail. The header bit-stuffing mutation is caught by the unit test rather than the round-trip. Two real defects found while writing the tests: - A corrupt marker length in a tile-part header walked the read cursor past the codestream and panicked on a slice. Found by the corruption sweep, not by review. The sweep now truncates at every length and flips every byte of a real file, and asserts only that nothing panics. - The 9/7 flat-signal test initially asserted an amplitude I had derived from my own arithmetic. That is a test agreeing with the code by construction. It now asserts flatness — a ripple means the lifting or the edge extension is wrong — and the amplitude is pinned by the OpenJPEG round-trips instead, which use pixels this code did not produce. Also removed two dead fields and an unused parameter that clippy found: Subband::x0/y0 are always zero in the single-tile case this supports, and dead state implying multi-tile support exists is worse than no state. JPX decodes on the image path, like JBIG2, because the codestream carries its own geometry; it stays in REFUSED_CODECS with a reason string saying where it is decoded rather than that it is missing. Engine suite 866 -> 920. Coverage 85.66% -> 86.16%; jpx.rs at 93.72% with a floor at 88. Phase 3 is complete: CCITT, JBIG2 and JPX all land, and the plan is updated to say so and to record how the two gating questions — JBIG2's CVE record and JPX's C dependency — were actually answered.
380 lines
14 KiB
Rust
380 lines
14 KiB
Rust
//! The bitonal image path: `CCITTFaxDecode` and `JBIG2Decode`.
|
|
//!
|
|
//! Both were refused by name until now. Both produce 1 bit per pixel, and
|
|
//! both are decoded on the *image* path rather than through the generic
|
|
//! filter facade — CCITT because its `/DecodeParms` defaults depend on the
|
|
//! image dimensions, JBIG2 because it needs `/Width` and `/Height` outright.
|
|
//!
|
|
//! Every assertion here checks *pixels*, not sizes or `Ok`-ness. ADR 0016
|
|
//! records a JPEG decoder that was a stub returning a correctly sized black
|
|
//! rectangle: it passed every test that asserted a length. The tests below
|
|
//! would have caught it, because black and white are different colours and
|
|
//! they say which they expect.
|
|
|
|
use nigig_pdf_cos::{PdfDict, PdfObj};
|
|
use nigig_pdf_graphics::image::ImageInfo;
|
|
|
|
/// Assemble a bit string, MSB first.
|
|
fn bits(s: &str) -> Vec<u8> {
|
|
let cleaned: String = s.chars().filter(|c| *c == '0' || *c == '1').collect();
|
|
let mut out = Vec::new();
|
|
let mut cur = 0u8;
|
|
let mut n = 0;
|
|
for c in cleaned.chars() {
|
|
cur = (cur << 1) | u8::from(c == '1');
|
|
n += 1;
|
|
if n == 8 {
|
|
out.push(cur);
|
|
cur = 0;
|
|
n = 0;
|
|
}
|
|
}
|
|
if n > 0 {
|
|
out.push(cur << (8 - n));
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Read an RGBA buffer back as a picture, so a failure prints something a
|
|
/// human can look at rather than a hex dump.
|
|
fn render(rgba: &[u8], width: usize, height: usize) -> Vec<String> {
|
|
(0..height)
|
|
.map(|y| {
|
|
(0..width)
|
|
.map(|x| {
|
|
let i = (y * width + x) * 4;
|
|
if rgba[i] < 128 {
|
|
'#'
|
|
} else {
|
|
'.'
|
|
}
|
|
})
|
|
.collect()
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
fn image_dict(filter: &str, width: i64, height: i64, parms: Option<PdfObj>) -> PdfDict {
|
|
let mut d = PdfDict::new();
|
|
d.set("Width", PdfObj::Int(width));
|
|
d.set("Height", PdfObj::Int(height));
|
|
d.set("BitsPerComponent", PdfObj::Int(1));
|
|
d.set("ColorSpace", PdfObj::Name("DeviceGray".into()));
|
|
d.set("Filter", PdfObj::Name(filter.into()));
|
|
if let Some(p) = parms {
|
|
d.set("DecodeParms", p);
|
|
}
|
|
d
|
|
}
|
|
|
|
fn ccitt_parms(k: i64, columns: i64, rows: i64) -> PdfObj {
|
|
let mut d = PdfDict::new();
|
|
d.set("K", PdfObj::Int(k));
|
|
d.set("Columns", PdfObj::Int(columns));
|
|
d.set("Rows", PdfObj::Int(rows));
|
|
PdfObj::Dict(d)
|
|
}
|
|
|
|
// ------------------------------------------------------------------ CCITT
|
|
|
|
#[test]
|
|
fn a_ccitt_image_decodes_to_the_right_pixels() {
|
|
// 4 white then 4 black, one row.
|
|
let dict = image_dict("CCITTFaxDecode", 8, 1, Some(ccitt_parms(0, 8, 1)));
|
|
let img = ImageInfo::from_dict(&dict, "Im0", bits("1011 011")).expect("builds");
|
|
let rgba = img
|
|
.decode_to_rgba()
|
|
.expect("CCITT decodes on the image path");
|
|
assert_eq!(rgba.len(), 8 * 4, "8 pixels, 4 bytes each");
|
|
assert_eq!(render(&rgba, 8, 1), vec!["....####"]);
|
|
}
|
|
|
|
/// Without `/DecodeParms` the image's own `/Width` and `/Height` must be
|
|
/// used, not the spec's 1728-column fax scan line. Falling back to 1728
|
|
/// would fail to decode every non-fax CCITT image in existence.
|
|
#[test]
|
|
fn a_ccitt_image_without_parms_uses_its_own_dimensions() {
|
|
let dict = image_dict("CCITTFaxDecode", 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", bits("1011 011")).expect("builds");
|
|
let rgba = img
|
|
.decode_to_rgba()
|
|
.expect("decodes using /Width and /Height");
|
|
assert_eq!(render(&rgba, 8, 1), vec!["....####"]);
|
|
}
|
|
|
|
/// `/DecodeParms` that omits `/Columns` must still fall back to `/Width`.
|
|
#[test]
|
|
fn partial_ccitt_parms_fall_back_to_the_image_dimensions() {
|
|
let mut p = PdfDict::new();
|
|
p.set("K", PdfObj::Int(0));
|
|
let dict = image_dict("CCITTFaxDecode", 8, 1, Some(PdfObj::Dict(p)));
|
|
let img = ImageInfo::from_dict(&dict, "Im0", bits("1011 011")).expect("builds");
|
|
let rgba = img
|
|
.decode_to_rgba()
|
|
.expect("an absent /Columns falls back to /Width");
|
|
assert_eq!(render(&rgba, 8, 1), vec!["....####"]);
|
|
}
|
|
|
|
/// The parallel-array case. `[/FlateDecode /CCITTFaxDecode]` has a matching
|
|
/// `/DecodeParms` array, and taking `arr[0]` hands the Flate parameters to
|
|
/// the fax decoder — the exact bug ADR 0015 records for the old chain code.
|
|
#[test]
|
|
fn ccitt_parms_are_taken_from_the_matching_array_slot() {
|
|
let mut dict = PdfDict::new();
|
|
dict.set("Width", PdfObj::Int(8));
|
|
dict.set("Height", PdfObj::Int(1));
|
|
dict.set("BitsPerComponent", PdfObj::Int(1));
|
|
dict.set("ColorSpace", PdfObj::Name("DeviceGray".into()));
|
|
dict.set(
|
|
"Filter",
|
|
PdfObj::Array(vec![
|
|
PdfObj::Name("ASCIIHexDecode".into()),
|
|
PdfObj::Name("CCITTFaxDecode".into()),
|
|
]),
|
|
);
|
|
dict.set(
|
|
"DecodeParms",
|
|
PdfObj::Array(vec![PdfObj::Null, ccitt_parms(0, 8, 1)]),
|
|
);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", Vec::new()).expect("builds");
|
|
let parms = img.ccitt_parms.as_ref().expect("found the CCITT slot");
|
|
assert_eq!(
|
|
parms.get_int("Columns"),
|
|
Some(8),
|
|
"the parms must come from index 1, matching CCITT's place in /Filter"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_corrupt_ccitt_image_returns_none_rather_than_a_black_rectangle() {
|
|
let dict = image_dict("CCITTFaxDecode", 8, 4, Some(ccitt_parms(0, 8, 4)));
|
|
// One row of data where four are declared.
|
|
let img = ImageInfo::from_dict(&dict, "Im0", bits("1011 011")).expect("builds");
|
|
assert!(
|
|
img.decode_to_rgba().is_none(),
|
|
"a short image must fail visibly, not decode to plausible pixels"
|
|
);
|
|
}
|
|
|
|
// ------------------------------------------------------------------ JBIG2
|
|
|
|
/// Build a minimal embedded JBIG2 stream carrying one MMR generic region.
|
|
///
|
|
/// MMR is used for the round-trip because it is CCITT G4, so the expected
|
|
/// pixels can be stated by hand with confidence rather than being whatever
|
|
/// the arithmetic decoder happens to produce.
|
|
fn jbig2_mmr_stream(width: u32, height: u32, coded: &[u8]) -> Vec<u8> {
|
|
let mut body = Vec::new();
|
|
body.extend_from_slice(&width.to_be_bytes());
|
|
body.extend_from_slice(&height.to_be_bytes());
|
|
body.extend_from_slice(&0u32.to_be_bytes()); // x
|
|
body.extend_from_slice(&0u32.to_be_bytes()); // y
|
|
body.push(0); // external combination operator
|
|
body.push(1); // flags: MMR
|
|
body.extend_from_slice(coded);
|
|
|
|
let mut out = Vec::new();
|
|
out.extend_from_slice(&1u32.to_be_bytes()); // segment number
|
|
out.push(38); // immediate generic region, 1-byte page association
|
|
out.push(0x00); // no referred-to segments
|
|
out.push(0x01); // page 1
|
|
out.extend_from_slice(&(body.len() as u32).to_be_bytes());
|
|
out.extend_from_slice(&body);
|
|
out
|
|
}
|
|
|
|
#[test]
|
|
fn a_jbig2_image_decodes_to_the_right_pixels() {
|
|
// G4 horizontal mode: white 4, black 4.
|
|
let stream = jbig2_mmr_stream(8, 1, &bits("001 1011 011"));
|
|
let dict = image_dict("JBIG2Decode", 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", stream).expect("builds");
|
|
let rgba = img
|
|
.decode_to_rgba()
|
|
.expect("JBIG2 decodes on the image path");
|
|
assert_eq!(
|
|
render(&rgba, 8, 1),
|
|
vec!["....####"],
|
|
"JBIG2 is natively 1=black and must be inverted to PDF convention"
|
|
);
|
|
}
|
|
|
|
/// A JBIG2 image and the equivalent CCITT image must produce the *same*
|
|
/// picture. They are the same coding scheme underneath, so a disagreement
|
|
/// means one of the two conventions is inverted — the commonest bug in
|
|
/// this area and invisible unless the two are compared directly.
|
|
#[test]
|
|
fn jbig2_mmr_and_ccitt_agree_on_the_same_coded_bits() {
|
|
let coded = bits("001 1011 011");
|
|
|
|
let ccitt_dict = image_dict("CCITTFaxDecode", 8, 1, Some(ccitt_parms(-1, 8, 1)));
|
|
let ccitt = ImageInfo::from_dict(&ccitt_dict, "Im0", coded.clone())
|
|
.expect("builds")
|
|
.decode_to_rgba()
|
|
.expect("CCITT decodes");
|
|
|
|
let jbig2_dict = image_dict("JBIG2Decode", 8, 1, None);
|
|
let jbig2 = ImageInfo::from_dict(&jbig2_dict, "Im1", jbig2_mmr_stream(8, 1, &coded))
|
|
.expect("builds")
|
|
.decode_to_rgba()
|
|
.expect("JBIG2 decodes");
|
|
|
|
assert_eq!(
|
|
ccitt, jbig2,
|
|
"the same G4 bits must give the same pixels through either codec"
|
|
);
|
|
}
|
|
|
|
/// A declared `/JBIG2Globals` must refuse. Globals carry symbol
|
|
/// dictionaries, which are not decoded; decoding without them yields a
|
|
/// blank or partial image that every caller would treat as a success.
|
|
#[test]
|
|
fn a_declared_jbig2_globals_refuses_rather_than_decoding_without_it() {
|
|
let mut parms = PdfDict::new();
|
|
// In a real file this is an indirect reference this layer cannot
|
|
// resolve; what matters is that its presence is noticed.
|
|
parms.set(
|
|
"JBIG2Globals",
|
|
PdfObj::Ref(nigig_pdf_cos::ObjRef { num: 9, gen: 0 }),
|
|
);
|
|
let dict = image_dict("JBIG2Decode", 8, 1, Some(PdfObj::Dict(parms)));
|
|
let img = ImageInfo::from_dict(&dict, "Im0", jbig2_mmr_stream(8, 1, &bits("001 1011 011")))
|
|
.expect("builds");
|
|
assert!(
|
|
img.jbig2_globals_declared,
|
|
"the declaration must be noticed even when it cannot be resolved"
|
|
);
|
|
assert!(
|
|
img.decode_to_rgba().is_none(),
|
|
"decoding without the declared symbol dictionary would be a \
|
|
silently wrong image"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn an_undeclared_globals_does_not_block_decoding() {
|
|
let dict = image_dict("JBIG2Decode", 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", jbig2_mmr_stream(8, 1, &bits("001 1011 011")))
|
|
.expect("builds");
|
|
assert!(!img.jbig2_globals_declared);
|
|
assert!(img.decode_to_rgba().is_some());
|
|
}
|
|
|
|
#[test]
|
|
fn a_jbig2_stream_needing_a_text_region_returns_none() {
|
|
// Segment type 6: immediate text region. Not decoded, by decision.
|
|
let mut stream = Vec::new();
|
|
stream.extend_from_slice(&1u32.to_be_bytes());
|
|
stream.push(6);
|
|
stream.push(0x00);
|
|
stream.push(0x01);
|
|
stream.extend_from_slice(&2u32.to_be_bytes());
|
|
stream.extend_from_slice(&[0, 0]);
|
|
|
|
let dict = image_dict("JBIG2Decode", 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", stream).expect("builds");
|
|
assert!(
|
|
img.decode_to_rgba().is_none(),
|
|
"a text region must not decode to a blank page"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_zero_sized_bitonal_image_returns_none() {
|
|
for (w, h) in [(0i64, 4i64), (4, 0)] {
|
|
let dict = image_dict("JBIG2Decode", w, h, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", vec![0; 16]).expect("builds");
|
|
assert!(img.decode_to_rgba().is_none(), "{w}x{h} has no pixels");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn both_codecs_report_as_bitonal_and_others_do_not() {
|
|
for filter in ["CCITTFaxDecode", "JBIG2Decode"] {
|
|
let dict = image_dict(filter, 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", Vec::new()).expect("builds");
|
|
assert!(img.is_bitonal_fax(), "{filter} is a bitonal codec");
|
|
}
|
|
for filter in ["DCTDecode", "FlateDecode", "JPXDecode"] {
|
|
let dict = image_dict(filter, 8, 1, None);
|
|
let img = ImageInfo::from_dict(&dict, "Im0", Vec::new()).expect("builds");
|
|
assert!(!img.is_bitonal_fax(), "{filter} is not a bitonal codec");
|
|
}
|
|
}
|
|
|
|
// ------------------------------------------------------------- JPEG 2000
|
|
|
|
fn jpx_corpus(name: &str) -> Vec<u8> {
|
|
let path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
|
.join("../tests/corpus/jpx")
|
|
.join(name);
|
|
std::fs::read(&path).unwrap_or_else(|e| panic!("missing {}: {e}", path.display()))
|
|
}
|
|
|
|
/// A JPX image decodes through the ordinary image path, to real pixels.
|
|
///
|
|
/// The gradient fixture is a horizontal ramp, so the assertion is that the
|
|
/// left edge is dark and the right edge is light. That is deliberately a
|
|
/// statement about the *content*: a decoder returning a uniform grey, or a
|
|
/// transposed image, passes any check on the buffer length.
|
|
#[test]
|
|
fn a_jpx_image_decodes_through_the_image_path() {
|
|
let mut dict = PdfDict::new();
|
|
dict.set("Width", PdfObj::Int(8));
|
|
dict.set("Height", PdfObj::Int(8));
|
|
dict.set("BitsPerComponent", PdfObj::Int(8));
|
|
dict.set("ColorSpace", PdfObj::Name("DeviceGray".into()));
|
|
dict.set("Filter", PdfObj::Name("JPXDecode".into()));
|
|
|
|
let img = ImageInfo::from_dict(&dict, "Im0", jpx_corpus("gray8_lossless.j2k")).expect("builds");
|
|
let rgba = img.decode_to_rgba().expect("JPX decodes on the image path");
|
|
assert_eq!(rgba.len(), 8 * 8 * 4);
|
|
|
|
let px = |x: usize, y: usize| rgba[(y * 8 + x) * 4];
|
|
assert!(
|
|
px(0, 0) < px(7, 0),
|
|
"the gradient must run left-to-right: got {} at x=0 and {} at x=7",
|
|
px(0, 0),
|
|
px(7, 0)
|
|
);
|
|
assert!(
|
|
rgba.chunks(4).any(|p| p[0] != rgba[0]),
|
|
"a uniform image means the codestream was not really decoded"
|
|
);
|
|
}
|
|
|
|
/// Three components must come out as three different channels. A decoder
|
|
/// that dropped the colour transform would return a grey image here and
|
|
/// pass every check that only looks at sizes.
|
|
#[test]
|
|
fn a_jpx_rgb_image_decodes_with_distinct_channels() {
|
|
let mut dict = PdfDict::new();
|
|
dict.set("Width", PdfObj::Int(8));
|
|
dict.set("Height", PdfObj::Int(8));
|
|
dict.set("BitsPerComponent", PdfObj::Int(8));
|
|
dict.set("ColorSpace", PdfObj::Name("DeviceRGB".into()));
|
|
dict.set("Filter", PdfObj::Name("JPXDecode".into()));
|
|
|
|
let img = ImageInfo::from_dict(&dict, "Im0", jpx_corpus("rgb8_lossless.j2k")).expect("builds");
|
|
let rgba = img.decode_to_rgba().expect("JPX RGB decodes");
|
|
assert_eq!(rgba.len(), 8 * 8 * 4);
|
|
|
|
let differs = rgba.chunks(4).any(|p| p[0] != p[1] || p[1] != p[2]);
|
|
assert!(
|
|
differs,
|
|
"every pixel came out grey, so the colour transform was skipped"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_corrupt_jpx_image_returns_none() {
|
|
let mut dict = PdfDict::new();
|
|
dict.set("Width", PdfObj::Int(8));
|
|
dict.set("Height", PdfObj::Int(8));
|
|
dict.set("Filter", PdfObj::Name("JPXDecode".into()));
|
|
let img = ImageInfo::from_dict(&dict, "Im0", vec![0xFF, 0x4F, 0x00, 0x01]).expect("builds");
|
|
assert!(
|
|
img.decode_to_rgba().is_none(),
|
|
"a malformed codestream must fail visibly"
|
|
);
|
|
}
|