Some checks failed
PDF engine / engine (push) Has been cancelled
PDF engine / makepad-integration (push) Has been cancelled
PDF engine / fuzz (push) Has been cancelled
email.yml / feat(pdf): JPEG 2000 decoding — Phase 3 complete, all three codecs (push) Failing after 0s
repo hygiene / hygiene (push) Has been cancelled
The last codec ADR 0015 deferred. The plan recorded the blocker as a dependency decision, not an algorithm: openjpeg would add a C dependency that breaks the Android cross-compile. This is pure Rust and adds no dependency at all. It shares the MQ arithmetic decoder with JBIG2 — T.800 and T.88 specify the same coder — so the previous tranche paid for most of this one. Context::with_state moved onto the shared type because JPEG 2000 starts three of its nineteen contexts away from state 0 and JBIG2 starts all of them at 0. Implemented: codestream and JP2 container parsing, packet headers with tag trees and the bit-stuffing rule, EBCOT tier-1 (all three passes, four zero-coding context tables, run-length mode), both 5/3 reversible and 9/7 irreversible wavelets, RCT and ICT, arbitrary decomposition levels, and multiple components. Refused by name: multiple tiles, custom precinct partitions, code-block style options, COC/QCC/RGN/POC overrides, subsampled components. Each error says which feature the file needs. This matters more here than anywhere else in the stack, because a JPEG 2000 decoder that quietly skips something does not fail — it returns a slightly soft or banded image that looks entirely fine. That property also dictates how this is tested. Fixtures are produced by OpenJPEG via Pillow and compared **exactly**, sample for sample: the fixtures are lossless 5/3 so no tolerance is needed, and a tolerance is where a subtly wrong decoder hides. Four images — grayscale raw codestream, the same in a JP2 container, a larger one whose tag trees actually branch, and RGB. A generator script is checked in beside them so CI can prove the fixtures still match what produced them. Verified by mutation. The first round was misleading and is worth recording, because it is the same lesson as ADR 0017: DC level shift dropped 3 fail 5/3 lifting rounding changed 2 fail RCT sign flipped PASSED <- survived RCT components swapped PASSED <- survived cleanup run-length disabled PASSED <- survived sign-context XOR dropped PASSED <- survived Four mutations survived because Pillow writes MCT=0 by default, so the RGB fixture coded its three components independently and never reached the colour transform at all. The RCT branch was completely untested while appearing covered — an untested branch that looks tested is worse than one that looks missing. Added rgb8_mct.j2k with mct=1; all four now fail. The header bit-stuffing mutation is caught by the unit test rather than the round-trip. Two real defects found while writing the tests: - A corrupt marker length in a tile-part header walked the read cursor past the codestream and panicked on a slice. Found by the corruption sweep, not by review. The sweep now truncates at every length and flips every byte of a real file, and asserts only that nothing panics. - The 9/7 flat-signal test initially asserted an amplitude I had derived from my own arithmetic. That is a test agreeing with the code by construction. It now asserts flatness — a ripple means the lifting or the edge extension is wrong — and the amplitude is pinned by the OpenJPEG round-trips instead, which use pixels this code did not produce. Also removed two dead fields and an unused parameter that clippy found: Subband::x0/y0 are always zero in the single-tile case this supports, and dead state implying multi-tile support exists is worse than no state. JPX decodes on the image path, like JBIG2, because the codestream carries its own geometry; it stays in REFUSED_CODECS with a reason string saying where it is decoded rather than that it is missing. Engine suite 866 -> 920. Coverage 85.66% -> 86.16%; jpx.rs at 93.72% with a floor at 88. Phase 3 is complete: CCITT, JBIG2 and JPX all land, and the plan is updated to say so and to record how the two gating questions — JBIG2's CVE record and JPX's C dependency — were actually answered.
188 lines
7 KiB
Rust
188 lines
7 KiB
Rust
//! JPEG 2000 decoded against real encoder output.
|
|
//!
|
|
//! Every other test in `jpx.rs` checks a parser or a refusal. These check
|
|
//! the only thing that ultimately matters: that a file produced by a
|
|
//! *different* implementation — OpenJPEG, via Pillow — decodes to the
|
|
//! pixels that went into it.
|
|
//!
|
|
//! This matters more for JPEG 2000 than for the other codecs. A wavelet bug
|
|
//! does not raise an error and does not produce garbage; it produces a
|
|
//! slightly soft or slightly banded image that looks entirely plausible. A
|
|
//! dropped coding pass is invisible without a reference. Only a comparison
|
|
//! against an independent encoder's pixels can tell the difference, which
|
|
//! is why the fixtures are checked in alongside the values they must
|
|
//! reproduce.
|
|
//!
|
|
//! The fixtures are lossless (5/3 reversible, no quantisation), so the
|
|
//! comparison is exact rather than approximate. An irreversible fixture
|
|
//! would need a tolerance, and a tolerance is where a wrong decoder hides.
|
|
//!
|
|
//! Regenerate with `crates/apps/pdf/tests/corpus/jpx/generate.py`.
|
|
|
|
use std::path::PathBuf;
|
|
|
|
use nigig_pdf_cos::jpx;
|
|
|
|
fn corpus(name: &str) -> Vec<u8> {
|
|
let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
|
.join("../tests/corpus/jpx")
|
|
.join(name);
|
|
std::fs::read(&path).unwrap_or_else(|e| panic!("missing {}: {e}", path.display()))
|
|
}
|
|
|
|
/// The expected pixels, interleaved as the encoder saw them.
|
|
///
|
|
/// A tiny extractor rather than a serde dependency: `pdf-cos` deliberately
|
|
/// carries almost none, and the file is a flat object of name to array of
|
|
/// integers written by the generator next to it.
|
|
fn expected(name: &str) -> Vec<u8> {
|
|
let raw = corpus("expected.json");
|
|
let text = String::from_utf8(raw).expect("expected.json is UTF-8");
|
|
let key = format!("\"{name}\":");
|
|
let start = text
|
|
.find(&key)
|
|
.unwrap_or_else(|| panic!("no entry for {name} in expected.json"))
|
|
+ key.len();
|
|
let rest = &text[start..];
|
|
let open = rest.find('[').expect("array opens");
|
|
let close = rest.find(']').expect("array closes");
|
|
rest[open + 1..close]
|
|
.split(',')
|
|
.filter(|s| !s.trim().is_empty())
|
|
.map(|s| s.trim().parse::<u8>().expect("a byte"))
|
|
.collect()
|
|
}
|
|
|
|
/// Interleave planar components the way the encoder's `tobytes()` did.
|
|
fn interleave(img: &jpx::Jpx) -> Vec<u8> {
|
|
let n = (img.width * img.height) as usize;
|
|
let mut out = Vec::with_capacity(n * img.components.len());
|
|
for i in 0..n {
|
|
for c in &img.components {
|
|
out.push(c[i].clamp(0, 255) as u8);
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
fn assert_decodes_exactly(name: &str, width: u32, height: u32, components: usize) {
|
|
let img = jpx::decode(&corpus(name)).unwrap_or_else(|e| panic!("{name} failed to decode: {e}"));
|
|
assert_eq!(img.width, width, "{name} width");
|
|
assert_eq!(img.height, height, "{name} height");
|
|
assert_eq!(img.components.len(), components, "{name} component count");
|
|
|
|
let got = interleave(&img);
|
|
let want = expected(name);
|
|
assert_eq!(
|
|
got.len(),
|
|
want.len(),
|
|
"{name} produced the wrong sample count"
|
|
);
|
|
|
|
if got != want {
|
|
// Print the first disagreement rather than two long arrays: where
|
|
// it diverges says which stage is wrong far faster than the values.
|
|
let at = got.iter().zip(&want).position(|(a, b)| a != b).unwrap_or(0);
|
|
let px = at / components;
|
|
panic!(
|
|
"{name} differs at sample {at} (pixel {},{}): got {}, want {}\n\
|
|
got {:?}\nwant {:?}",
|
|
px % width as usize,
|
|
px / width as usize,
|
|
got[at],
|
|
want[at],
|
|
&got[..got.len().min(32)],
|
|
&want[..want.len().min(32)],
|
|
);
|
|
}
|
|
}
|
|
|
|
/// A lossless 8x8 grayscale gradient, raw codestream, 3 decomposition
|
|
/// levels. The simplest thing that exercises the whole pipeline: packet
|
|
/// headers, tag trees, EBCOT, and three rounds of inverse wavelet.
|
|
#[test]
|
|
fn a_lossless_grayscale_codestream_decodes_exactly() {
|
|
assert_decodes_exactly("gray8_lossless.j2k", 8, 8, 1);
|
|
}
|
|
|
|
/// The same image in a JP2 container must give identical pixels — the
|
|
/// container is packaging, not content.
|
|
#[test]
|
|
fn the_jp2_container_decodes_to_the_same_pixels_as_the_raw_codestream() {
|
|
let raw = jpx::decode(&corpus("gray8_lossless.j2k")).expect("raw decodes");
|
|
let boxed = jpx::decode(&corpus("gray8_lossless.jp2")).expect("jp2 decodes");
|
|
assert_eq!(
|
|
raw.components, boxed.components,
|
|
"the JP2 wrapper must not change a single sample"
|
|
);
|
|
}
|
|
|
|
/// A larger image with more code-blocks per subband, so the packet header's
|
|
/// tag trees actually branch rather than being 1x1.
|
|
#[test]
|
|
fn a_larger_lossless_image_decodes_exactly() {
|
|
assert_decodes_exactly("gray16_lossless.j2k", 16, 16, 1);
|
|
}
|
|
|
|
/// Three components coded independently — Pillow's default, with the
|
|
/// multiple component transform switched *off*.
|
|
#[test]
|
|
fn a_lossless_rgb_image_decodes_exactly() {
|
|
assert_decodes_exactly("rgb8_lossless.j2k", 8, 8, 3);
|
|
}
|
|
|
|
/// The same image with the reversible colour transform actually enabled.
|
|
///
|
|
/// This fixture exists because a mutation proved it had to. Flipping the
|
|
/// sign in the RCT left the whole suite green: Pillow writes `MCT=0` by
|
|
/// default, so `rgb8_lossless.j2k` never reaches the transform at all and
|
|
/// the RCT branch was entirely untested while looking covered. With
|
|
/// `mct=1` the same mutation fails here. An untested branch that *looks*
|
|
/// tested is worse than one that looks missing.
|
|
#[test]
|
|
fn an_rgb_image_using_the_colour_transform_decodes_exactly() {
|
|
assert_decodes_exactly("rgb8_mct.j2k", 8, 8, 3);
|
|
}
|
|
|
|
/// Truncating a codestream must not yield the complete correct image. A
|
|
/// partial JPEG 2000 decode is a complete, blurry, entirely plausible
|
|
/// picture — declared-versus-delivered in its purest form.
|
|
#[test]
|
|
fn a_truncated_codestream_does_not_return_the_whole_image() {
|
|
let full = corpus("gray8_lossless.j2k");
|
|
let cut = &full[..full.len() / 2];
|
|
match jpx::decode(cut) {
|
|
Err(_) => {}
|
|
Ok(img) => {
|
|
let got = interleave(&img);
|
|
let want = expected("gray8_lossless.j2k");
|
|
assert_ne!(
|
|
got, want,
|
|
"half a codestream produced the complete correct image, \
|
|
which means the second half is never read"
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Corrupting the entropy-coded data must not panic. This is the fuzzing
|
|
/// property stated as a test, on a real file rather than random bytes,
|
|
/// because a valid header with a corrupt body reaches far deeper into the
|
|
/// decoder than noise does.
|
|
#[test]
|
|
fn a_corrupt_codestream_never_panics() {
|
|
let full = corpus("gray8_lossless.j2k");
|
|
for cut in 0..full.len() {
|
|
let _ = jpx::decode(&full[..cut]);
|
|
}
|
|
for flip in 0..full.len() {
|
|
let mut broken = full.clone();
|
|
broken[flip] ^= 0xFF;
|
|
let _ = jpx::decode(&broken);
|
|
}
|
|
for flip in (0..full.len()).step_by(3) {
|
|
let mut broken = full.clone();
|
|
broken[flip] = 0xFF;
|
|
let _ = jpx::decode(&broken);
|
|
}
|
|
}
|