9d37874453
Mirror of the CAD engine harness for the doc crate, minus the shim gymnastics (doc-engine depends only on serde/serde_json, so it instruments directly): an isolated toolchain + cargo + target dir under one mktemp directory, removed by a shell trap on every exit path; nothing enters the host, the workspace target/, or $HOME. Runs the unit tests plus tests/materialize.rs under -C instrument-coverage, enforces a 96% total-lines floor against a 99.00% baseline plus per-file floors (losing one module's tests must not hide in the total), and with KEEP_COVERAGE=1 writes the uncovered-line listing that makes adding branch tests directed rather than guesswork. COVERAGE.md records the baseline, the exclusions, and the arms that are deliberately left uncovered (defensive CRDT merge arms, one unreachable!, and the Compensation::inverse arms unreachable through the public API).
214 lines
9.1 KiB
Bash
Executable file
214 lines
9.1 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Temporary LLVM source-coverage run for the standalone CRDT doc engine.
|
|
#
|
|
# WHAT THIS COVERS
|
|
# ----------------
|
|
# `doc-engine` is the pure engine the doc workspace builds on: the CRDT
|
|
# document/order core, the projection (styled runs, tables, nodes), the
|
|
# controller op surface, undo/redo history and the session wire format.
|
|
# It is dependency-free beyond serde + serde_json, so unlike nigig-build
|
|
# it can be instrumented DIRECTLY -- no host-only shim crate, no Makepad
|
|
# checkout, no native packages. The harness copies the crate (sources
|
|
# plus tests/materialize.rs, the real integration suite) into a scratch
|
|
# directory outside the workspace and runs it under
|
|
# -C instrument-coverage.
|
|
#
|
|
# WHAT IT EXCLUDES FROM THE REPORT (step 4 of the coverage plan)
|
|
# - the cargo registry / git dirs -- third-party code
|
|
# - the rustc sysroot -- std
|
|
# There is no Makepad generated code and no platform startup in this
|
|
# crate at all: it is a plain `lib` target, which is exactly why it can
|
|
# be measured without the CAD-style shim gymnastics.
|
|
#
|
|
# USAGE
|
|
# ./tools/test-doc-engine-coverage.sh # run, enforce floors, clean up
|
|
# KEEP_COVERAGE=1 ./tools/test-doc-engine-coverage.sh # keep env + uncovered lines
|
|
# DOC_ENGINE_COVERAGE_REPORT_ONLY=1 ./tools/test-doc-engine-coverage.sh # measure, don't gate
|
|
#
|
|
# Everything -- toolchain, cargo home, target dir, profraw data, the
|
|
# crate copy and the report -- lives under a single mktemp directory a
|
|
# shell trap removes on success, failure, interrupt or termination
|
|
# (step 7 of the coverage plan: nothing is installed into the host,
|
|
# nothing is written into the repository or left in $HOME).
|
|
set -Eeuo pipefail
|
|
IFS=$'\n\t'
|
|
|
|
ROOT="$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
|
|
# Floors, set a couple of points under the measured baseline so ordinary
|
|
# refactoring does not trip them while a real loss of coverage does.
|
|
# A single total hides the failure this is meant to catch: losing every
|
|
# test in one module moves the total by a point or two and a lone number
|
|
# would wave that through. Lowering a floor is a reviewable edit here,
|
|
# not something to do quietly.
|
|
#
|
|
# Baseline (2026-08-17, llvm-cov lines percent):
|
|
# controller 98.68 / crdt-document 97.63 / crdt-operations 100 /
|
|
# version_vector 100 / history 92.00 / projection-text 100 / TOTAL 98.99.
|
|
# lib.rs, session.rs, crdt/mod.rs and projection/mod.rs carry no
|
|
# executable lines (re-export and module files), so llvm-cov never
|
|
# reports them and they must NOT appear here: an unmeasured file with a
|
|
# floor fails the run by design.
|
|
#
|
|
# history.rs is the honest floor: Compensation::inverse arms unreachable
|
|
# through the public controller API (undo/redo decompose groups before
|
|
# materializing) sit in its remaining lines.
|
|
TOTAL_FLOOR="${DOC_ENGINE_COVERAGE_TOTAL_FLOOR:-96}"
|
|
PER_FILE_FLOORS="${DOC_ENGINE_COVERAGE_PER_FILE_FLOORS:-\
|
|
src/controller.rs:96
|
|
src/history.rs:89
|
|
src/crdt/document.rs:95
|
|
src/crdt/operations.rs:97
|
|
src/crdt/version_vector.rs:97
|
|
src/projection/text.rs:97}"
|
|
TOOLCHAIN="$(sed -n 's/^channel = "\(.*\)"/\1/p' "$ROOT/rust-toolchain.toml")"
|
|
HOST_TRIPLE="${DOC_ENGINE_COV_HOST:-x86_64-unknown-linux-gnu}"
|
|
|
|
# Same TMPDIR reasoning as tools/test-spreadsheet-coverage.sh: a mktemp
|
|
# default under $HOME/.cache -- roomy, outside the workspace, and removed
|
|
# on every exit path.
|
|
DEFAULT_TMP="${HOME:-/var/tmp}/.cache/nigig-coverage"
|
|
mkdir -p "${TMPDIR:-$DEFAULT_TMP}"
|
|
WORK="$(mktemp -d "${TMPDIR:-$DEFAULT_TMP}/doc-engine-coverage.XXXXXXXX")"
|
|
KEEP_COVERAGE="${KEEP_COVERAGE:-0}"
|
|
|
|
cleanup() {
|
|
local status=$?
|
|
if [[ "$KEEP_COVERAGE" == "1" ]]; then
|
|
echo "coverage environment retained: $WORK" >&2
|
|
else
|
|
rm -rf -- "$WORK"
|
|
rmdir "$DEFAULT_TMP" 2>/dev/null || true
|
|
rmdir "${HOME:-/var/tmp}/.cache" 2>/dev/null || true
|
|
echo "cleaned isolated coverage environment" >&2
|
|
fi
|
|
exit "$status"
|
|
}
|
|
trap cleanup EXIT HUP INT TERM
|
|
|
|
CRATE="$ROOT/crates/apps/doc/doc-engine"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Isolated toolchain with the coverage instrumentation components
|
|
# ---------------------------------------------------------------------------
|
|
export RUSTUP_HOME="$WORK/rustup"
|
|
export CARGO_HOME="$WORK/cargo"
|
|
export CARGO_TARGET_DIR="$WORK/target"
|
|
export PATH="$CARGO_HOME/bin:$PATH"
|
|
export LLVM_PROFILE_FILE="$WORK/profiles/%p-%m.profraw"
|
|
export RUSTFLAGS="-C instrument-coverage -C codegen-units=1 -C opt-level=0"
|
|
mkdir -p "$WORK/profiles"
|
|
|
|
curl --fail --location https://sh.rustup.rs -o "$WORK/rustup-init"
|
|
chmod 700 "$WORK/rustup-init"
|
|
"$WORK/rustup-init" -y --profile minimal --default-toolchain "$TOOLCHAIN" \
|
|
--component llvm-tools-preview --no-modify-path
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. The crate copy, outside the workspace (a copy inside the repository
|
|
# would be seen as a workspace member and refuse to build; there is no
|
|
# crate-local lockfile -- the workspace root owns the lock -- so plain
|
|
# `cargo test` re-resolves serde/serde_json, a two-crate registry hit).
|
|
# ---------------------------------------------------------------------------
|
|
COPY="$WORK/doc-engine"
|
|
mkdir -p "$COPY"
|
|
cp "$CRATE/Cargo.toml" "$COPY/"
|
|
cp -r "$CRATE/src" "$COPY/src"
|
|
cp -r "$CRATE/tests" "$COPY/tests"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Instrumented run: unit tests (in-file #[cfg(test)]) + tests/materialize.rs
|
|
# ---------------------------------------------------------------------------
|
|
cargo test --manifest-path "$COPY/Cargo.toml" --all-targets
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. Report
|
|
# ---------------------------------------------------------------------------
|
|
LLVM_BIN="$RUSTUP_HOME/toolchains/$TOOLCHAIN-$HOST_TRIPLE/lib/rustlib/$HOST_TRIPLE/bin"
|
|
"$LLVM_BIN/llvm-profdata" merge -sparse "$WORK/profiles"/*.profraw \
|
|
-o "$WORK/coverage.profdata"
|
|
|
|
mapfile -t BINS < <(find "$CARGO_TARGET_DIR/debug/deps" -maxdepth 1 -type f -executable \
|
|
\( -name 'doc_engine-*' -o -name 'materialize-*' \) ! -name '*.d')
|
|
[[ ${#BINS[@]} -gt 0 ]] || { echo "no instrumented test binaries found"; exit 1; }
|
|
OBJECTS=("${BINS[0]}")
|
|
for b in "${BINS[@]:1}"; do OBJECTS+=(-object "$b"); done
|
|
|
|
# The explicit source list does double duty: only the engine sources and
|
|
# the integration suite contribute to TOTAL, so the number cannot be
|
|
# diluted by anything that happens to sit under $WORK, and a file dropped
|
|
# from src/ shows up as a missing floor measurement below.
|
|
IGNORE="(/cargo/registry|/cargo/git|/rustc/)"
|
|
|
|
SOURCES=()
|
|
while IFS= read -r f; do SOURCES+=("$COPY/$f"); done < <(
|
|
cd "$CRATE" && find src tests -name '*.rs' | sort
|
|
)
|
|
|
|
"$LLVM_BIN/llvm-cov" report "${OBJECTS[@]}" \
|
|
-instr-profile="$WORK/coverage.profdata" -ignore-filename-regex="$IGNORE" \
|
|
"${SOURCES[@]}"
|
|
|
|
if [[ "$KEEP_COVERAGE" == "1" ]]; then
|
|
"$LLVM_BIN/llvm-cov" show "${OBJECTS[@]}" \
|
|
-instr-profile="$WORK/coverage.profdata" -ignore-filename-regex="$IGNORE" \
|
|
"${SOURCES[@]}" \
|
|
| grep -E '^ *[0-9]+\| *0\|' > "$WORK/uncovered-lines.txt" || true
|
|
echo "uncovered line report: $WORK/uncovered-lines.txt" >&2
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. Enforce the floors
|
|
# ---------------------------------------------------------------------------
|
|
"$LLVM_BIN/llvm-cov" export "${OBJECTS[@]}" \
|
|
-instr-profile="$WORK/coverage.profdata" -ignore-filename-regex="$IGNORE" \
|
|
"${SOURCES[@]}" > "$WORK/coverage.json"
|
|
|
|
if [[ "${DOC_ENGINE_COVERAGE_REPORT_ONLY:-0}" == "1" ]]; then
|
|
echo 'report-only mode: the floors were not enforced'
|
|
exit 0
|
|
fi
|
|
|
|
python3 - "$WORK/coverage.json" "$TOTAL_FLOOR" "$PER_FILE_FLOORS" <<'PY'
|
|
import json, sys
|
|
|
|
path, total_floor, per_file = sys.argv[1], float(sys.argv[2]), sys.argv[3]
|
|
with open(path) as fh:
|
|
data = json.load(fh)
|
|
export = data["data"][0]
|
|
total = export["totals"]["lines"]["percent"]
|
|
measured = {f["filename"]: f["summary"]["lines"]["percent"] for f in export["files"]}
|
|
|
|
failures = []
|
|
if total < total_floor:
|
|
failures.append(f" total {total:.2f}% is below the floor of {total_floor:.2f}%")
|
|
|
|
for line in per_file.split():
|
|
if not line.strip():
|
|
continue
|
|
name, _, floor = line.rpartition(":")
|
|
floor = float(floor)
|
|
hits = [v for k, v in measured.items() if k.endswith("/doc-engine/" + name)]
|
|
if not hits:
|
|
failures.append(
|
|
f" {name} has a floor but was not measured -- was it renamed, "
|
|
"deleted, or dropped from sources? A floor on a file that "
|
|
"is not measured silently protects nothing.")
|
|
continue
|
|
if hits[0] < floor:
|
|
failures.append(
|
|
f" {name} {hits[0]:.2f}% is below its floor of {floor:.2f}%")
|
|
|
|
if failures:
|
|
print("coverage floors not met:", file=sys.stderr)
|
|
print("\n".join(failures), file=sys.stderr)
|
|
print(
|
|
"\nEither the change removed tested code, or it added untested code.\n"
|
|
"Lowering a floor is a reviewable edit to tools/test-doc-engine-coverage.sh,\n"
|
|
"not something to do quietly.",
|
|
file=sys.stderr,
|
|
)
|
|
sys.exit(1)
|
|
|
|
print(f"all coverage floors met (total {total:.2f}%)")
|
|
PY
|