andodeki
9989043a37
Some checks failed
repo hygiene / hygiene (push) Has been cancelled
nigig-build (CAD) / supply-chain (push) Has been cancelled
nigig-build (CAD) / cad-module (push) Has been cancelled
nigig-build (CAD) / full-crate-check (push) Has been cancelled
nigig-build (CAD) / cad-engine-coverage (push) Has been cancelled
nigig-build (CAD) / doc-workspace-coverage (push) Has been cancelled
nigig-build (CAD) / cad-widget-coverage (push) Has been cancelled
spreadsheet / engine-coverage (push) Has been cancelled
spreadsheet / ui-controller-coverage (push) Has been cancelled
Phase 0 of REVIEWS/REPO_COVERAGE_100_PLAN.md, and the reason it is Phase 0: no new tests, no new measurement, just ratchets on numbers that were already good and already decaying-capable. **spreadsheet** — `tools/test-spreadsheet-coverage.sh` has had a 96 floor for the engine and another for the UI controllers, and no CI job has ever run it. New `.forgejo/workflows/spreadsheet.yml`, two jobs: engine-coverage 98.83% of lines (floor 96) ui-controller-coverage 98.85% of lines (floor 96) Split in two because the halves cost very differently. The engine is pure Rust and finishes in about three minutes; the UI half has to build Makepad's Linux backend to link a test binary. One job would hide an engine regression behind a ten-minute build. **CAD widget layer** — `cad-widget-coverage` in nigig-build.yml, deliberately REPORT-ONLY. It sits at 13.25% of 10,637 lines with six files at exactly zero, and a floor there would read as a blessing rather than a debt. What the job buys is that the number is printed on every push instead of being rediscovered in six months. The first real input test should set a floor behind it. Also corrects the plan. It claimed the doc workspace module was ungated; it is not — nigig-build.yml has run doc-workspace-coverage since before the plan was written. I had surveyed by grepping workflow files for the word "coverage" and attributed nigig-build's coverage jobs to CAD alone. I nearly committed a duplicate workflow on the strength of it. The census table was right; the prose under it was not, and the correction is in the file. One thing checked and deliberately NOT changed: the spreadsheet script appears to skip its UI half when the native packages are absent. It does not. `makepad-native-libs.sh --check` returns 1, the script runs under `set -e`, and it aborts. What misled me was reading `$?` after piping the script into `tail` — which reports tail's status, not the script's. The same class of mistake this repository's CI comments warn about; no fix was needed and none was made. Verified by running each job's exact command line: COVERAGE_TARGET=engine ./tools/test-spreadsheet-coverage.sh rc=0 COVERAGE_TARGET=ui ./tools/test-spreadsheet-coverage.sh floors met ./tools/test-cad-widget-coverage.sh 13.25%, rc=0
238 lines
10 KiB
Markdown
238 lines
10 KiB
Markdown
# Repository test coverage: a phased plan to 100%
|
||
|
||
Scope: all 58 workspace members, 310,287 lines of source. A CAD-module
|
||
plan already exists at `REVIEWS/CAD_COVERAGE_100_PLAN.md`; this is the
|
||
level above it, and where the two disagree the CAD document is more
|
||
specific and wins.
|
||
|
||
Every number here was produced by a command in this repository. None is
|
||
estimated unless it says so.
|
||
|
||
## The census
|
||
|
||
| Tier | Crates | Source lines | State |
|
||
|---|---|---|---|
|
||
| **A** | 5 | 45,190 | measured **and** gated in CI |
|
||
| **B** | 2 | 15,373 | measured, **not** gated |
|
||
| **C** | 27 | 225,149 | has tests, **never measured** |
|
||
| **D** | 5 | 4,160 | **zero tests** |
|
||
| **E** | 19 | 20,415 | templated app shells, zero tests |
|
||
| | **58** | **310,287** | 5,155 tests, 27,419 lines of test code |
|
||
|
||
Tier C is 73% of the repository and the whole problem. Everything else
|
||
is either already handled or arguably not worth handling.
|
||
|
||
### Tier A — measured and gated
|
||
|
||
| Area | Floor | Tool |
|
||
|---|---|---|
|
||
| CAD engine (`nigig-build` cad/, 15 files) | 96 total, per-file | `test-cad-coverage.sh` |
|
||
| `doc-engine` | 96 | `test-doc-engine-coverage.sh` |
|
||
| email domain (7 `nigig-core` files) | 88 | `test-email-coverage.sh` |
|
||
| pdf (4 crates) | 80 | `test-pdf-coverage.sh` |
|
||
|
||
### Tier B — measured, nobody enforces it
|
||
|
||
`spreadsheet-engine` + `spreadsheet-ui` have a coverage script with
|
||
per-file floors and **no CI job that runs it**. The CAD widget layer
|
||
likewise: 13.25%, measured, ungated. This is the cheapest gap in the
|
||
document.
|
||
|
||
*Correction, made while executing Phase 0:* an earlier draft of this
|
||
section also listed the doc workspace module here. It is not ungated —
|
||
`nigig-build.yml` has run `doc-workspace-coverage` since before this
|
||
plan was written. The census table above was right; this paragraph was
|
||
wrong, because the survey that produced it matched workflow files
|
||
containing the word "coverage" and attributed nigig-build's coverage
|
||
jobs to CAD alone.
|
||
|
||
### Tier C — the elephant, biggest first
|
||
|
||
| Crate | Lines | Tests | Measured? |
|
||
|---|---|---|---|
|
||
| `nigig-build` | 77,758 | 1,485 | only cad/ (~24k of it) |
|
||
| `map` | 44,518 | 887 | no |
|
||
| `pageflipnav` | 24,117 | **20** | no |
|
||
| `nigig-core` | 18,879 | 327 | only the 7 email files |
|
||
| `nigig-sms` | 7,127 | 64 | no |
|
||
| `nigig-pay-domain` | 6,321 | 161 | CI runs tests, no coverage |
|
||
| `nigig-pay` | 6,107 | 72 | no |
|
||
| `nigig-pay-ui` | 5,724 | 84 | no |
|
||
| `nigig-mpesa` | 5,198 | 45 | no |
|
||
| `robius-sms` | 4,045 | 58 | no |
|
||
| `nigig-pay-platform` | 2,567 | 64 | no |
|
||
| `nigig-pay-storage` | 2,405 | 46 | no |
|
||
|
||
`pageflipnav` is the standout: 24,117 lines, 20 tests, no gate, and
|
||
nothing in this repository's review documents mentions it.
|
||
|
||
### Tier E — nineteen copies of one file
|
||
|
||
`nigig-shop`, `nigig-tow`, `nigig-rider`, `nigig-book`, `nigig-health`
|
||
and fourteen more are 992–1,033 lines each and are templated clones:
|
||
diffing two `main.rs` files after normalising the name yields a
|
||
background colour and a root screen identifier. Treating these as
|
||
nineteen crates to cover is nineteen times the work for one crate's
|
||
worth of risk. See Phase 5.
|
||
|
||
## What 100% would actually cost
|
||
|
||
The only honest estimate available is the measured one. The CAD engine
|
||
is 13,752 lines and reaching 97.16% took roughly six concentrated
|
||
sessions, producing 568 tests and finding four real defects. That is
|
||
~2,300 lines brought under coverage per session, on code that was
|
||
already the *easiest* half — pure, dependency-light, and already
|
||
partially tested.
|
||
|
||
Tier C is 225,149 lines. Straight-line extrapolation is ~98 sessions,
|
||
and the extrapolation is optimistic because tier C contains the GUI
|
||
layers, which are strictly harder: the CAD widget half sits at 13.25%
|
||
after all that work.
|
||
|
||
That number is not an argument against the goal. It is an argument
|
||
about **sequence**: at this scale, the order in which coverage arrives
|
||
matters more than the destination, because the first 20% of the effort
|
||
can cover most of the risk if it is pointed at the right code.
|
||
|
||
## The principle this plan is sequenced on
|
||
|
||
Coverage is a search strategy, not a product. Of the defects this work
|
||
has found so far — a matrix inverse wrong for every rotate-plus-translate,
|
||
three exporters that only ever dispatched one geometry variant, a nav-pad
|
||
hit zone a pixel from its button, two benchmarks measuring a function
|
||
that cannot cache — **not one was found by driving a percentage up**.
|
||
All four came from reading uncovered regions and asking why they were
|
||
unreachable.
|
||
|
||
So: measure everything early because measurement is cheap and tells you
|
||
where to look; close gaps in risk order, not size order; and gate each
|
||
number the day it is good, because an ungated number decays.
|
||
|
||
## Phase 0 — Gate what is already measured (2–3 days)
|
||
|
||
Three coverage scripts exist with floors and no CI job. Add them.
|
||
|
||
1. `spreadsheet` (15,373 lines, floors already written) → CI job.
|
||
**Done:** `.forgejo/workflows/spreadsheet.yml`, two jobs, measured
|
||
98.83% engine / 98.85% ui-controllers against floors of 96.
|
||
2. ~~doc workspace module~~ — already gated by `nigig-build.yml`; see
|
||
the correction in the tier B section.
|
||
3. CAD widget layer → CI job, **report-only**, then floors at
|
||
measured-minus-one once a real input test lands. It is at 13.25%; a
|
||
floor there today records the debt rather than blessing it.
|
||
**Done:** `cad-widget-coverage` in `nigig-build.yml`.
|
||
|
||
**Exit:** every existing coverage script runs in CI. No new tests
|
||
written. This is pure ratchet installation.
|
||
|
||
## Phase 1 — Measure tier C (1–2 weeks)
|
||
|
||
You cannot plan 225,149 lines without knowing where they stand. Today
|
||
there are seven bespoke coverage scripts, each hand-rolled for one area.
|
||
Fifty-eight crates cannot have fifty-eight scripts.
|
||
|
||
1. Generalise the harness into one parameterised runner: crate list,
|
||
file list, floors, exclusions. The existing scripts become config.
|
||
`test-cad-coverage.sh` is the most evolved and is the right base —
|
||
it already handles isolated toolchains, reuse knobs, floors and
|
||
drift detection.
|
||
2. Run it over every tier C crate. Record a number per crate in
|
||
`TEST_BASELINE.md`.
|
||
3. Re-sort the phases below by what it finds. **Phases 2–4 are
|
||
provisional until this lands.**
|
||
|
||
**Exit:** one runner, a coverage number for all 58 members, and a table
|
||
that makes the next argument evidence-based instead of size-based.
|
||
|
||
## Phase 2 — Money and safety first (4–6 weeks)
|
||
|
||
Regardless of what Phase 1 measures, this goes first. `nigig-pay`,
|
||
`nigig-pay-domain`, `-storage`, `-platform`, `nigig-mpesa`,
|
||
`nigig-sms`, `robius-sms`: 34,000 lines that move money and send
|
||
messages on a user's behalf. `REVIEWS/NIGIG_PAY_CONSOLIDATED_REVIEW.md`
|
||
and `PAYMENT_RISK_REGISTER.md` already describe unresolved hazards here
|
||
— thread-local coordinator, plaintext pending store, SMS-as-evidence.
|
||
|
||
Target 100% on the pure domain crates (`pay-domain`, `pay-storage`,
|
||
`pay-platform` — 11,293 lines, all host-testable), and ≥90% on the
|
||
transport and UI crates.
|
||
|
||
**Exit:** the four payment crates gated at 95+, with `workflow.md`'s
|
||
nine non-negotiable payment rules each pinned by a named test.
|
||
|
||
## Phase 3 — The large engines (3–6 months)
|
||
|
||
`map` (44,518), `nigig-build` outside cad/ (~53,000), `nigig-core`
|
||
(18,879), `pageflipnav` (24,117).
|
||
|
||
Use the pattern that worked on CAD, in this order per crate:
|
||
|
||
1. Split pure logic from widget code and measure the pure half in a
|
||
host-only harness. CAD's engine/widget split is the template.
|
||
2. Extract decision logic out of event handlers into pure modules —
|
||
~60 lines per extraction, half a day each, and on the one data point
|
||
available it found a real defect on the first try.
|
||
3. Accept that the imperative widget glue stays uncovered until Phase 4.
|
||
|
||
`pageflipnav` first despite not being the largest: 20 tests over 24,117
|
||
lines is the worst ratio in the repository and nothing has ever looked
|
||
at it.
|
||
|
||
## Phase 4 — The GUI layers (2–3 months, gated on a spike)
|
||
|
||
`REVIEWS/CAD_COVERAGE_100_PLAN.md` Phase 0 is the spike for this whole
|
||
phase: 148 `#[makepad_test]` tests exist in `nigig-build/tests/ui.rs`,
|
||
have never compiled, and run under `xvfb-run` once a one-line
|
||
dev-dependency is added. `pdf.yml` already runs its UI suite under
|
||
xvfb, so the mechanism is proven in this repository.
|
||
|
||
If that spike succeeds, the same approach applies to `map`,
|
||
`spreadsheet-ui`, `pay-ui` and the shells. If it does not, GUI coverage
|
||
stays out of reach and this plan tops out around 60% repo-wide — which
|
||
is worth knowing before committing three months.
|
||
|
||
**Do not start Phase 4 before the CAD spike reports.**
|
||
|
||
## Phase 5 — Decide what the templated shells are (1 week)
|
||
|
||
Nineteen crates, 20,415 lines, one program. Options, in preference
|
||
order:
|
||
|
||
1. **Extract the template.** One shared crate with the app scaffold, 19
|
||
thin crates that configure it. Covering the shared crate once covers
|
||
all nineteen. This is a refactor, not a test task, and it is the only
|
||
option that makes the number honest.
|
||
2. Cover one representative and exclude the rest by policy, documented.
|
||
3. Cover all nineteen. Nineteen times the work for one crate of risk.
|
||
|
||
**Exit:** a written decision. Not covering them silently is the only
|
||
outcome that is wrong.
|
||
|
||
## Phase 6 — Close the last mile and enforce (ongoing)
|
||
|
||
- Structural exclusions, because `#[coverage(off)]` is **unstable on the
|
||
pinned 1.97.1 toolchain** — verified, `error[E0658]`. Genuinely
|
||
unreachable code must be covered, moved to a file the report excludes,
|
||
or subtracted openly in the report. There is no annotation.
|
||
- Raise every floor to measured-minus-one, quarterly.
|
||
- Report a repo-wide adjusted total, with the subtracted set listed by
|
||
file and line count.
|
||
|
||
## What this plan will not deliver, and why
|
||
|
||
- **A literal 100% of 310,287 lines.** GL draw calls, native save
|
||
dialogs, platform startup and `panic!` arms inside tests are either
|
||
unreachable in a test process or assert nothing when reached. The
|
||
honest target is 100% of what is worth executing, with the remainder
|
||
named. In CAD that remainder is ~48 lines out of 13,752; repo-wide it
|
||
will be larger.
|
||
- **Assurance.** Coverage counts lines executed, not assertions made.
|
||
The 148 CAD UI tests are mostly `wait_visible()` and will move the
|
||
number a long way while proving that widgets exist. Nothing in this
|
||
plan prevents that; only review does.
|
||
|
||
## Recommended first action
|
||
|
||
Phase 0. It is two or three days, writes no tests, and stops the 15,373
|
||
already-measured lines in tier B from decaying while the rest of this is
|
||
argued about.
|