106 lines
9.2 KiB
Markdown
106 lines
9.2 KiB
Markdown
# rwbench: real-world workloads for Relax's code quality
|
|
|
|
The same Rust source runs under `rustc --release` and under `makepad run`. Every workload is a headless
|
|
program over the real library crates (real call depth, generics, cross-crate inlining) on fixed
|
|
in-repo inputs; no UI, GPU or window. The micro-kernels live in `../bench_rt`.
|
|
|
|
| workload | what runs |
|
|
|---|---|
|
|
| `text_shape_raster` | rustybuzz shaping, ttf-parser outlines, ab_glyph_rasterizer coverage (the sequence `draw/src/text` uses) over `README.md` + `AGENTS.md` |
|
|
| `inflate_real` | makepad-fast-inflate on a font, source text, markdown |
|
|
| `zip_read` | makepad-zip-file reader over an archive built in setup from repo files |
|
|
| `png_decode` | makepad-zune-png (the platform's decoder) on three repo PNGs |
|
|
| `brotli_decode` | vendored brotli decoder on streams made in setup from repo files |
|
|
| `maps_tile_decode` | MbtilesReader (SQLite b-tree, brotli codec) + MVT protobuf/geometry decode of a generated 196-tile z14 archive (the repo carries no tile set) |
|
|
| `layout_widgets` | a 120-row widget tree (View/Label/Button/Chip) laid out and drawn into a draw list by the real `Cx2d` turtle code, 12 pass widths; headless `Cx`, nothing submitted to a GPU |
|
|
| `rust_tokenize` | makepad-rust-tokenizer over a large Rust source file |
|
|
| `markdown_parse` | pulldown-cmark over two in-repo markdown documents |
|
|
| `html_parse` | makepad-html over `libs/piano_model/piano-explained.html` |
|
|
| `llm_host` | the LLM host path of one CUDA decode token with no GPU and no weight data (`makepad-ai-llm::host_bench`, restored in libs/ai/llm from makepad commit `17b9036971`): a scaled-down qwen35 GGUF (64 blocks, real 248,320 vocab) written in setup, mapped as a session maps it; per iteration one decode-graph build plus 256 tokens of layout validation, input writes, readback shaping of a 1 MB logits vector, collapse and greedy sampling. `RW_LLM_GGUF=<path>` runs a real GGUF. The CUDA graph planner (`plan_raw_graph`) is not covered: it compiles only with the CUDA kernels built |
|
|
| `piano_render` | makepad-piano-model: 0.3 s of a fixed pedalled chord phrase through the physical `Piano` (strings, soundboard, sympathetic bank, room) and through `LearnedPiano`; both built fresh per iteration (their `reset()` does not return to the constructed state) |
|
|
| `ogg_decode` | makepad-audio-decode `decode_any` on the one in-repo compressed audio file (`libs/audio_decode/testdata/button-press.ogg`), 8 decodes per iteration. The repo holds no MP3 (the decoder's MP3 tests build streams bit by bit inside the test) |
|
|
| `audio_mix` | makepad-audio-synth `SynthBus` (the game mixer): recipe voices placed around the listener, a tone, two engines with tyres moving past, room send and master dynamics, 40 blocks of 512 |
|
|
| `splash_eval` | makepad-script VM calling a compute-heavy Splash fn (recursion, loops, array and object access, arithmetic); parsed and evaluated once in setup; between iterations the host GC runs when the heap asks for it (`needs_gc`), as the event loop does per frame |
|
|
| `csg_boolean` | makepad-csg: primitives meshed at fixed resolution, intersection / difference / union / `union_all` |
|
|
| `box3d_physics` | makepad-box3d (single precision, serial solver, sleeping off): `src/box3d_scene.rs` built fresh per iteration (a hull-box pyramid, a revolute plank bridge with a motor and limits, a spherical-joint sphere rope ending in a capsule, a distance-joint spring pendulum, a rain of spheres, capsules and rocks), 90 steps of 1/60 s with 4 substeps; checksum = every body's transform and velocity bits plus the contact count at steps 30/60/90. The same scene at its small size is the diff_fast package program `relax-diffpkg box3d` (tools/run/fixture/diffpkg) |
|
|
|
|
### Commercial set
|
|
|
|
With the commercial checkout at `apps/commercial`, `run.py` also runs the workloads of `apps/commercial/relax_tests/src/bench.rs`
|
|
(package `relax-commercial-tests`, binary `relax-rwbench-commercial`, this crate's `src/harness.rs` included by path), after the
|
|
OSS ones and in the same rounds and tables: `sandbox_sim` (Sandbox's game sim over box3d f64: a crate stack and a capsule-mover
|
|
scene, 300 ticks each, plus a vehicle lap), `level_walk` (nav grid build and seeded walker tours), `model_build` (a props program
|
|
and a character preset built to GLB through the model-script registry and libs/model). `RW_COMMERCIAL=0` leaves them out;
|
|
`NATIVE_COMMERCIAL` names a prebuilt rustc binary of it. The same package's default binary is diff_fast's commercial package program set.
|
|
|
|
Not here: an Amp program (there is no `apps/commercial/amp` in the tree), whisper. Build with `MAKEPAD_GGML_NO_CUDA=1` (the submit scripts set it) so the llm crate's CUDA store does not run nvcc.
|
|
|
|
## Runtime port status
|
|
|
|
The fixed `layout_widgets` scene now constructs View/Label/Button/Chip from the compiled typed
|
|
styles. Its row count, text, ordering, measured rows and pass widths are unchanged.
|
|
The parser workloads use the Rust tokenizer, markdown parser and HTML parser directly.
|
|
The audio score workload still uses the score document composer.
|
|
|
|
### makepad relax `a3b0786428`
|
|
|
|
`render_kernels` already uses native Rust functions at this revision; its old compute compiler
|
|
is gone. `makepad-ai-llm::host_bench::HostDecodeBench` is still exported. The benchmark does not
|
|
replace either library with benchmark-local implementations.
|
|
|
|
`layout_widgets` uses `Flow::right()` for the current data-carrying `Right` variant: top row
|
|
alignment and no row wrapping, matching the former `Flow::Right`. It retains all 120 rows,
|
|
the same two labels/button/chip per row, text, child order, row alignment, twelve widths and
|
|
rectangle checksum. No per-iteration work is removed. The historical table below predates
|
|
the typed widget migration and is not a baseline for this implementation; compare compiler
|
|
builds on the same makepad revision and fixed input tree.
|
|
|
|
Process failures or checksum mismatches make the runner exit nonzero.
|
|
|
|
## Output
|
|
|
|
`relax-rwbench <workload>...` prints per workload
|
|
`| name steady_ms=.. tts_main_ms=.. tts_proc_ms=.. iters=.. first_ms=.. <checksum>`.
|
|
`steady_ms` is the median of the last iterations that start after 5 s (`RW_STEADY_AFTER_MS`; Relax installs its optimised code about 2 s in); `tts_*` is the time until the
|
|
per-iteration time is within 10% of steady (3-iteration running median): `tts_main` from entering
|
|
`main`, `tts_proc` from process start (`RW_WALL0=<epoch ns>`, set by `run.py`, so it includes Relax's
|
|
compile). Setup (reading inputs, building archives) counts towards time-to-steady. Env: `RW_MS`
|
|
(iteration budget per process, default 8000), `RW_MIN_ITERS` (20), `RW_ROOT` (tree holding the inputs).
|
|
|
|
## Run
|
|
|
|
```
|
|
python3 run.py <target dir> <name=makepad binary>...
|
|
```
|
|
builds the rustc side (or uses `NATIVE=<binary>`; the commercial set `NATIVE_COMMERCIAL`) and runs every workload (`WORKLOADS="a b"` picks some) under rustc and
|
|
under each Relax binary, `ROUNDS` (default 4) interleaved rounds: each round runs every workload once, rustc first, then
|
|
the binaries in alternating order (base, new / new, base / ...). Per name it prints, per workload, the median over rounds
|
|
of the steady ms with the [min..max] spread, the Relax/rustc ratio (median of the per-round ratios, so drift hits both
|
|
sides) with its spread, time-to-steady, O3 coverage and whether all checksums agree. With several names a final A/B
|
|
table gives new/base per workload (median of per-round ratios, spread) and the geomean. The first name is the base; the
|
|
same binary under two names measures the noise floor. `RW_MS` (4000) is the per-process iteration budget, `MAKEPAD_RUN_STD`
|
|
defaults to `../std`, `RUN_PREFIX` defaults to `nice -n 10` (`none` on Windows). Timing belongs on the quiet machines
|
|
through the bench queue:
|
|
|
|
* x64 (`win` box): `local/agent_state/fast-rust/codegen2/rwtime_submit.sh <tag> <name=fastrust-commercial>...`
|
|
(names sharing a key share one Relax build; `BENCH_KEY=<key>` takes the rwbench tree from another commit)
|
|
* arm64 (mini): `.../codegen2/rwtime_a64_submit.sh <name=fastrust-commercial>...`
|
|
|
|
Both pack `src_tar.py` trees plus the commercial `engine/` and `stage/libs/` dirs, build Relax and rwbench on the box and
|
|
print these tables. The crate resolves offline from `vendor/` (its own `[patch.crates-io]` mirrors the root manifest's).
|
|
|
|
## Known results
|
|
|
|
Node 165 (x86-64 Linux, shared, NOT a quiet machine; frozen `relax-t19`):
|
|
|
|
| workload | rustc steady ms | Relax steady ms | Relax/rustc | rustc time-to-steady ms | Relax time-to-steady ms | Relax first iter ms | Relax O3 coverage | checksums equal |
|
|
|---|---:|---:|---:|---:|---:|---:|---|---|
|
|
| text_shape_raster | 8.035 | 9.623 | 1.20 | 36 | 2750 | 278.7 | O3: 300 hot, 300 installed | yes |
|
|
| inflate_real | 4.728 | 7.115 | 1.50 | 79 | 1426 | 27.3 | O3: 26 hot, 26 installed | yes |
|
|
| zip_read | 5.074 | 7.378 | 1.45 | 102 | 1431 | 26.5 | O3: 62 hot, 62 installed | yes |
|
|
| png_decode | 11.996 | 64.425 | 5.37 | 5 | 1432 | 129.4 | O3: 57 hot, 57 installed | yes |
|
|
| brotli_decode | 7.097 | 11.865 | 1.67 | 482 | 2526 | 74.9 | O3: 113 hot, 114 installed | yes |
|
|
| maps_tile_decode | 19.523 | 19.173 | 0.98 | 1650 | 1846 | 141.4 | O3: 301 hot, 302 installed | yes |
|
|
| layout_widgets | 29.792 | 85.201 | 2.86 | 40 | 4303 | 868.3 | O3: 1104 hot, 116 installed | yes |
|
|
|
|
Quiet-machine results (x64 win box, arm64 mini) are posted by the bench queue under local/agent_state/fast-rust/bench/results/.
|