Compare commits

...

44 commits

Author SHA1 Message Date
ca7bc8439b nigig: carry workspace.dependencies into android wrapper manifest
The generated android wrapper re-creates a standalone workspace and only
forwarded [patch.*] sections from the workspace root manifest, so deps
declared via [workspace.dependencies] + workspace = true failed to inherit
in wrapped crate builds. Extract the [workspace.dependencies] section the
same way patches are handled and inject it into the wrapper manifest.
2026-09-02 10:14:11 +03:00
282845b7e0 nigig: NIGIG test-mode forwarding, custom manifest hook, ortho camera support
- makepad_test/runtime.rs: forward NIGIG_TEST_MODE from host env to the
  Android app via 'am start' intent extra; add wait_timeout (60s) used by
  wait_visible/wait_hidden/wait_count; make query_widgets tolerant of
  snapshot timeouts; grant READ_CONTACTS during adb setup
- makepad-platform android_jni.rs: read makepad.NIGIG_TEST_MODE intent
  extra and surface it as the NIGIG_TEST_MODE env var via apply_studio_env
- cargo_makepad compile.rs: support verbatim custom AndroidManifest.xml in
  addition to the templated variant
- makepad-xr xr_root.rs: add ortho camera controls (ortho, ortho_height,
  min/max), derive Debug on XrCamera
- docs: ANDROID.md and DESKTOP_VISIBLE.md for makepad_test
2026-09-02 10:14:10 +03:00
318b480c0e makepad_test: grant runtime permissions after install to prevent dialog overlay
The GrantPermissionsActivity pops up during navigation and blocks the
app's event loop, preventing hub responses. Pre-grant all runtime
permissions after APK install to avoid this.
2026-09-02 10:14:10 +03:00
1f6e881880 makepad_test: force-stop interfering Robrix app during tests
PID 28203 (rs.robius.robrix) was the actual zombie reclaiming foreground
and killing our test app - not our own package. Force-stop both the
target package and known interfering Makepad apps (Robrix) during test
setup to prevent cross-app foreground competition.

Also remove the pm disable-user approach as it doesn't help against
a different package's zombie process.
2026-09-02 10:14:09 +03:00
26d831c891 makepad_test: use pm disable-user to prevent Samsung zombie resurrection
Samsung devices keep killed app processes alive and bring them back to
the foreground ~15s later, killing our fresh test instance. force-stop
and kill -9 don't prevent this. pm disable-user fully prevents the
zombie from being restarted. Re-enable before launching the new instance.
2026-09-02 10:14:09 +03:00
c17a21830e android: send BeforeStartup/AfterStartup over websocket
The Android platform never sent BeforeStartup or AfterStartup messages
via the studio websocket. Desktop platforms send these through their
stdin event loops, but Android uses websockets instead of stdin.

Without AfterStartup, the hub never broadcasts AppStarted to UI
clients, causing makepad-test to time out waiting for app startup.
2026-09-02 10:14:09 +03:00
debf970c36 feat(test): extend protocol for touch, long-press, paste, IME composition
Add RemoteTouchState, RemoteTouchPoint, RemoteTouchUpdate, RemoteLongPress,
RemoteTextPaste, RemoteIMEComposition wire structs to StudioToApp enum.

Dispatch new events through cx_shared.rs (TouchUpdate→Event::TouchUpdate,
LongPress→MouseUp+MouseDown, TextPaste/IMEComposition→Event::TextInput).

TestApp: touch_down/move/up, long_press, paste_text, ime_composition.
Locator: touch_down/move/up, long_press, paste, ime_composition.
2026-09-02 10:14:09 +03:00
757849940c feat(widgets): reexport optional Makepad sibling crates
Adds feature-gated optional deps and re-exports (makepad-test, makepad-csg,
makepad-gltf, makepad-mbtile-reader, makepad-fast-inflate) so a downstream
workspace can depend on makepad-widgets as its sole Makepad source.
2026-09-02 10:14:09 +03:00
9e7afd661e feat(test): Android makepad_test via adb + in-process hub (legacy Java path)
Adds the Android test runtime to makepad_test: builds the APK with
cargo-makepad's standard Java path, installs and launches via adb with
makepad.STUDIO_* intent extras (incl. STUDIO_BUILD), connects the app to an
in-process hub over adb reverse, and waits for startup + responsiveness.
Adds clean in-process hub shutdown (HttpServerHandle + GatewayHandle Drop)
and the STUDIO_BUILD intent parsing on the app side. No native-activity or
NDK APK compilation code is included.
2026-09-02 10:14:09 +03:00
Kevin Boos
79f938c09d
Splash isolate host: keep inserted subtrees reachable, survive foreign-heap values, contain isolate panics (#1208)
* widget_tree: keep manually inserted children linked across a refresh

A container that owns a child outside its own child vec inserts it with
`insert_child_deep` and never reports it from `children()`, so a refresh
unlinked it from every top-down search and the removal pass then deleted
its subtree. Everything under it went quiet, and invisibly: an empty
`WidgetRef` is a silent no-op.

Mark such children `manual` and keep the live ones linked, clearing the
flag once the parent reports the child itself.

* script: survive a foreign-heap value in the GC mark walk

`GenVec`'s `Index` bounds-checks the raw Vec before the generation check,
and `len` never shrinks, so an index past the end belongs to a different
heap. Aborting the process over one takes the whole app down for a fault
confined to a single script heap.

Add bounds-checked accessors and use them in `mark_value_fields!`, so
marking skips and reports a foreign value; name the table where an index
does still panic.

* splash: contain isolate panics, and report the silent failures

A Splash isolate runs user script on the UI thread, and the VM swap in
`with_isolate_installed` was not panic-safe: a panic could leave the app
VM swapped out, with later script resolving against the wrong heap.
Restore through the unwind, and contain panics at each entry point (pump
arms, timers, host callbacks, isolate GC, body eval, hook calls).

Report what used to fail silently too: callback errors, a script->widget
call whose target is gone, and wrong-VM routing. Adds three headless
regression tests.
2026-09-02 02:26:52 +02:00
Admin
915ce7c4e2 asset-server: the embed module — fully-local mode becomes shared
The VJ's attach-or-host logic (health probe, beacon listen, the main
store's server.lock as proof-of-life; loopback-only Host with the library
publisher, rooted in the main library when one exists) moves into
makepad-app-asset-server as embed::resolve, steered by <PREFIX>_ASSET_
EMBED/ROOT/PORT. vj/local_store.rs becomes the thin VJ-flavoured wrapper.
First second consumer: the sandbox.
2026-09-01 18:01:21 +02:00
Admin
0131ba49ed mpwm: --test-action launch-<app> — put one registered app on the desk, deterministically
the existing startup test hook learns the launcher's verb, which is how a
hidden instance hosts a real child for framebuffer verification.
2026-09-01 17:43:47 +02:00
Admin
df265f7a55 vj: a short console draws smaller — height joins the scale chain
console_scale defended the width and rearranged the very short, but
between the fold thresholds (~820 points) and the height the console
actually needs (~1000) lay a dead zone where the lower region simply
clipped — an mpwm tile at 844 points lost its rail from MAP down and half
a grid row. TARGET_HEIGHT_POINTS closes it: console_dpi now takes the
window's physical height and a short console shrinks exactly as a narrow
one does, to the same MIN_SCALE floor, before the folds take over —
measured in the shrunk points. A wide, short window still prefers the
lists-beside arrangement (judged at native scale, so the chain stays a
pure function of the physical window). The whole rule chain threads the
height; tests cover the tile shape that exposed this and the new ladder.
2026-09-01 17:43:47 +02:00
Admin
d5a6d64d1b vj: LIGHTS and ARCHIVE are rail chips under MAP — any content chip is the way back to the grid
The tiny icon page-tabs are gone. The special pages join the lane rail as
first-class chips: when one is up no content lane reads selected, and
clicking VIDEO (or any lane) flips the lower region back to the grid — the
persisted ARCHIVE page can no longer read as an empty library.
2026-09-01 17:21:35 +02:00
Admin
1d97d63e2f rustc 1.98 sweep: script module glob imports nothing uses, and strlen declared as libc has it
1.98's sharper macro-use tracking flags the live-id macro globs five script
modules kept without using; windows-strings' strlen extern takes *const
c_char with a cast at the two call sites instead of tripping
suspicious_runtime_symbol_definitions (PCSTR is transparent over *const u8,
so the ABI never changed).
2026-09-01 17:12:15 +02:00
Admin
e4a2e75d66 ai-hub: backend imports and helpers follow their feature gates
the vision and ocr backends' ContentBackend machinery lives behind the
llm feature; the imports and the lane-refill helpers now do too, so a
client crate compiling the hub with a leaner feature set (vj, chat-ui)
builds without warnings. pipeline_submit_collect stays feature-free for
its tests and says so.
2026-09-01 17:09:26 +02:00
Admin
89fe23fb68 vj: the self-hosted store is the full host — the library publisher rides along, loopback only, no beacon
Attach still wins whenever discovery finds a server (health probe, UDP
beacon, main-store lock). When nothing answers, the VJ now boots the
standalone asset-server's Host instead of a bare AssetServer: catalog +
CAS plus the ai-content library publisher over local/ai_content_library,
so a self-hosted VJ sees the same rows asset-ui publishes — not an empty
seed store. The fallback root is the user's main library when one holds a
catalog (server.lock arbitrates the race; losing it means attach). Both
planes bind 127.0.0.1 and no beacon runs: a private host is not a thing
the network can discover. HostConfig grows a blob_refs knob so the VJ
keeps its reference-import policy; the store decision is logged at boot.
2026-09-01 17:09:26 +02:00
Admin
e6c84d9b6a zune: the crate-root macro_use attrs go — rustc 1.98 phases the inner form out
zune-core's was load-bearing (alloc macros, no_std) and becomes the outer
form on its extern crate; bmp, jpeg and qoi never used it — deleted.
2026-09-01 17:08:31 +02:00
Admin
46cd1735d0 chore: workspace members and remaining sources
Squashed from work:
- frametween: the VJ's in-betweener becomes a library — the whole mode set, one definition
- rtsmap: one seeded generator for tiled strategy maps
- workspace: register the score crates, archive_org and mp4_index
- map: bake a runnable Amsterdam test map from inside the route app
- platform: file dialogs on Android and iOS, and unbreak the Android build
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
2026-09-01 16:46:35 +02:00
Admin
0257d6c6dc engine libs: render and sim for a strategy round, the per-unit decision hook, the mp4 sample index, stitch and xatlas quieted, example refresh
Squashed from work:
- docs: button shader annotations — widgets/button.rs complete + splash demo buttons
- tweaker: the Shader tab shows the pinned widget's ANIMATOR STATES as little posed swatches under the well — one per tr
- widgets: the glass stops reading draw_pass.time, so glass apps idle again
- render, sim, platform: what a strategy round needs underneath
- draw, platform: overlays now composite above content that uses draw_depth
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- platform: native file and save dialogs, in-house on all three desktops
- sim: a per-unit decision hook — one commandable unit can think for itself, before the kit steers
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
2026-09-01 16:46:35 +02:00
Admin
87474bd68d finance: a personal ledger, budgets and reports on the DataGrid
Squashed from work:
- finance: a personal ledger, budgets and reports on the DataGrid
- zero-warning sweep, round two — the first full-workspace pass
2026-09-01 16:46:35 +02:00
Admin
79ccb10e81 mp*: the treemap camera era — canon-space packing, 2.5D and orbit views, streaming walks, the qwen sidepanel; mpwm linux desktop integration; pdf_parse recovery
Squashed from work:
- pdf_parse: indirect /Length, and recover the glyph names a symbol font uses
- mpwm: make the Linux desktop integration usable (#1204)
- mpfiles: the size map goes all the way down, and remembers what it found
- mpfiles: the treemap walk streams every directory at every depth — no more opaque growing blocks over deep heavy subtr
- mpfiles: local qwen chat sidepanel over the treemap selection
- mpfiles: a camera over the treemap — scroll zooms at the cursor with real re-layout detail, drag pans, double-click fi
- mpfiles: a 2.5D treemap — the map extrudes by nesting depth under a 3d toolbar mode
- mpfiles: a real orbit camera over the block view — left-drag orbits, right-drag pans, click stays a click; the project
- mpfiles: the wheel glides, the filter stops paying per frame and per keystroke — measured weights cached against the t
- mpfiles: the 3d view wears no name tags — labels only on hover, the tooltip carries the numbers
- mpfiles: a camera gesture never re-flows the map — mid-motion the picture rides one rigid remap of the last layout, an
- mpfiles: a small zoom keeps the map still — the layout is only remade when the camera truly leaves it (a 1.3x detail b
- mpfiles: a camera reveal is not an arrival — cells the cull swings into view appear instantly at full strength, cells
- mpfiles: the treemap packs in canon space — the arrangement is zoom-invariant by construction; zoom only refines the t
- mpfiles: the map stops breathing — insets are fractions of map space, names reserve no room and float on scrims with a
- mpfiles: a camera settle is calm — zoom detail materialises in place, no arrival fade; the ceremony stays with filter
- mpfiles: camera motion animates nothing but the camera — flat labels freeze per relayout and ride the remap; a calm se
- mpfiles: a fast zoom-out never outruns the map — cull escape relayouts the same frame, the glide's destination is laid
- mpfiles: wheel zoom in the raised modes anchors on the tile top under the cursor, not the ground hiding behind it
- mpfiles: first consumer of the ai-hub — the ask panel rides AiHub::start_local_chat, its private engine is gone (aicor
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples

Co-authored-by: Alex <blackanger.z@gmail.com>
2026-09-01 16:46:34 +02:00
Admin
517308e676 vj: deck sync as a held lock, archive.org as a content source, frametween becomes a library, runs execute in the app
Squashed from work:
- vj decks: sync is a held lock — pinned master, per-pump rate servo
- vj: the deck explorer opens on music
- frametween: the VJ's in-betweener becomes a library — the whole mode set, one definition
- frametween: the gate says which tier moved the picture, from pixels
- frametween: BGRA words go in as they are, and a host may own the clock
- vj: archive.org as a content source
- platform: native file and save dialogs, in-house on all three desktops
- vj: rounder slider caps, and the cap body reads as a blob
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- vj: plain generations execute in the app too — the store's job queue loses its last vj client (aicore §9 / F2)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- importer + asset-server host: the coordination era ends (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades
- zero-warning sweep, round seven — the last two
2026-09-01 16:46:34 +02:00
Admin
c460905e2a maps + route: the self-baking Amsterdam test map, reversible mkmap weave, the rtsmap generator, road faces, guarded first-run
Squashed from work:
- rtsmap: one seeded generator for tiled strategy maps
- map: bake a runnable Amsterdam test map from inside the route app
- route: the first-run test map starts itself, and its buttons are guarded
- map: the test map bakes its road faces too, from the same shared pass
- map_tiles: mkmap-extract — the weave is reversible, world-cells reconstructible from world.mkmap
- route: the local dispatcher rides the hub — its copied engine is deleted, limits preserved (aicore P1)
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
2026-09-01 16:46:33 +02:00
Admin
145d0b1fe2 score: the notation suite — engraving, layout, playback, midi and musicxml import, the physical piano model, and the score app
Squashed from work:
- score: a headless music engraving, playback and notation engine
- score: the notation app — pianist mode, editing, playback
- piano_model: it was a plucked string by construction, and 20 voicings
- score: one document you can pan, zoom and navigate
- piano_model: the body tap was a click, and the objective was rewarding noise
- score: add the sound panel and library modules
- piano_model: a second engine, and the attack that finally sounded right
- score: two instruments, reverb and brightness — and the rest of the panel gone
- score model: a note remembers how it was struck, and the score remembers the pedal
- score import: keep the velocities and the pedal the file was carrying
- score playback: play the performance, not a flattened copy of it
- score ui: the music list moves to the sidebar, and the view stops fighting itself
- score: the application ships its font and eight performances
- piano_model: a limiter that rides the music, so the knee stops shaping chords
- piano_model: the forte bell was the treble's dynamic slope, and the bass was dying at its own prompt rate
- piano_model: the bridge decides each partial's decay, and a fixed multiplier cannot say that
- piano_model: each partial gets its own two coupled modes, from the eigen algebra
- piano_model: a median that fell between the peaks made every bass partial a drain
- score-ai: LocalBroker — the seam's in-process implementation over the session engine (aicore P8)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- score_pdf: the score model grew a pedal map — the pdf importer initialises it
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- score app: the shipped-piece test speaks the PERFORMANCES table
- zero-warning sweep, round three — the model lanes and the deep examples
2026-09-01 16:46:32 +02:00
Admin
e37c263b9c ai backbone: the hub era — makepad_ai deleted, every backend is a hub pipe; machine residency elections, job leases, ETA placement; the store only stores; creator pipelines run in the app (aicore)
Squashed from work:
- asset-ai: FastH3 4-step fast video backend; clip keyframes on the wire
- asset-ui: loop video chains — text→image→video that ends where it began
- h3: safetensors -> pruned-Q4_K GGUF quantizer for the 24GB DiT tiers
- h3_quant_gguf verify: row-error gates calibrated to the measured Q4_K floor
- asset-ai realtime: the feedback loop — the source anchors, the drifted frame inits
- asset-ai realtime: a feedback loop survives a resize and travels by default
- asset-ai realtime: the feedback loop frees itself from the feed handshake and pauses for its listener
- asset-ai realtime: the outbound encode leaves the loop's critical path
- asset-ai ocr: the ocr domain — Chandra 2 at page resolution, and the tower goes planner-owned
- llm slots: a lane can hold an image span — embedding prefill and a rope cursor of its own
- vision tower on CUDA: the encode leg gets its two missing kernels
- llm/ocr: one M-RoPE grid encoder for both image paths, and a livelock made an error
- vision tower on CUDA: the f16 GEMM keeps the precision it was throwing away
- live: a feed that moves box takes its trip with it — one seed image
- vision tower on CUDA: the tiled attention becomes bit-exact, and tensor cores go
- llm prefill on CUDA: the MMA attention kernel gets the tile a 4-to-1 model needs
- asset-ai ocr: the CUDA encode lane joins the integration — vision-parity sits beside run's three arms, and the kernels
- Merge branch 'ocr-perf-integration' into work
- asset-ai: the live anchor can follow the trip, and text leaves the 5090
- asset-ai: the camera moves the world, and the world starts still
- asset-import: the EA strategy classics, in the one 2D contract
- rtsmap: one seeded generator for tiled strategy maps
- asset-ui: one card for the strategy classics, with a pack dropdown
- asset-ai: music3 reference-audio path, ocr/h3 backends, registry
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- cnc: tiberium is twelve growth frames, not twelve empty variants
- platform: native file and save dialogs, in-house on all three desktops
- chat: the scan holds out for a lane home
- chat: a full home queues you — take the free lane
- chat: the preload has a percentage, and the boundless cap stops showing
- llm cuda: the 32x2 attention tile — even GQA ratios stay on MMA
- sa3 gets a bake path: the sfx model's tables precomputed by a diffusion-side bin
- sqlite_query: anti-join regression test
- td import: HARV's second frame block is its harvesting cycle, not a turret
- asset-ui: sprite enhancement runs on the 32B dev DiT — distillation, not the prompt, was the ceiling
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- asset-ui: test health fixtures grow the realtime field they were born without
- ai-hub: one home at ~/.makepad — weights/ run/ cache/ logs/, the service cache migrates from ai_content by a single re
- ai-hub: subprocess workers die with the node — process groups everywhere, PDEATHSIG on linux, one KILL_ON_JOB_CLOSE Jo
- ai-hub: the hub object — AiHub::in_process, pipes vocabulary, and the local LLM engine generalized out of mpfiles (aic
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- ai-hub: the machine layer — node entries, the 0600 machine token, and the residency election that IS the lock (aicore
- ai-hub: MPHUB1 — the fabric beacon only dedicated nodes can send (aicore §4)
- ai-hub: job leases — work lives only while it is renewed (aicore §8)
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- ai-hub: RAM residency facts — the CPU-side twin of residency.rs (aicore §3)
- ai-hub: ETA placement primitives — relative GPU throughput, the four-term estimate, and an observable breakdown (aicor
- ai-hub: leases go live on the wire — origin fields on submit, /job/<id>/keepalive, /bye, and the reaper that cancels w
- ai-hub: the chat providers move in — fleet qwen, openai, grok, claude/codex/grok CLIs, the responses driver, and the w
- asset-creator: the engine — one pipeline run against the hub, deps-gated, spliced, cancellable, resumable-by-construct
- ai-hub: the machine node mode — --machine binds loopback, registers in ~/.makepad/run, and exits on its own once idle
- asset-creator: makepad-creator-run — the detached client for runs that must outlive a window (aicore §9)
- ai-hub: a native Claude Messages-API provider — API-key or Claude Code OAuth, bounded SSE streaming, injected tools (a
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- asset-creator: the preset tables move in — fifteen chain-policy constants shared by every creator app (aicore §9 / P6)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- ai-hub: loads hold the machine residency election — set_model_state claims on Loaded and publishes the service port (a
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- ai-hub: pick_for_domain_eta — ETA-ranked placement over the shared hard-filter core (aicore §6 / P4)
- asset-creator: the engine picks a provider per stage at dispatch time — a chain's later stages see fresh fleet state (
- ai-hub: the fabric secret gates the service HTTP surface — bearer on everything but /health and the ticketed peer path
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- chat-ui: the session runs in the app — no broker anywhere on the chat path (aicore P8 / F5)
- asset-store: assets.query is a first-class query endpoint — the bounded SQL surface outlives the broker (aicore P8 / F
- asset-creator: CreatorTools — the chat tool pack for a store that only stores (aicore §9 / P8)
- asset-store: the shrink — the store stores (aicore P7)
- importer + asset-server host: the coordination era ends (aicore P7)
- store config purge + asset-ui goes fleet-direct; the derive protocol gets its route proof (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- ai-hub: 0.3.0 — the health version says which era a node runs
- ai-hub: the default fleet is 'gen' — apps hear the LAN without env plumbing
- ai-hub: the preload note percents the prefill, not the job bar
- ai-hub: conversations keep their KV — the wire mirror, the lane identity, the in-turn dynamic context (aicore §7)
- ai-hub: an open-think model is thinking from its first token
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades

Co-authored-by: Claude <info@makepad.nl>
2026-09-01 16:46:31 +02:00
Admin
dbb82b8b61 chore: the zero-warning sweeps — every target of every workspace crate compiles clean on macos, windows, linux, ios and android
Squashed from work:
- platform, draw, widgets: the cross-target zero-warning sweep
- zero-warning sweep, round two — the first full-workspace pass
2026-09-01 16:46:29 +02:00
Admin
bada23dda2 frame witnesses: presents, uploads, encode and gpu time, a pinned display link, SHIFT+F12 screen capture, MPINPUT latency
Squashed from work:
- every window can record itself: SHIFT+F12 writes picture and sound to local/screencap
- the frame has witnesses now: presents, uploads, encode time, gpu time, and a pinned display link
- the upload counter names its kind: instances and textures split, and any single item over half a megabyte logs itself
- MPINPUT: input-to-glass latency in the present pulse
2026-09-01 16:46:29 +02:00
Admin
7e6e87fa1a platform: native file and save dialogs, in-house on all three desktops plus Android and iOS
Squashed from work:
- platform: native file and save dialogs, in-house on all three desktops
- platform: file dialogs on Android and iOS, and unbreak the Android build
2026-09-01 16:46:28 +02:00
Admin
db3864b0bd map rendering: overlays and labels, per-view clip, the world-clamped centre, carto's road ladder, the @cam gate
Squashed from work:
- map: overlay and label rendering
- map: bake a runnable Amsterdam test map from inside the route app
- map: gate the @cam readout; a .mkmap archive is a directory
- map: the view's own draw list carries its clip
- the map's centre is clamped to the world, not wrapped past it
- a country is drawn in hairlines: carto's road ladder joins by zoom
2026-09-01 16:46:28 +02:00
Admin
3b0b16e8ee draw: overlays composite above draw_depth content, per-session DrawVector geometry, one uniform into a whole draw list
Squashed from work:
- draw: DrawVector reused one geometry slot for every session in a frame
- draw, platform: overlays now composite above content that uses draw_depth
- DrawVars::set_uniform_on_draw_list — one uniform into every retained call of a shader in a list, pass repainted
2026-09-01 16:46:28 +02:00
Admin
149d19fcbb platform: midi pitch-bend byte order, many http connects at once, what a strategy round needs underneath
Squashed from work:
- render, sim, platform: what a strategy round needs underneath
- platform: midi pitch bend was sent with its bytes the wrong way round
- http: the connect slot is a gate, not a turnstile — many connects at once
2026-09-01 16:46:27 +02:00
Admin
e0530b061b glass + widgets: realtime parent blur, glass button icons, idling without draw_pass.time, fold_header opens honestly
Squashed from work:
- glass: a glass button can carry an icon
- widgets: the fab palette cell's colour is a field, not an object
- widgets: the glass stops reading draw_pass.time, so glass apps idle again
- live-with-parent passes: the glass blurs the world in realtime again
- fold_header: visually open is open
- fold_header: the ease snaps to its ends
- fold_header: settled state is the truth, the pane edge is the law, the area is the fold
2026-09-01 16:46:27 +02:00
Admin
b408131172 video: hardware first-frame decode from RAM, one encoder-transform report, stills without an AVAssetWriter
Squashed from work:
- video: hardware first-frame decode straight from RAM — no temp files
- video: the encoder transform is reported once, not once per encoder
- video: a single still does not need a whole AVAssetWriter
2026-09-01 16:46:27 +02:00
Admin
b3dd79978b tweaker + shader const-table: the design-feedback campaign — annotated props, hot-patchable shader constants, animator-state wells, the theme tab, typed editors, undo through anonymous paths
Squashed from work:
- widget_tree: skip-search nodes bound path-cache invalidation
- tweaker: fold the shader source view; plain TextInput, not CodeView
- docs: button shader annotations — widgets/button.rs complete + splash demo buttons
- tweaker: the Shader tab shows the pinned widget's ANIMATOR STATES as little posed swatches under the well — one per tr
- tweaker: the animator-state swatches are full-size wells stacked vertically under the main material well (same width,
- tweaker: typed editors for structured values — a reflected Vec2/3/4 is fused x/y/z/w scrub fields (the whole vector re
- shader: const-table mode — /** name min..max step s */ float literals inside fn bodies compile to hot-patchable scope-
- tweaker: the Props tab opens with TWEAKABLES — every annotated value (a /** */ doc on the key) hot-first, each with it
- tweaker: the sidebar row templates are hoisted into shared let bindings (one source of truth for the Props list, the S
- docs: shader-code constants annotated in the six core widgets
- tweaker: SHADER CONSTANTS — the Props tab opens with the annotated literals inside the pinned widget's draw-layer fn b
- tweaker: undo/redo resolve the pinned widget when a history step's path runs through anonymous segments (a '-' or a li
- shader const table: whole-number literal operands lift too — the parser packs `x + 6.`, `y - 2. - b`, `w * 4.`, `n - 1
- tweaker: the Shader tab reads well stack → SHADER CONSTANTS for the mirrored layer → INPUTS (its uniforms/instances, a
- tweaker: the panel cleaned up to the user's list — the filter (with the exploded-view and note buttons) is the top row
- tweaker: three panel polish nits — a section whose rows are all secondary leads with its first three instead of an emp
- tweaker: the VJ session's fix batch — (1) mirrored material wells clip to their scroll viewport: the well is its own d
- tweaker: theme colours, chunk 1 — the panel reads the app's palette from mod.theme's cascade once per session (every c
- tweaker: the theme hover pulse — hovering a colour chip pulses that theme colour live across the whole app, and grabs
- tweaker: the theme palette strip in the colour popover — hover names and pulses a theme colour, a click binds the prop
- tweaker: the Theme tab — every theme colour and number in a fourth panel tab, colours edited live app-wide, all of it
- tweaker: F12 works without the bridge
- tweaker: the selection wears viewfinder corners, not a box
- tweaker: radius handles come to the hand, not the eye
2026-09-01 16:46:26 +02:00
Kevin Boos
0e19a65eac
x11/opengl/wayland/metal: implement what these backends silently skipped (#1201)
* windows: publish shader-cache entries atomically, and unwind a cancelled resize

Two ways a second instance, or an interrupted drag, leaves a window permanently
degraded. Both were found auditing the backend rather than reported, and both
are cheap.

The DXBC cache is a plain per-user directory that every makepad process on the
machine reads and writes, and entries were written straight to their final path.
`fs::write` truncates first, so a second instance starting at the same moment
could read the prefix of a shader the first was still writing -- the file exists
and the read succeeds, so nothing notices until `CreateVertexShader` is handed a
truncated blob. `shader_bytes_cached` even documented the read path as catching
this, which it did not. Entries are now written to a temporary file and renamed
into place, which is atomic, and a blob that is not a complete DXBC container
(the 32-byte header and its magic) is recompiled instead of used. The temporary
name carries the process id so two writers cannot collide on it either.

`is_in_resize` and the 8 ms resize timer are armed by WM_ENTERSIZEMOVE and were
disarmed only by WM_EXITSIZEMOVE. Win32 does not guarantee that pairing, and a
drag that ends without it leaves the timer forcing repaints forever and the
window presenting unpaced for the rest of its life -- it never returns to vsync.
WM_CANCELMODE now unwinds the loop, gated on having actually been in it, since
that message also arrives for menus and capture changes and unwinding a resize
that was not running would cost a needless ResizeBuffers each time.

Verified against a release build: two instances launched simultaneously on a
cold cache both come up correctly with no panics and no temporary files left
behind, and window resizing is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* d3d11: validate a cached shader against the size its own header declares

Checking the magic and a 32-byte minimum accepts a blob that was truncated
after its header, which is exactly the shape a half-written cache entry takes.
The DXBC header carries the container's own total size at offset 24, so
comparing it to the length rejects those too.

Publishing entries atomically stops this backend from writing a partial one,
but the cache is a plain shared directory that older builds have already
written to, so the read path still has to be able to tell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* windows: stop double-scaling a popup window's position

`CxOsOp::CreatePopupWindow` arrives with a position the caller has already put
in physical screen pixels -- it adds the parent window's physical origin to an
offset it scaled by the parent's per-monitor DPI -- and `new_popup` then
multiplied it by the DPI again. The popup landed at `dpi` times its intended
screen coordinates: exact at 100%, and progressively further away above it.

The size was scaled by the *system* DPI, which is the primary display's and
goes stale after a live scale change, so on a second display of a different
scale it was the wrong number twice over. It is now passed through unscaled,
because it is provisional either way: `init` runs `set_inner_size` immediately
afterwards, which scales by the window's own per-monitor DPI once the HWND
exists on its target display.

Note for reviewers: this is unreachable today. `WindowHandle::new_popup` is the
only producer of the op and nothing in the tree calls it, so there is no symptom
to reproduce -- which is also why the arithmetic was free to drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* x11: implement window resizing, and stop a protocol error killing the app

Two things the X11 backend did not do, both of which it silently pretended to.

`set_inner_size` and `set_outer_size` were empty function bodies, so
`CxOsOp::ResizeWindow` dispatched to nothing: an app calling `WindowHandle`'s
resize on X11 got no error, no log and no resize. They now call
`XResizeWindow`, clamped to the CARD16 range the protocol encodes an extent in,
since a zero is a BadValue. `set_outer_size` delegates rather than pretending:
the window manager owns the decoration frame, so a client can only ask for its
own extent.

Xlib's default error handler prints to stderr and calls `exit(1)`, and makepad
installed none, so a single rejected request killed the process outright -- a
bad geometry, a race against a window the WM has already destroyed, a missing
extension. Protocol errors are asynchronous and frequently not caused by the
code that happens to be running, so terminating is never the proportionate
response. A handler is now installed before the first request and logs the
error, request and minor codes plus the resource id.

`XResizeWindow` and `XSetErrorHandler` were not in the hand-written Xlib
bindings; both are added, along with the `XErrorHandler` callback type.

Untested: the author has no X11 machine. Both changes are small, local and fail
closed -- an unusable size is clamped rather than sent, and the handler only
turns an existing hard exit into a log line -- but neither has been run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* opengl/wayland/metal: name three failures the backends currently swallow

None of these is device-loss recovery -- that stays a D3D11-only feature. They
are the cases where a backend already fails and says nothing useful, so a bug
report arrives as "the window went black" with no cause attached.

EGL: `eglMakeCurrent` and `eglSwapBuffers` failures were logged unlatched on
paths that run every frame, so a persistent failure emitted thousands of lines a
second. Both are latched once per outage and cleared on the next success, and
`EGL_CONTEXT_LOST` (0x300E) is now named explicitly, since that is EGL saying
the GPU reset and every GL object is dead -- which this backend cannot yet
recover from, and should at least say so. `OpenglCx::make_current` returned `()`
while discarding the result of a call that can fail; it returns `bool` now, so a
caller can skip GL work that would otherwise run with no current context and be
silently dropped.

Wayland: `wp_viewport.set_destination` raises the `bad_value` protocol error --
which disconnects the client -- on a zero or negative extent, and a float-to-int
cast turns both a negative and a NaN into zero. `WaylandWindow::new` clamps the
size for the EGL call but stores it unclamped into the geometry, so an app
created at a degenerate size reached that request. The two destination writes
are floored the way the EGL extent already is.

Wayland `CxOsOp::ResizeWindow` was an empty arm. A client has no "set my size"
request, but window geometry defaults to whatever the surface commits, and the
paint path derives the EGL extent and viewport destination from
`window_geom.inner_size` -- so writing it is the whole operation. Refused for a
maximized or fullscreen toplevel, which must keep the configured geometry or the
compositor raises `invalid_surface_state`. `RepositionWindow` stays a no-op and
now says why: `move` is interactive and serial-gated, and `reposition` is an
xdg_popup request needing a protocol version this backend does not bind.

Metal: a command buffer that ends in `MTLCommandBufferStatusError` produced no
pixels, and nothing noticed -- the completion handler runs either way, so the
in-flight queue drains and the hang watchdog stays quiet over a stale window.
The status is now checked in the handler that already exists and the error named,
capped at 32 reports.

Untested: none of these three backends can be run here. All three are
type-checked for their targets. Deliberately NOT written: anything for Vulkan
(vulkan.rs is not compiled by any target available here, so it cannot even be
type-checked) or the web backend (no JS runtime available to syntax-check the
shim), and any Metal device-loss latch or recovery, because `MTLCreateSystemDefaultDevice`
returns the non-removable SoC GPU and loss in the D3D11 sense does not occur there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* x11/wayland: cap a window extent, and refuse a resize that says nothing

Running the branch on Linux turned up four things it and #1197 let through.

A saved size is floored but never capped, and Wayland is the one backend with
no displays to fit against, so `{"inner_size":[1e9,1e9]}` reached
`eglCreateWindowSurface` and tripped its `assert!`. The app then died before it
could rewrite the state file that was killing it, so every later launch died the
same way -- the exact failure #1197 exists to prevent, one line further down.
`sanitize_window_geom` now caps as well as floors, and both Wayland EGL surface
creations fall back to the default size rather than taking the process with them.

`CxOsOp::ResizeWindow` had the same hole and no cap at all on Wayland:
`resize(100000, 100000)` went straight into the window geometry, which left the
surface EGL_BAD_SURFACE and corrupted the size the toplevel restores to -- a
maximize/restore round trip came back 1259x1259, a fullscreen one 100000x100000.
X11 had the opposite problem: it clamped a zero or a negative up to the CARD16
floor and produced a one-pixel window, silently, where Wayland refused the same
request and said why. Both now go through one `sanitize_resize`, so they answer
a bad request identically.

Last, the new X11 error handler is an `extern "C"` frame and logging panics on a
closed stdout -- `app | head` is enough -- so a protocol error aborted the
process instead of the `exit(1)` the handler exists to prevent. Reporting is
wrapped in `catch_unwind`.

Verified on Ubuntu 25.10 / GNOME on both backends: the 1e9 file now starts and
repairs itself, the resize matrix caps at 16384 with no EGL error and restores
correctly, and the two backends log the same refusal. 76 unit tests.

Still open, deliberately not fixed here: a Wayland self-resize dispatches no
WindowGeomChange. handle_platform_ops holds the Cx borrow for its whole loop, so
sending one needs a deferred-event path that does not exist yet.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-31 21:50:37 +02:00
Kevin Boos
fd6f307a2b
windows: publish shader-cache entries atomically, and unwind a cancelled resize (#1200)
* windows: publish shader-cache entries atomically, and unwind a cancelled resize

Two ways a second instance, or an interrupted drag, leaves a window permanently
degraded. Both were found auditing the backend rather than reported, and both
are cheap.

The DXBC cache is a plain per-user directory that every makepad process on the
machine reads and writes, and entries were written straight to their final path.
`fs::write` truncates first, so a second instance starting at the same moment
could read the prefix of a shader the first was still writing -- the file exists
and the read succeeds, so nothing notices until `CreateVertexShader` is handed a
truncated blob. `shader_bytes_cached` even documented the read path as catching
this, which it did not. Entries are now written to a temporary file and renamed
into place, which is atomic, and a blob that is not a complete DXBC container
(the 32-byte header and its magic) is recompiled instead of used. The temporary
name carries the process id so two writers cannot collide on it either.

`is_in_resize` and the 8 ms resize timer are armed by WM_ENTERSIZEMOVE and were
disarmed only by WM_EXITSIZEMOVE. Win32 does not guarantee that pairing, and a
drag that ends without it leaves the timer forcing repaints forever and the
window presenting unpaced for the rest of its life -- it never returns to vsync.
WM_CANCELMODE now unwinds the loop, gated on having actually been in it, since
that message also arrives for menus and capture changes and unwinding a resize
that was not running would cost a needless ResizeBuffers each time.

Verified against a release build: two instances launched simultaneously on a
cold cache both come up correctly with no panics and no temporary files left
behind, and window resizing is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* d3d11: validate a cached shader against the size its own header declares

Checking the magic and a 32-byte minimum accepts a blob that was truncated
after its header, which is exactly the shape a half-written cache entry takes.
The DXBC header carries the container's own total size at offset 24, so
comparing it to the length rejects those too.

Publishing entries atomically stops this backend from writing a partial one,
but the cache is a plain shared directory that older builds have already
written to, so the read path still has to be able to tell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* windows: stop double-scaling a popup window's position

`CxOsOp::CreatePopupWindow` arrives with a position the caller has already put
in physical screen pixels -- it adds the parent window's physical origin to an
offset it scaled by the parent's per-monitor DPI -- and `new_popup` then
multiplied it by the DPI again. The popup landed at `dpi` times its intended
screen coordinates: exact at 100%, and progressively further away above it.

The size was scaled by the *system* DPI, which is the primary display's and
goes stale after a live scale change, so on a second display of a different
scale it was the wrong number twice over. It is now passed through unscaled,
because it is provisional either way: `init` runs `set_inner_size` immediately
afterwards, which scales by the window's own per-monitor DPI once the HWND
exists on its target display.

Note for reviewers: this is unreachable today. `WindowHandle::new_popup` is the
only producer of the op and nothing in the tree calls it, so there is no symptom
to reproduce -- which is also why the arithmetic was free to drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-31 21:50:26 +02:00
Admin
fd0ea7e264 chore: workspace members and remaining sources
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- Modal claims no layout slot: the DJ page fills its window again
2026-08-29 09:26:27 +02:00
Admin
4ddba62d4b engine libs: csg document, stitch simd, sim providers, teamtalk LAN voice, gltf writer, render, route and maps, bench hygiene
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- work: land the sources the last commits reference
- route: the assistant icon the committed UI references
- fast_inflate: the benches name their dev-deps
- gif/weezl: drop the vendored benches nobody can run
- weezl: the decode tests generate their own LZW fixture
2026-08-29 09:26:27 +02:00
Admin
d25cce9ae6 mp*: the window-manager app family — quick-look, warm-instance pool, pty teardown, theme bridge
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- mpwm: ghost-launch/menu-flash root cause, scroll z-gate, browser binds, path-free status, viewers delisted, Fab opens
- mpwm: Hyprland-exact close animation over a frozen snapshot
- mpwm: warm-instance pool, flat-luminance opens, flicker-free CEF resize
- work: land the sources the last commits reference
- mpwm: quick-look gap fixes; image cache eviction on preview unload
- tweaker: material thumbnails + vibecode popup + ctrl-space notes, undo/redo over the edit ledger, capture-semantics pi
- tweaker: tabbed side panel (Props/Shader/Tree) - shader tab with checkerboard material well + prompt, complete widget-
- mpwm+mpterm: text quick-look — pty teardown deadlock fix, in-place retarget verified, debug probes stripped
2026-08-29 09:26:26 +02:00
Admin
19097335b6 vj: the responsive DJ console (PR #1199 by vjroger), the DREAM expand-flux-video pipeline as one store graph, 16:9 image particles, XF defaults
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- work: land the sources the last commits reference
- vj: responsive DJ mixer + Windows drag-and-drop, cherry-picked from PR #1199 (vjroger)
- vj particles: image mode homes on a 16:9 picture plane
- vj: DREAM pipeline — expand -> flux1 still -> minimax i2v loop, with a canvas picker
- vj: the row shows the real prompt, a flux picker, XF default, and new clips stop vanishing
- a DREAM run is one declared graph the store runs without us
2026-08-29 09:26:26 +02:00
Admin
d24cbee60d asset platform: store pipelines with honest banded progress, vision annotation through the job queue, fleet scheduling (roles, spread, peer provisioning), the music contract, synonym search, the runs card
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- work: land the sources the last commits reference
- kenney: catalogue all 50 free 3D kits; Modal dismissed() never fired
- asset store: central vision-annotation queue; Kenney donate prompt
- asset-ai: vision domain — image + prompt -> text on every fleet node
- annotation runs the fleet's vision services through the normal job queue
- vj: responsive DJ mixer + Windows drag-and-drop, cherry-picked from PR #1199 (vjroger)
- store search: WordNet synonym expansion, query-side; per-term seeks
- video scheduling: a cold model pin never downloads past a warm one, and a dedicated box only serves its role
- an expansion can never lose a run — and `expand: true` finally means something
- fleet panel: a slow box is not a missing box
- h3: first+last keyframe conditioning (the weights were always FL2VA)
- registry: the three H3 FL2VA tiers name their real conditioning
- fleet scheduling: spread before stacking, and a stuck job moves
- a download never steals a job from a box that has the weights
- the faster GPU takes the tie: 6000 > 5090 > 4090
- an evicted flux model gives the card its VRAM back
- a job now records what each of its stages was handed
- open a stage in RUNS and read what it sent
- every expanded song was an instrumental: the lyrics had nowhere to go
- the writer never overwrites words the person wrote
- store: a dependent job's body is spliced from its deps' results at claim
- a fleet box's job row says what that job was asked for
- the store runs a whole pipeline, and one record says how far it got
- a client can declare a whole run, watch it, and stop it
- 100% now means published, and the expander is a job you can queue
- one card says what a spawned task is doing, everywhere
- a run that ends says so, instead of being noticed later
- when every box holding the model is busy, buy another copy
2026-08-29 09:26:25 +02:00
Admin
3b4d6a4ff7 platform+widgets: sploded 3D inspect view, the tweaker design-feedback suite, script math-AOT and docs channel, pointer-pin and pixel-probe, modal layout, map warp
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- mpwm: warm-instance pool, flat-luminance opens, flicker-free CEF resize
- work: land the sources the last commits reference
- kenney: catalogue all 50 free 3D kits; Modal dismissed() never fired
- platform: windows check green again — SetWindowTextW binding
- map: exact warp-aware inverse projections — pointer ops work folded
- mpwm: quick-look gap fixes; image cache eviction on preview unload
- tweaker: material thumbnails + vibecode popup + ctrl-space notes, undo/redo over the edit ledger, capture-semantics pi
- tweaker: vibe popup card chrome + dispatch order, ctrl-space notes verified, sploded design v2 chapter
- tweaker: tabbed side panel (Props/Shader/Tree) - shader tab with checkerboard material well + prompt, complete widget-
- sploded v2: nesting-depth z, hairline scope frames, body pass
- sploded: pin the depth convention with a test, kill the draw_depth residue
- sploded: real body-pass split (scene-pass capture, panel flat) + y-convention source of truth with anti-flip gate test
- sploded: hollow outlines, flat-band input, ray-pick unprojection
- tweaker: shader tab defaults to the selection's first draw layer, stale hint trimmed
- sploded: outlines become clipped, antialiased strips; tighter deck
- sploded: merge the lane's v2 (nesting-depth z, clipped AA strip outlines, flat-band input, unproject, SplodedStack bod
- sploded: the exploded view is a LIVE view — pointer events route through the inverse explode transform (ray -> plane -
- tweaker: tabs are real widgets (uid, tree node under the dock, own plane in 3D) and pickable; navigation-class clicks
- sploded: pinned/hover outlines render on the widget's own plane in 3D — per-widget nesting depth lives on the platform
- tweaker: the material well renders the pinned widget's actual shader — the swatch byte-copies the widget's live draw c
- tweaker: the Shader tab shows the shader as written — the layer's pixel/vertex fn source (nearest definition up the co
- sploded: I = true isometric preset (yaw 45°, pitch atan(1/√2))
- tweaker: eyedropper — the colour popover's pick button arms a pixel probe; the next press in the app samples that devi
- tweaker: the shader loop closes — /tweak/apply resolves the pinned widget by uid (anonymous path segments never round-
- vj: responsive DJ mixer + Windows drag-and-drop, cherry-picked from PR #1199 (vjroger)
- tweaker: the material well is a magnifier — the mirrored instance draws at the widget's native size in the well's own
- tweaker: per-layer material thumbnails — the Widget derive emits WidgetNode::layer_areas() (every #[live] Draw… field
- tweaker: the shader source view is the real CodeView (syntax highlighting, selection, editing) when the app registers
- tweaker: Ctrl+Enter sends on every platform (TextInput treated only Cmd as primary on macOS, so Ctrl+Enter inserted a
- Modal claims no layout slot: the DJ page fills its window again
- tweaker: every fn apply recompiles (eval_chunk ran every chunk under ONE synthetic callsite, so the script body — and
- tweaker: an apply whose draw shader fails to compile is rejected — the layer goes back (last live fns / the fn as writ
- tweaker: the Shader tab's source view owns its scrolling (the ScrollYView around the CodeView double-scrolled the care
- widgets: set_visible belongs to every widget, not just View (#1194)
- script: a dead heap's resource handles must not outlive it (#1195)
- Resources: search the executable's directory, not only the working directory (#1196)
- Windows: fit a restored window to the displays that are actually attached (#1197)
- d3d11: a failing GPU call reports the loss instead of killing the process (#1198)

Co-authored-by: Kevin Boos <1139460+kevinaboos@users.noreply.github.com>
2026-08-29 09:26:24 +02:00
Kevin Boos
7a342be4d4
d3d11: a failing GPU call reports the loss instead of killing the process (#1198)
* d3d11: a failing GPU call reports the loss instead of killing the process

The backend already notices a removed device when `Present` returns
DXGI_ERROR_DEVICE_REMOVED, but a device rarely dies at a moment as convenient
as a present. It dies between frames, and the next thing that touches it is a
resource creation or a buffer map -- of which this file had 76 unwrapped, plus
three `std::process::exit(1)`. So the usual outcome of a GPU driver reset,
a TDR or a hybrid-GPU transition across suspend/resume was a panic, and the
graceful path was unreachable.

Route the calls a dead device actually reaches through `D3d11Cx::note_error`,
which asks `GetDeviceRemovedReason` rather than pattern-matching the HRESULT the
failing call happened to return -- creation calls do not reliably return the two
DXGI device-lost codes, while the device itself always knows and keeps saying
so. That answer sets a process-wide `device_lost` latch and logs once.

The softened sites are the ones a dead device lands on first: draw-list and pass
uniform buffers, which upload every frame with no dirty gate; texture and render
target creation; and shader object creation, whose `CxOsDrawShader::new` already
returned `Option` with both callers handling `None`.

Three latent bugs fall out of auditing them, each of which loses content
permanently rather than noisily:

  - `update_vec_texture` consumed the dirty flag with `take_updated()` and then
    returned early when the pixel buffer was out on loan, so a texture that hit
    that window was never uploaded again. For the glyph atlas that means all
    text disappears for the life of the process. The Metal backend already
    guards this; D3D11 did not.
  - `hlsl_compile_shaders` drains `compile_set` destructively, so a shader whose
    object creation failed was dropped from the queue forever and everything
    drawn with it silently stopped rendering. Failed creations go back in the
    queue.
  - `render_view` unwrapped the geometry index buffer but passed the vertex
    buffer through as an `Option`, so a missing one bound null and drew nothing
    with no error anywhere. Both are now checked together, and a draw call with
    either missing is skipped under a `debug_assert!` that it only happens on a
    lost device.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit cc980805b5774253e592e183f577c5ba08ce85db)

* d3d11: rebuild the device and every GPU resource after a device loss

Detection landed already: present() sets a per-window device_lost and the paint
loop stops re-dirtying the pass. Nothing rebuilt anything, so the window stayed
frozen until the app was restarted.

Recovery runs at the top of win32_event_callback, the one place holding both
&mut D3d11Cx and &mut Vec<D3d11Window> while nothing is mid-render. It releases
each window's swap chain, back buffer, view and paint-beat registration (DXGI
allows one flip-model chain per HWND, so the dead one must be gone first),
recreates the device tier when the device really is gone, drops every GPU handle
the Cx holds, and rebuilds each swap chain against the same HWND.

Clearing handles is only half of a sweep. Geometry and instance uploads are
gated on dirty flags cleared unconditionally once the upload runs, and textures
on a dirty rect consumed by take_updated, so every gate is re-armed or the empty
slot is never refilled. The CPU-side sources all survive: texture pixels live in
TextureFormat::Vec*, geometry in CxGeometry, and shaders keep their compiled
DXBC plus the on-disk cache, so recovery recreates shader objects without
compiling any HLSL.

Retries are spaced 250ms to 4s and driven by the existing signal heartbeat
rather than a new timer, because the GPU can stay absent for a long time. While
lost, the loop is forced to Wait at both EventFlow decisions -- every condition
that would otherwise choose Poll is unsatisfiable when nothing can paint, so it
would spin for the whole outage -- and pending screenshot requests are failed,
both to release their requester and because a non-empty queue is itself one of
those conditions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit d603d3e7996cd71b5da6ed3372c9dc75eebb4a77)

* d3d11: bind the hot-path buffers by reference, not by clone

The device-loss bails introduced an AddRef/Release pair per uniform-buffer
upload and per draw call, on paths that run for every draw list, pass and
geometry every frame. Borrowing reads the same Option without touching the
refcount; only IASetVertexBuffers genuinely needs an owned Option, which is what
the code built before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit ecfab9a9355d50754a7117de8303d66f8919b2ee)

* d3d11: make the device-loss recovery actually work, and add a way to exercise it

Three defects, each of which stopped recovery dead, and none of which is visible
without running it. `MAKEPAD_D3D11_TEST_DEVICE_LOSS=<seconds>` forces a full
device recreation on a timer so the path can be exercised without a driver
reset; a real removal cannot be provoked from inside the process. It is a
stronger test than merely setting the latch, because the device really is
replaced, so any GPU object the sweep fails to rebuild still belongs to the old
device and cannot render against the new one.

  - The sweep called `set_updated` on every texture, which panics for anything
    that is not a `Vec*` format. The first render target it reached took the app
    down. Only vec textures carry a dirty rect; render targets, depth buffers
    and shared textures have no CPU-side contents and get their alloc record
    cleared instead.

  - Every rebuilt swap chain failed with `E_ACCESSDENIED`. DXGI allows one
    flip-model swap chain per HWND at a time and D3D11 destroys lazily, so the
    immediate context's own reference to the back-buffer view kept the old chain
    -- and its claim on the window -- alive after the application had dropped
    every handle it held. `ClearState` + `Flush` once, after all the windows have
    released and before any rebuild.

  - The post-recovery redraw marked every pass slot dirty, including ones
    nothing had drawn into, and `draw_pass_to_texture` unwraps
    `main_draw_list_id` immediately. Only passes that have one are marked.

Verified against a release build: six consecutive forced device recreations,
no panics, the UI rendering correctly after each (text included, so the glyph
atlas re-uploads from its retained pixels), `ResizeBuffers` working on a rebuilt
chain, handle count flat across recoveries, and the loop idle afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit bb7d1c019612a4fb2668640f47b8e16c3886e1cf)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 09:41:33 +02:00
1231 changed files with 418741 additions and 42061 deletions

2
.gitignore vendored
View file

@ -5,6 +5,8 @@
# Private Arcade frontend. Clone into this path:
# git clone git@github.com:makepad/sandbox.git apps/sandbox
/apps/sandbox/
# git clone git@github.com:makepad/source-library.git apps/source-library
/apps/source-library/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

View file

@ -1,18 +1,33 @@
workspace.members = [
# === app ===
"apps/mpbrowser",
"apps/mpterm",
"apps/mpwm",
"apps/finance",
"apps/mpsheets",
"libs/mp_theme",
"libs/mp_wm_api",
"apps/mptask",
"apps/mpimage",
"apps/mpvideo",
"apps/mppdf",
"apps/mpfiles",
"apps/route",
"apps/asset-ui",
"apps/asset-server",
"apps/ai-hub",
"apps/vj",
"apps/mixer",
"libs/asset/importer",
# The tiled-RTS map generator, shared by the importer and the sandbox.
"libs/rtsmap",
"libs/texcomp",
"libs/asset/data",
"libs/asset/client",
"libs/asset/widgets",
"libs/asset/store",
"libs/asset/chat",
"libs/asset/chat_ui",
"libs/chat_ui",
"libs/asset/annotate",
"libs/render",
"libs/raytrace",
@ -22,6 +37,8 @@ workspace.members = [
"libs/sim",
"libs/sim/math",
"libs/show_control",
"libs/asset/creator",
"libs/strict_json",
# === remesh (FaithC port) / xatlas (jpcy port) ===
"libs/remesh",
"libs/xatlas",
@ -39,6 +56,7 @@ workspace.members = [
"examples/datagrid",
"examples/todo",
"examples/portallist_hit",
"examples/modal_footprint",
"examples/vector",
"examples/aichat",
"examples/pdf",
@ -71,6 +89,20 @@ workspace.members = [
"studio/desktop",
# === own SQLite-format database engine (P0 reader, sqlq CLI) ===
"libs/sqlite_query",
# === headless music score engine ===
"libs/score",
"libs/score_layout",
"libs/score_play",
"libs/score_render",
"libs/score_ai",
"libs/score_import",
"libs/score_pdf",
"libs/score_ui",
"apps/score",
"libs/midi_file",
"libs/soundfont",
"libs/piano_model",
"libs/musicxml",
# === own MP3 / Ogg Vorbis decoders ===
"libs/audio_decode",
# === own Ogg Vorbis encoder (stem side-channels) ===
@ -79,14 +111,25 @@ workspace.members = [
"libs/audio_lyrics",
# === stems + lyrics side-channel bake/publish (asset-ui + VJ) ===
"libs/audio_sidechannels",
"libs/teamtalk",
# === pictures of audio (spectrogram, wave strip, composite) ===
"libs/audio_picture",
# === mkfl motion payload + classical optical flow + all-intra re-encode
# (shared by the enhance backend and the VJ's flow-warp import) ===
"libs/video_flow",
# === realtime frame tweening: the classical GPU optical-flow pass
# chain, the RIFE producer behind the same warp, and the mode set
# (extracted from the VJ, which is still its reference user) ===
"libs/frametween",
# === archive.org search + download content input (VJ / asset-ui) ===
"libs/archive_org",
# === mp4 sample index for range-streaming playback ===
"libs/mp4_index",
# === necessary tools ===
"platform/video",
"tools/cargo_makepad",
# === OSM PBF -> tile archive + nav artifact bake passes (CLI + in-app) ===
"libs/map_build",
"tools/map_tiles",
"tools/map_bake",
"tools/remote",
@ -99,10 +142,12 @@ workspace.exclude = [
# like a subdirectory. Not a required member so a clean makepad tree
# still loads.
"apps/sandbox",
"apps/sandbox/tools/sandbox-eval",
# Private Source Library picture wall. Same shape as sandbox: clone into
# apps/source-library; path deps stay ../../.
"apps/source-library",
"libs/terminal_core",
# standalone GPU generate service (own workspace, like diffusion)
"libs/asset/ai",
"libs/ai/hub",
"libs/diffusion",
# the AI model workspace (loader + cuda/metal stores + model crates) — aiarch.md
"libs/ai",

35
apps/ai-hub/Cargo.toml Normal file
View file

@ -0,0 +1,35 @@
[package]
name = "makepad-app-ai-hub"
version = "0.1.0"
edition = "2021"
publish = false
[[bin]]
name = "makepad-ai-hub"
path = "src/main.rs"
[dependencies]
makepad-ai-hub = { path = "../../libs/ai/hub", default-features = false }
# Feature passthrough: the headless node exposes exactly the library's
# feature set, so box launch scripts keep their --features flags working.
[features]
default = ["flux", "paint", "paint-cuda", "llm", "tts", "indextts", "video", "interpolate", "audio", "mesh", "matte-native", "depth-native", "segment-native", "upscale-native", "motion-native", "rig-native", "splat-native"]
python-backends = ["makepad-ai-hub/python-backends"]
flux = ["makepad-ai-hub/flux"]
paint = ["makepad-ai-hub/paint"]
paint-cuda = ["makepad-ai-hub/paint-cuda"]
llm = ["makepad-ai-hub/llm"]
tts = ["makepad-ai-hub/tts"]
indextts = ["makepad-ai-hub/indextts"]
video = ["makepad-ai-hub/video"]
interpolate = ["makepad-ai-hub/interpolate"]
audio = ["makepad-ai-hub/audio"]
mesh = ["makepad-ai-hub/mesh"]
matte-native = ["makepad-ai-hub/matte-native"]
depth-native = ["makepad-ai-hub/depth-native"]
segment-native = ["makepad-ai-hub/segment-native"]
upscale-native = ["makepad-ai-hub/upscale-native"]
motion-native = ["makepad-ai-hub/motion-native"]
splat-native = ["makepad-ai-hub/splat-native"]
rig-native = ["makepad-ai-hub/rig-native"]

View file

@ -1,14 +1,14 @@
//! makepad-asset-ai service binary. Runs on each GPU box; wraps all AI
//! makepad-ai-hub service binary. Runs on each GPU box; wraps all AI
//! content generation behind a port.
//!
//! ```text
//! makepad-asset-ai [--port N] [--host ADDR] [--cache-dir PATH] [--registry PATH]
//! makepad-ai-hub [--port N] [--host ADDR] [--cache-dir PATH] [--registry PATH]
//!
//! --port listen port (env MAKEPAD_ASSET_AI_PORT, default 8765)
//! --host bind address (default 0.0.0.0)
//! --fleet partition name (env MAKEPAD_ASSET_AI_FLEET, default default)
//! --cache-dir model + artifact dir (env MAKEPAD_ASSET_AI_CACHE,
//! default <home>/.makepad/ai_content)
//! default <home>/.makepad/weights)
//! --registry registry json path (default: <cache-dir>/registry.json if it
//! exists, else the embedded registry)
//!
@ -16,10 +16,10 @@
//! env MAKEPAD_ASSET_AI_HF_BASE alternate HF endpoint / LAN mirror
//! ```
use makepad_asset_ai::download::Downloader;
use makepad_asset_ai::registry::Registry;
use makepad_asset_ai::server::{start_service, ServiceConfig};
use makepad_asset_ai::{AssetAiError, DEFAULT_PORT, SERVICE_NAME, SERVICE_VERSION};
use makepad_ai_hub::download::Downloader;
use makepad_ai_hub::registry::Registry;
use makepad_ai_hub::server::{start_service, ServiceConfig};
use makepad_ai_hub::{AssetAiError, DEFAULT_PORT, SERVICE_NAME, SERVICE_VERSION};
use std::path::PathBuf;
fn main() {
@ -35,6 +35,7 @@ fn run() -> Result<(), AssetAiError> {
let mut fleet: Option<String> = None;
let mut cache_dir: Option<PathBuf> = None;
let mut registry_path: Option<PathBuf> = None;
let mut machine = false;
let mut args = std::env::args().skip(1);
while let Some(arg) = args.next() {
@ -69,9 +70,15 @@ fn run() -> Result<(), AssetAiError> {
AssetAiError::Io("--registry needs a value".into())
})?));
}
// The machine node (aicore §3): loopback-only, registered in
// ~/.makepad/run for the apps on this machine, and gone on its
// own once nothing needs it — a cache, not a daemon.
"--machine" => {
machine = true;
}
"--help" | "-h" => {
println!(
"{SERVICE_NAME} {SERVICE_VERSION}\nusage: {SERVICE_NAME} [--port N] [--host ADDR] [--fleet NAME] [--cache-dir PATH] [--registry PATH]"
"{SERVICE_NAME} {SERVICE_VERSION}\nusage: {SERVICE_NAME} [--port N] [--host ADDR] [--fleet NAME] [--cache-dir PATH] [--registry PATH] [--machine]"
);
return Ok(());
}
@ -91,8 +98,8 @@ fn run() -> Result<(), AssetAiError> {
},
};
let cache_dir = cache_dir.unwrap_or_else(default_cache_dir);
let fleet = makepad_asset_ai::discovery::normalize_fleet(
&fleet.unwrap_or_else(makepad_asset_ai::discovery::fleet_from_env),
let fleet = makepad_ai_hub::discovery::normalize_fleet(
&fleet.unwrap_or_else(makepad_ai_hub::discovery::fleet_from_env),
);
// Registry: explicit path > registry.json dropped into the cache dir
@ -108,6 +115,11 @@ fn run() -> Result<(), AssetAiError> {
}
};
// The machine node is machine-local by definition: loopback bind, no
// matter what --host said.
if machine {
host = "127.0.0.1".to_string();
}
let downloader = Downloader::from_env()?;
let handle = start_service(ServiceConfig {
host,
@ -134,19 +146,63 @@ fn run() -> Result<(), AssetAiError> {
" endpoints: /health /models /jobs /loras POST:/generate /job/<id> POST:/job/<id>/cancel /artifact/<id> /v1/model_inventory /v1/model_blob/<sha256> POST:/realtime GET(ws):/realtime/<id>"
);
if machine {
return run_machine_node(handle);
}
// The http listener thread runs until the process is killed.
let _ = handle.http_thread.join();
Ok(())
}
/// The machine node's life: register in ~/.makepad/run so the apps on this
/// machine find it, then idle down and exit once nothing has needed it for
/// the TTL — it reads as a cache, not a daemon (aicore §3). Visible in any
/// process list as makepad-ai-hub.
fn run_machine_node(handle: makepad_ai_hub::server::ServiceHandle) -> Result<(), AssetAiError> {
use makepad_ai_hub::machine::{write_node_entry, NodeEntry};
use std::time::{Duration, Instant};
let ttl_min: u64 = std::env::var("MAKEPAD_AI_HUB_MACHINE_TTL_MIN")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(15);
let entry = NodeEntry {
pid: std::process::id() as u64,
port: handle.addr.port(),
pipes_hash: 0,
};
let entry_path = write_node_entry(&handle.shared.node_key, &entry)
.map_err(|e| AssetAiError::Io(format!("write node entry: {e}")))?;
println!(" machine node: registered {} (ttl {ttl_min}m idle)", entry_path.display());
let mut idle_since = Instant::now();
loop {
std::thread::sleep(Duration::from_secs(30));
// Busy = queued/running work, or a model somebody paid to load.
let pending = handle.shared.jobs.with(|store| store.pending_count()) > 0;
let resident = handle
.shared
.models
.lock()
.unwrap()
.values()
.any(|track| matches!(track, makepad_ai_hub::server::ModelTrack::Loaded));
if pending || resident {
idle_since = Instant::now();
} else if idle_since.elapsed() > Duration::from_secs(ttl_min * 60) {
println!("{SERVICE_NAME}: machine node idle for {ttl_min}m — exiting");
let _ = std::fs::remove_file(&entry_path);
return Ok(());
}
}
}
fn default_cache_dir() -> PathBuf {
if let Some(dir) = std::env::var_os("MAKEPAD_ASSET_AI_CACHE") {
return PathBuf::from(dir);
}
// USERPROFILE on Windows, HOME elsewhere; temp dir as a last resort.
let home = std::env::var_os("USERPROFILE")
.or_else(|| std::env::var_os("HOME"))
.map(PathBuf::from)
.unwrap_or_else(std::env::temp_dir);
home.join(".makepad").join("ai_content")
makepad_ai_hub::home::default_weights_dir_with_migration(&mut |message| {
eprintln!("{SERVICE_NAME}: {message}");
})
}

View file

@ -0,0 +1,358 @@
//! FULLY-LOCAL MODE, shared: an app hosting its own Asset Server.
//!
//! The VJ and the sandbox are thin clients over a store somebody else runs —
//! normally asset-ui's embedded one, or the standalone host. That stays the
//! default and still wins whenever it is actually reachable. This module is
//! the shared answer to "and if it is not?": the app brings up [`Host`] in
//! its own process, on 127.0.0.1 only, rooted in the user's main library
//! when one exists on disk (a private seed root only when none does), with
//! the ai-content library publisher riding along.
//!
//! The thin-client law does not bend for this. Hosting changes WHERE the
//! store runs, never who owns the content: the app still browses, publishes
//! and fetches over HTTP, through the same `AssetClient`, with the same
//! catalog-event subscription. Nothing durable moves into the app.
//!
//! ## Loopback, deliberately
//!
//! The embedded host binds 127.0.0.1 for both planes (port 0 = OS-assigned,
//! or pinned by `<PREFIX>_ASSET_PORT`) and runs NO discovery beacon: an app
//! hosting for itself has no business accepting connections from the
//! network, and announcing the store would invite other apps onto a private
//! instance of the user's library. That is also what makes reference
//! imports safe to enable here (see `makepad_asset_store::blobrefs`).
//!
//! ## Choosing
//!
//! `<PREFIX>_ASSET_EMBED` decides, `auto` by default:
//!
//! - `never` — attach only; if nothing is reachable, behave as before.
//! - `auto` — attach if an external store ANSWERS, else host.
//! - `always` — host, regardless of what else is running.
//!
//! "Answers" is a real probe, not a guess: `GET /v1/health` against the
//! endpoints the caller advertises, failing that a short listen for a UDP
//! beacon, failing that the main store root's `server.lock` — a held lock
//! is proof the user's library is hosted RIGHT NOW whatever its ports, and
//! self-hosting then would silently split their library in two.
use crate::{Host, HostConfig};
use makepad_asset_client::ApiEndpoints;
use makepad_asset_store::BlobRefPolicy;
use std::io::{Read, Write};
use std::net::{SocketAddr, TcpStream};
use std::path::{Path, PathBuf};
use std::time::Duration;
/// How the app got its store this run.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum StoreMode {
/// Attached to a store some other process runs (the usual case).
Attached,
/// Hosting the store in this process, on loopback.
Hosting,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum EmbedPolicy {
Never,
Auto,
Always,
}
/// `<PREFIX>_ASSET_EMBED`, `auto` when unset or unrecognised.
pub fn embed_policy(prefix: &str) -> EmbedPolicy {
match std::env::var(format!("{prefix}_ASSET_EMBED"))
.unwrap_or_default()
.trim()
.to_ascii_lowercase()
.as_str()
{
"never" | "off" | "0" => EmbedPolicy::Never,
"always" | "force" | "1" => EmbedPolicy::Always,
_ => EmbedPolicy::Auto,
}
}
/// The checkout this binary was built in — every default root hangs off it.
fn checkout_root() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..")
}
/// Where asset-ui keeps the user's main store.
pub fn main_store_root() -> PathBuf {
checkout_root().join("local/asset-ui/asset-server")
}
/// The ai-content library this checkout generates into — the same default
/// the standalone asset-server and asset-ui use.
fn library_root() -> PathBuf {
checkout_root().join("local/ai_content_library")
}
/// The root the app hosts over when it hosts.
///
/// `<PREFIX>_ASSET_ROOT` pins it. Otherwise the user's MAIN library —
/// asset-ui's store root — whenever it holds a catalog: hosting only happens
/// after the probe found nobody serving, and `AssetServer::start` takes the
/// same `server.lock` the daemon would, so losing the race resolves to
/// "attach instead", never to two servers over one WAL. The private seed
/// root is only for a machine with no main library at all — self-hosting a
/// fresh empty root next to a full library reads as "the app lost my
/// content".
pub fn default_store_root(prefix: &str, seed_dir: &str) -> PathBuf {
if let Ok(root) = std::env::var(format!("{prefix}_ASSET_ROOT")) {
return PathBuf::from(root);
}
let main = main_store_root();
if main.join("catalog.sqlite3").exists() {
return main;
}
checkout_root().join("local").join(seed_dir)
}
/// Does something on the other end of `addr` answer `GET /v1/health` like an
/// Asset Server?
///
/// Deliberately minimal and deliberately SHORT: this runs on the startup
/// path, and its only job is to tell "the user's server is up" from "that
/// port file is stale". A 400 ms budget is generous for loopback and
/// invisible to a human. Anything unexpected reads as "no" — the cost of a
/// false negative is one extra local store; the cost of a false positive is
/// an app that never connects.
fn health_answers(addr: SocketAddr) -> bool {
let Ok(mut stream) = TcpStream::connect_timeout(&addr, Duration::from_millis(400)) else {
return false;
};
let _ = stream.set_read_timeout(Some(Duration::from_millis(400)));
let _ = stream.set_write_timeout(Some(Duration::from_millis(400)));
let req = format!(
"GET /v1/health HTTP/1.1\r\nHost: {}\r\nConnection: close\r\n\r\n",
addr
);
if stream.write_all(req.as_bytes()).is_err() {
return false;
}
let mut buf = [0u8; 256];
let mut got = 0usize;
// One short read is enough: the status line is the first 15 bytes.
while got < 16 {
match stream.read(&mut buf[got..]) {
Ok(0) => break,
Ok(n) => got += n,
Err(_) => break,
}
}
buf[..got].starts_with(b"HTTP/1.1 200")
}
/// Listen briefly for an Asset Server beacon on the LAN discovery port.
///
/// This is the second chance for "a server is running": HTTP ports are
/// ephemeral and `listen` files go stale every launch, but a beacon is
/// live. One beacon period is 2 s, so we wait a little over one.
fn beacon_heard(wait_ms: u64) -> bool {
use makepad_asset_client::discovery::DiscoveryListener;
use makepad_asset_client::util::now_ms;
let Ok(listener) = DiscoveryListener::start(
makepad_asset_client::wire::DEFAULT_DISCOVERY_PORT,
10_000,
now_ms,
) else {
return false;
};
let deadline = std::time::Instant::now() + Duration::from_millis(wait_ms);
while std::time::Instant::now() < deadline {
if !listener.snapshot(now_ms()).is_empty() {
return true;
}
std::thread::sleep(Duration::from_millis(100));
}
false
}
/// Is an external store actually reachable right now?
pub fn external_store_reachable(hinted: Option<ApiEndpoints>) -> bool {
if let Some(endpoints) = hinted {
if health_answers(endpoints.control) {
return true;
}
}
// Ports move every launch; the beacon does not.
if beacon_heard(2_400) {
return true;
}
// THE USER'S MAIN STORE IS ALIVE BUT NOT ANSWERING YET (a succession
// handover, a stale `listen` file, a beacon missed by a hair): the
// lock holder is proof it exists. Self-hosting here would SILENTLY
// put this app on a private empty store — "no content in the grid" —
// so treat a held lock as reachable and let the attach path keep
// discovering; it retries on its own.
main_store_lock_held()
}
/// True when another process holds the main (asset-ui) store's server
/// lock — i.e. the user's library is hosted right now, whatever its ports.
fn main_store_lock_held() -> bool {
let root = main_store_root();
let Ok(file) = std::fs::OpenOptions::new()
.write(true)
.open(root.join("server.lock"))
else {
return false;
};
// The SAME advisory lock the store takes (File::try_lock): if we can
// take it, nobody serves that root; the file drop releases it.
match file.try_lock() {
Ok(()) => {
let _ = file.unlock();
false
}
Err(_) => true,
}
}
/// The in-process host (server + library publisher), held for as long as
/// the app runs.
///
/// Dropping it stops the publisher first and the server last, so the field
/// holding this must be declared AFTER anything that talks to the store —
/// the same drop-order discipline asset-ui's `AssetStore` uses.
pub struct LocalStore {
host: Host,
root: PathBuf,
}
impl LocalStore {
pub fn endpoints(&self) -> ApiEndpoints {
self.host.endpoints()
}
pub fn root(&self) -> &Path {
&self.root
}
pub fn control_addr(&self) -> SocketAddr {
self.host.endpoints().control
}
pub fn data_addr(&self) -> SocketAddr {
self.host.endpoints().data
}
pub fn server_id(&self) -> [u8; 16] {
self.host.server_id()
}
pub fn token(&self) -> &str {
self.host.token()
}
pub fn publisher_running(&self) -> bool {
self.host.publisher_running()
}
}
/// What [`resolve`] decided, and why — the `note` belongs in the app's
/// status line, because "which store am I on" is the first thing that
/// matters when content does not show up.
pub struct Resolved {
pub mode: StoreMode,
/// The embedded host, when hosting. The caller owns it for the life of
/// the app and points its client at [`LocalStore::endpoints`] with
/// [`LocalStore::server_id`] and [`LocalStore::token`].
pub local: Option<LocalStore>,
pub note: String,
}
/// Decide between attaching and hosting.
///
/// `pinned` is the caller saying "an explicit server was named" — naming a
/// server is not something you do by accident, so it always means attach.
/// `hinted` is where the caller last knew a server to live, for the health
/// probe.
pub fn resolve(
prefix: &str,
seed_dir: &str,
pinned: bool,
hinted: Option<ApiEndpoints>,
) -> Resolved {
let policy = embed_policy(prefix);
if pinned || policy == EmbedPolicy::Never {
let note = if pinned {
format!("asset server pinned by {prefix}_ASSET_SERVER")
} else {
format!("attach-only ({prefix}_ASSET_EMBED=never)")
};
return Resolved { mode: StoreMode::Attached, local: None, note };
}
if policy == EmbedPolicy::Auto && external_store_reachable(hinted) {
return Resolved {
mode: StoreMode::Attached,
local: None,
note: "attached to the running asset server".to_string(),
};
}
let root = default_store_root(prefix, seed_dir);
match host_at(&root, prefix) {
Ok(local) => {
let note = format!(
"local store on {} over {} · library publisher {}",
local.control_addr(),
root.file_name().and_then(|n| n.to_str()).unwrap_or("store"),
if local.publisher_running() { "on" } else { "off" }
);
Resolved { mode: StoreMode::Hosting, local: Some(local), note }
}
Err(error) => {
// Hosting failed — most often because another process already
// holds this root's lock. Say so and fall back to the attach
// path, which keeps retrying discovery on its own.
Resolved {
mode: StoreMode::Attached,
local: None,
note: format!("local store unavailable ({error}); attaching instead"),
}
}
}
}
/// Bring up the in-process host on loopback.
///
/// This is the standalone asset-server's own [`Host`] — catalog + CAS plus
/// the ai-content LIBRARY PUBLISHER, so a self-hosted app sees the same
/// `local/ai_content_library` rows asset-ui and the standalone server
/// publish, not a bare seed store. Two deployment defaults are overridden,
/// deliberately, and both stay: LOOPBACK ONLY, and NO discovery beacon.
fn host_at(root: &Path, prefix: &str) -> Result<LocalStore, String> {
std::fs::create_dir_all(root).map_err(|e| format!("create store root: {e}"))?;
let port: u16 = std::env::var(format!("{prefix}_ASSET_PORT"))
.ok()
.and_then(|p| p.trim().parse().ok())
.unwrap_or(0);
let mut cfg = HostConfig::new(root.to_path_buf());
cfg.control_addr = SocketAddr::from(([127, 0, 0, 1], port));
// The data plane always takes an OS-assigned port: pinning one would
// only create a second thing to collide.
cfg.data_addr = SocketAddr::from(([127, 0, 0, 1], 0));
cfg.beacon = false;
// The user's generation library, when this checkout has one: the
// publisher keeps turning it into catalog rows exactly as the
// standalone server would.
let library = library_root();
cfg.library = library.join("index.json").exists().then_some(library);
// Reference imports ON: the whole point of the local mode is pointing
// the store at content that stays where it is. Loopback-only and no
// prefix restriction, which is exactly the privilege this process
// already has over the user's own files.
cfg.blob_refs = BlobRefPolicy::local_host();
cfg.log = true;
let host = Host::start(&cfg).map_err(|e| format!("{e}"))?;
if host.token().is_empty() {
return Err("admin token file empty".to_string());
}
Ok(LocalStore { host, root: root.to_path_buf() })
}

View file

@ -6,39 +6,31 @@
//! and headless workers are all clients of ONE catalog, and they come and go
//! independently. When the catalog lives inside one of those apps, that
//! app's lifetime becomes everybody's lifetime: closing the Asset UI window
//! takes the store, the chat broker and the events hub down under every
//! other connected client, which they see as `503 state unavailable`
//! mid-session. This binary breaks that coupling — the server outlives every
//! window.
//! takes the store and the events hub down under every other connected
//! client, which they see as `503 state unavailable` mid-session. This
//! binary breaks that coupling — the server outlives every window.
//!
//! # What it carries
//!
//! [`Host::start`] composes three things that a fleet of clients needs, all
//! of them existing, tested code:
//! [`Host::start`] composes two things, both existing, tested code:
//!
//! 1. [`makepad_asset_store::AssetServer`] — catalog + CAS over the control
//! and data planes, the chat broker (including client-executed tool
//! parking for game sessions), the games publish path, the committed
//! events hub, the job queue and worker/lease protocol, the lease + blob
//! GC janitor, and the LAN discovery beacon.
//! and data planes, the games publish path, the committed events hub,
//! game rooms, the blob GC janitor, and the LAN discovery beacon.
//! 2. The **library publisher** (`makepad_asset_importer::watch`) — whatever
//! the generation pipelines write into the ai-content library becomes
//! catalog rows. Headless, so it belongs beside the server rather than
//! inside a UI.
//! 3. The **fleet job coordinator** (`makepad_asset_importer::gen_service`)
//! — claims queued generation jobs, dispatches them to the asset-ai GPU
//! boxes the LAN announces, publishes the verified results, and
//! advertises what the fleet can execute right now on
//! `GET /v1/job-profiles`. Without it, jobs any client enqueues sit at
//! "waiting for agent" forever.
//!
//! # What deliberately stays client-side
//!
//! Loops that DERIVE content using resources only a UI process has — the
//! offscreen thumbnail renders (`Cx`, a GPU surface, the splat/mesh
//! viewers), the classic-game import wizards, the stems/lyrics analysis
//! bake — stay in the app and reach the catalog as ordinary clients. Moving
//! them here would mean giving a headless daemon a window.
//! Everything that CREATES content (aicore: "the store stores, the client
//! creates"). Generation runs in the creating apps over their own ai-hub
//! fleet connections (`makepad-asset-creator`), chat sessions live in-app,
//! and loops that derive content with resources only a UI process has — the
//! offscreen thumbnail renders, the classic-game import wizards, the
//! stems/lyrics analysis bake — stay in the app and reach the catalog as
//! ordinary clients.
//!
//! # Single-owner laws
//!
@ -48,23 +40,19 @@
//! let it attach (see the README).
//! - An `AssetClient` cache root is single-owner too, so each loop gets its
//! own child of the work root.
//! - The job coordinator is at most ONE per process (its stop flag is a
//! `'static`, borrowed by the service for the thread's whole life). A
//! second [`Host`] with `jobs` enabled in the same process refuses the
//! coordinator and says so in [`Host::jobs_error`] rather than starting a
//! second claimer that would fight the first for leases.
pub mod embed;
use makepad_asset_client::{ApiEndpoints, AssetClient, ClientConfig, PublishRights};
use makepad_asset_store::{AssetServer, DiscoveryConfig, ServerConfig};
use makepad_asset_store::{AssetServer, BlobRefPolicy, DiscoveryConfig, ServerConfig};
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread::JoinHandle;
/// Default namespace the coordinator advertises and publishes into. Same
/// value the Asset UI's in-process coordinator uses, so a job enqueued
/// against either host is routed identically.
/// Default namespace the library publisher publishes into. Same value the
/// creator apps publish with, so rows from either path sit side by side.
pub const DEFAULT_NAMESPACE: &str = "gen";
/// Everything the host needs. Every knob is a named field: nothing here is
@ -85,37 +73,21 @@ pub struct HostConfig {
pub beacon: bool,
/// ai-content library to publish continuously. `None` = no publisher.
pub library: Option<PathBuf>,
/// Run the fleet job coordinator.
pub jobs: bool,
/// Let the coordinator advertise the fleet's executable job profiles.
/// Ignored when `jobs` is false.
pub announce: bool,
/// Explicit GPU-box URL list; `None` = LAN discovery.
pub fleet_file: Option<PathBuf>,
/// Namespace for published rows and advertised profiles.
/// Namespace for published rows.
pub namespace: String,
/// Parent for the loops' single-owner client caches. Defaults to the
/// server root's parent, which puts them exactly where the Asset UI's
/// own hosting mode puts them.
pub work_root: PathBuf,
/// Chat: local Qwen fleet node base URLs. Empty = LAN fleet discovery,
/// same as the Asset UI's embedded broker.
pub chat_fleet_bases: Vec<String>,
/// Named fleet the chat broker talks to. Empty = `default`.
pub chat_fleet: String,
/// Live chat sessions this server will hold at once, and how many any
/// one principal may hold. A box that serves several parallel chat
/// slots wants these raised; the defaults (32 / 8) match the library.
/// 0 = keep the library default.
pub chat_max_sessions: usize,
pub chat_max_sessions_per_owner: usize,
/// Reference-import policy handed to the server. The deployment default
/// (owned blobs only); a loopback-only embedder may open this up.
pub blob_refs: BlobRefPolicy,
/// Log to stderr.
pub log: bool,
}
impl HostConfig {
/// Deployment defaults: ephemeral planes on every interface, beacon on,
/// both background loops on, LAN fleet.
/// Deployment defaults: ephemeral planes on every interface, beacon on.
pub fn new(root: PathBuf) -> Self {
let work_root = root
.parent()
@ -127,81 +99,49 @@ impl HostConfig {
data_addr: SocketAddr::from(([0, 0, 0, 0], 0)),
beacon: true,
library: None,
jobs: true,
announce: true,
fleet_file: None,
namespace: DEFAULT_NAMESPACE.to_string(),
work_root,
chat_fleet_bases: Vec::new(),
chat_fleet: String::new(),
chat_max_sessions: 0,
chat_max_sessions_per_owner: 0,
blob_refs: BlobRefPolicy::default(),
log: true,
}
}
}
/// A running host. Dropping it (or calling [`Host::shutdown`]) stops the
/// loops first and the server last, so nothing is still publishing into a
/// catalog that is closing.
/// publisher first and the server last, so nothing is still publishing into
/// a catalog that is closing.
pub struct Host {
// Declaration order IS drop order: both loops are joined while the
// server they talk to is still answering.
// Declaration order IS drop order: the loop is joined while the server
// it talks to is still answering.
publish: Option<BackgroundLoop>,
jobs: Option<BackgroundLoop>,
server: Option<AssetServer>,
endpoints: ApiEndpoints,
server_id: [u8; 16],
token: String,
library_error: Option<String>,
jobs_error: Option<String>,
}
/// One owned background thread plus the flag that stops it.
struct BackgroundLoop {
stop: Stop,
stop: Arc<AtomicBool>,
join: Option<JoinHandle<()>>,
}
/// A loop's stop flag. The library watcher takes `&AtomicBool` and can own a
/// per-host `Arc`; the generation service borrows a `&'static` for the
/// thread's whole life, so there is exactly one of those per process.
enum Stop {
Owned(Arc<AtomicBool>),
Static(&'static AtomicBool),
}
impl Stop {
fn raise(&self) {
match self {
Stop::Owned(flag) => flag.store(true, Ordering::Release),
Stop::Static(flag) => flag.store(true, Ordering::Release),
}
}
}
impl Drop for BackgroundLoop {
fn drop(&mut self) {
self.stop.raise();
self.stop.store(true, Ordering::Release);
if let Some(join) = self.join.take() {
let _ = join.join();
}
}
}
/// Stop flag for the one in-process job coordinator; see the module header.
static JOBS_STOP: AtomicBool = AtomicBool::new(false);
/// Set while a coordinator is live, so a second one is refused instead of
/// silently competing for the same leases.
static JOBS_RUNNING: AtomicBool = AtomicBool::new(false);
impl Host {
/// Bring the server up, then the background loops.
/// Bring the server up, then the publisher.
///
/// The SERVER is the only thing whose failure is fatal — a loop that
/// cannot start is reported ([`Host::library_error`],
/// [`Host::jobs_error`]) and logged, never a reason to deny every client
/// its catalog.
/// The SERVER is the only thing whose failure is fatal — a publisher
/// that cannot start is reported ([`Host::library_error`]) and logged,
/// never a reason to deny every client its catalog.
pub fn start(config: &HostConfig) -> Result<Host, String> {
let mut cfg = ServerConfig::new(config.root.clone());
cfg.control_addr = config.control_addr;
@ -211,14 +151,7 @@ impl Host {
// root the same way it does against an Asset-UI-hosted server.
cfg.bootstrap_admin = true;
cfg.discovery = config.beacon.then(DiscoveryConfig::lan_default);
cfg.chat.fleet_bases = config.chat_fleet_bases.clone();
cfg.chat.fleet = config.chat_fleet.clone();
if config.chat_max_sessions > 0 {
cfg.chat.max_sessions = config.chat_max_sessions;
}
if config.chat_max_sessions_per_owner > 0 {
cfg.chat.max_sessions_per_owner = config.chat_max_sessions_per_owner;
}
cfg.blob_refs = config.blob_refs.clone();
cfg.log = config.log;
let server = AssetServer::start(cfg).map_err(|error| format!("asset server: {error}"))?;
@ -242,37 +175,23 @@ impl Host {
}
},
};
let (jobs, jobs_error) = if config.jobs {
match start_coordinator(config, endpoints, server_id, &token) {
Ok(handle) => (Some(handle), None),
Err(error) => {
log(config.log, &format!("job coordinator: {error}"));
(None, Some(error))
}
}
} else {
(None, None)
};
log(
config.log,
&format!(
"asset host up: control {} data {} · publisher {} · coordinator {}",
"asset host up: control {} data {} · publisher {}",
endpoints.control,
endpoints.data,
publish.as_ref().map_or("off", |_| "on"),
jobs.as_ref().map_or("off", |_| "on"),
),
);
Ok(Host {
publish,
jobs,
server: Some(server),
endpoints,
server_id,
token,
library_error,
jobs_error,
})
}
@ -295,24 +214,14 @@ impl Host {
self.publish.is_some()
}
pub fn coordinator_running(&self) -> bool {
self.jobs.is_some()
}
/// Why the library publisher is not running, when it was asked for.
pub fn library_error(&self) -> Option<&str> {
self.library_error.as_deref()
}
/// Why the job coordinator is not running, when it was asked for.
pub fn jobs_error(&self) -> Option<&str> {
self.jobs_error.as_deref()
}
/// Stop the loops, then the server. Idempotent; also runs on drop.
/// Stop the publisher, then the server. Idempotent; also runs on drop.
pub fn shutdown(&mut self) {
drop(self.publish.take());
drop(self.jobs.take());
if let Some(mut server) = self.server.take() {
server.shutdown();
}
@ -379,64 +288,7 @@ fn start_publisher(
log(log_enabled, "library publisher: stopped");
})
.map_err(|error| format!("cannot spawn the publisher thread: {error}"))?;
Ok(BackgroundLoop {
stop: Stop::Owned(stop),
join: Some(join),
})
}
/// Claim queued generation jobs and dispatch them to the GPU fleet.
fn start_coordinator(
config: &HostConfig,
endpoints: ApiEndpoints,
server_id: [u8; 16],
token: &str,
) -> Result<BackgroundLoop, String> {
use makepad_asset_importer::gen_service::{FleetSource, GenServiceConfig};
if JOBS_RUNNING.swap(true, Ordering::AcqRel) {
return Err("a job coordinator is already running in this process".to_string());
}
JOBS_STOP.store(false, Ordering::Release);
let service = GenServiceConfig {
servers: vec![endpoints],
server_id: Some(server_id),
token: token.to_string(),
cache_root: config.work_root.join("jobs-cache"),
namespace: config.namespace.clone(),
suffix: "asset-host".to_string(),
rights: PublishRights::generated_cc0(),
fleet: match &config.fleet_file {
Some(path) => FleetSource::File(path.clone()),
None => FleetSource::Lan,
},
announce: config.announce,
log: config.log,
};
let log_enabled = config.log;
let join = std::thread::Builder::new()
.name("asset-host-jobs".to_string())
.spawn(move || {
log(
log_enabled,
&format!(
"job coordinator: {} -> the GPU fleet",
service.servers[0].control
),
);
makepad_asset_importer::gen_service::run(&service, &JOBS_STOP);
log(log_enabled, "job coordinator: stopped");
JOBS_RUNNING.store(false, Ordering::Release);
});
match join {
Ok(join) => Ok(BackgroundLoop {
stop: Stop::Static(&JOBS_STOP),
join: Some(join),
}),
Err(error) => {
JOBS_RUNNING.store(false, Ordering::Release);
Err(format!("cannot spawn the coordinator thread: {error}"))
}
}
Ok(BackgroundLoop { stop, join: Some(join) })
}
fn log(enabled: bool, message: &str) {
@ -462,15 +314,13 @@ mod tests {
}
/// An isolated host: loopback-only ephemeral planes, NO beacon (a test
/// must never advertise itself to the operator's LAN), no coordinator
/// (it would claim the real fleet's jobs).
/// must never advertise itself to the operator's LAN).
fn isolated(name: &str) -> HostConfig {
let base = test_root(name);
let mut config = HostConfig::new(base.join("server"));
config.control_addr = "127.0.0.1:0".parse().unwrap();
config.data_addr = "127.0.0.1:0".parse().unwrap();
config.beacon = false;
config.jobs = false;
config.log = false;
config.work_root = base.join("work");
config
@ -551,7 +401,6 @@ mod tests {
fn the_work_root_defaults_beside_the_server_root() {
let config = HostConfig::new(PathBuf::from("/store/local/asset-ui/asset-server"));
assert_eq!(config.work_root, PathBuf::from("/store/local/asset-ui"));
assert!(config.jobs, "a fleet host coordinates jobs by default");
assert!(config.beacon, "a fleet host is discoverable by default");
assert_eq!(config.namespace, DEFAULT_NAMESPACE);
assert_eq!(

View file

@ -3,8 +3,8 @@
//! Parse flags into a [`HostConfig`], start the host, and wait for
//! SIGINT/SIGTERM to shut it down cleanly. With no flags at all it serves the
//! checkout's standard store root on ephemeral ports, announces itself on the
//! LAN, publishes the ai-content library, and coordinates fleet jobs — i.e.
//! everything the Asset UI's embedded mode does, minus the window.
//! LAN, and publishes the ai-content library — i.e. everything the Asset
//! UI's embedded mode does, minus the window.
//!
//! See `README.md` beside this file for the deployment runbook.
@ -17,10 +17,11 @@ use std::time::Duration;
const USAGE: &str = "\
makepad-asset-server [options]
The standalone, multiplayer-first Asset Server: catalog + CAS, chat broker,
events hub, job queue, LAN beacon, plus the ai-content library publisher and
the GPU-fleet job coordinator. Clients (asset-ui, vj, sandbox, workers)
attach to it and may come and go without ever taking the store down.
The standalone, multiplayer-first Asset Server: catalog + CAS, events hub,
game rooms, LAN beacon, plus the ai-content library publisher. Clients
(asset-ui, vj, sandbox) attach to it and may come and go without ever
taking the store down. Generation runs in the creating apps over their own
fleet connections the store stores.
Options:
--root <dir> Server root. Default: $AI_CONTENT_ASSET_ROOT, else
@ -32,21 +33,9 @@ Options:
--library <dir> ai-content library to publish continuously.
Default: <checkout>/local/ai_content_library
--no-library Do not run the library publisher
--no-jobs Do not run the fleet job coordinator
--no-announce Coordinate jobs, but do not advertise the fleet's
executable profiles on GET /v1/job-profiles
--fleet <path> GPU-box URL list (default: LAN discovery)
--namespace <ns> Namespace for published rows (default gen)
--work <dir> Parent for the loops' client caches
(default: the server root's parent)
--chat-fleet <url> Local Qwen fleet node for the chat broker
(repeatable; default: LAN fleet discovery)
--chat-fleet-name <n> Named fleet the chat broker talks to
--chat-max-sessions <n> Live chat sessions this server holds at once
(default 32)
--chat-max-sessions-per-owner <n>
Live chat sessions one principal may hold
(default 8)
--quiet No stderr logging
--help This text
";
@ -105,15 +94,8 @@ fn parse_config() -> HostConfig {
let mut beacon = true;
let mut library: Option<PathBuf> = None;
let mut no_library = false;
let mut jobs = true;
let mut announce = true;
let mut fleet_file: Option<PathBuf> = None;
let mut namespace: Option<String> = None;
let mut work: Option<PathBuf> = None;
let mut chat_fleet_bases: Vec<String> = Vec::new();
let mut chat_fleet = String::new();
let mut chat_max_sessions = 0usize;
let mut chat_max_sessions_per_owner = 0usize;
let mut log = true;
let value_of = |name: &str, args: &mut dyn Iterator<Item = String>| -> String {
@ -136,37 +118,8 @@ fn parse_config() -> HostConfig {
"--no-beacon" => beacon = false,
"--library" => library = Some(PathBuf::from(value_of("--library", &mut args))),
"--no-library" => no_library = true,
"--no-jobs" => jobs = false,
"--no-announce" => announce = false,
"--fleet" => fleet_file = Some(PathBuf::from(value_of("--fleet", &mut args))),
"--namespace" => namespace = Some(value_of("--namespace", &mut args)),
"--work" => work = Some(PathBuf::from(value_of("--work", &mut args))),
"--chat-fleet" => {
let value = value_of("--chat-fleet", &mut args);
if value.is_empty() {
fail("malformed --chat-fleet");
}
chat_fleet_bases.push(value);
}
"--chat-fleet-name" => chat_fleet = value_of("--chat-fleet-name", &mut args),
"--chat-max-sessions" => {
let value = value_of("--chat-max-sessions", &mut args);
chat_max_sessions = value
.parse()
.unwrap_or_else(|_| fail("malformed --chat-max-sessions"));
if chat_max_sessions == 0 {
fail("--chat-max-sessions must be at least 1");
}
}
"--chat-max-sessions-per-owner" => {
let value = value_of("--chat-max-sessions-per-owner", &mut args);
chat_max_sessions_per_owner = value
.parse()
.unwrap_or_else(|_| fail("malformed --chat-max-sessions-per-owner"));
if chat_max_sessions_per_owner == 0 {
fail("--chat-max-sessions-per-owner must be at least 1");
}
}
"--quiet" => log = false,
"--help" | "-h" => {
println!("{USAGE}");
@ -192,9 +145,6 @@ fn parse_config() -> HostConfig {
} else {
Some(library.unwrap_or_else(|| checkout_root().join("local/ai_content_library")))
};
config.jobs = jobs;
config.announce = announce;
config.fleet_file = fleet_file;
if let Some(namespace) = namespace {
if namespace.is_empty() {
fail("--namespace needs a value");
@ -204,10 +154,6 @@ fn parse_config() -> HostConfig {
if let Some(work) = work {
config.work_root = work;
}
config.chat_fleet_bases = chat_fleet_bases;
config.chat_fleet = chat_fleet;
config.chat_max_sessions = chat_max_sessions;
config.chat_max_sessions_per_owner = chat_max_sessions_per_owner;
config.log = log;
config
}

View file

@ -9,7 +9,7 @@ license = "MIT OR Apache-2.0"
makepad-widgets = { path = "../../widgets" }
# Service wire types + fleet/affinity scheduler (transport here is
# cx.http_request; the lib's blocking client is for worker-thread consumers).
makepad-asset-ai = { path = "../../libs/asset/ai", default-features = false }
makepad-ai-hub = { path = "../../libs/ai/hub", default-features = false }
makepad-micro-serde = { path = "../../libs/micro_serde" }
makepad-base64 = { path = "../../libs/base64" }
# Mesh viewer: Renderer stage for a single GLB.
@ -23,6 +23,7 @@ makepad-xr = { path = "../../xr" }
# retry), catalog runtimes, committed-event subscriber, verified cache. The
# VJ worker owns these crates; this app consumes the public API only.
makepad-asset-client = { path = "../../libs/asset/client" }
makepad-asset-creator = { path = "../../libs/asset/creator" }
# Shared preview/viewer widgets: the pool every host draws catalog content
# with (VJ and the DJ surface adopt the same set).
# The renderer feature brings the 3D faces (mesh turntable, world walker).
@ -33,13 +34,18 @@ makepad-asset-chat = { path = "../../libs/asset/chat" }
# broker session + its worker channel, the transcript with its tool chips
# and rate meter, and the list widget. This app supplies the personality
# (namespace `gen`, client profile `gen`, its own client-executed tools).
makepad-asset-chat-ui = { path = "../../libs/asset/chat_ui" }
makepad-chat-ui = { path = "../../libs/chat_ui" }
# Embedded catalog/chat broker for now: the app starts the real Asset
# Server in-process instead of waiting for a LAN beacon.
makepad-asset-store = { path = "../../libs/asset/store" }
# Local licensed-pack compiler (Kenney Import). Same fail-closed walk as
# `makepad-asset-importer --import-pack`; no network fetch.
makepad-asset-importer = { path = "../../libs/asset/importer" }
# The vision-annotation pass, for its VERSION only: this app draws the
# annotation bar and must agree with the store on which annotator version
# the catalog owes. The work itself is `annotate.asset`, a vision job the
# fleet coordinator claims — nothing here runs a model.
makepad-asset-annotate = { path = "../../libs/asset/annotate" }
# The library is 241 MP3s and 4 WAVs: the transport has to be able to play
# what the catalog actually holds, and the container has to be knowable from
# the bytes when a digest-named cache object has no name to read. Zero

View file

@ -911,7 +911,7 @@ mod tests {
// pixels + its cache-source key; garbage comes back as an honest
// None (the app pins a badge instead of retrying forever).
let sidecar = dir.join("lib-1.glb.thumb");
let png = makepad_asset_ai::testpattern::encode_png_rgba(
let png = makepad_ai_hub::testpattern::encode_png_rgba(
&[255u8; 6 * 4 * 4],
6,
4,
@ -996,7 +996,7 @@ mod tests {
let y = (i / 1024) as u8;
px.copy_from_slice(&[x, y, 128, 255]);
}
let png = makepad_asset_ai::testpattern::encode_png_rgba(&rgba, 1024, 1024).unwrap();
let png = makepad_ai_hub::testpattern::encode_png_rgba(&rgba, 1024, 1024).unwrap();
std::fs::write(&render, &png).unwrap();
io.request(IoRequest {
file: "lib-flux.png".into(),
@ -1066,7 +1066,7 @@ mod tests {
let v = if y < 64 { 200 } else { 20 };
px.copy_from_slice(&[v, v, v, 255]);
}
let png = makepad_asset_ai::testpattern::encode_png_rgba(&rgba, 512, 128).unwrap();
let png = makepad_ai_hub::testpattern::encode_png_rgba(&rgba, 512, 128).unwrap();
std::fs::write(&composite, &png).unwrap();
let fft_views = vec![
ThumbnailView {

View file

@ -50,8 +50,8 @@
use makepad_asset_client::{
ApiEndpoints, AssetDetailDto, CatalogEventDto, CatalogFacet, CatalogHit, CatalogQuery,
CatalogSubscriptionEvent, ClientError, ClientEvent, ClientOutput, ClientRequest, GcRequest,
GcStatusDto, JobProfileDto, PageCursor, RequestId, RetireDto, SessionConfig, SessionConnector,
SessionHandles, SessionMsg, SessionStatus,
GcStatusDto, JobProfileDto, PageCursor, PipelineId, RequestId, RetireDto, SessionConfig,
SessionConnector, SessionHandles, SessionMsg, SessionStatus,
};
use makepad_asset_data::{AssetId, AssetRevisionId};
pub use makepad_asset_data::AssetKind;
@ -180,6 +180,8 @@ pub fn server_kind_label(kind: AssetKind) -> &'static str {
AssetKind::Billboard => "billboard",
AssetKind::Game => "game",
AssetKind::VjEffect => "vjeffect",
AssetKind::Data => "data",
AssetKind::ModelProgram => "model-program",
}
}
@ -362,10 +364,6 @@ pub struct AssetStore {
/// `embedded` so it is joined while the server it publishes into is
/// still alive.
publish: Option<PublishLoop>,
/// In-process job coordinator: claims generation jobs queued on the
/// hosted server (the VJ's GEN tab, chat) and dispatches them to the
/// LAN fleet. Without it those jobs sit at "waiting for agent" forever.
jobs: Option<JobLoop>,
/// LIVECODING: observed origin directories → catalog, no copy. Declared
/// BEFORE `embedded` for the same reason `publish` is: joined while the
/// server it publishes into is still alive.
@ -395,10 +393,11 @@ pub struct AssetStore {
pub selected: Option<AssetId>,
pub detail: Remote<AssetDetailDto>,
detail_req: Option<RequestId>,
/// Advertised generation capabilities (`/v1/jobs/profiles`) — the REAL
/// server-side generation surface for the Runs panel.
/// Generation capabilities of the LIVE LAN fleet, built by probing the
/// boxes directly (the store advertises nothing any more — generation
/// is client-driven, aicore §9).
pub profiles: Remote<Vec<JobProfileDto>>,
profiles_req: Option<RequestId>,
profiles_rx: Option<std::sync::mpsc::Receiver<Vec<JobProfileDto>>>,
/// Committed catalog events, newest first, capped.
pub events: VecDeque<CatalogEventDto>,
/// The event feed delivered its initial cursor and is following commits.
@ -410,6 +409,12 @@ pub struct AssetStore {
/// drains this to re-open what it is showing: a new revision means a new
/// blob digest, so re-resolving is the whole of "stay current".
changed_assets: Vec<AssetId>,
/// Declared runs the server announced as OVER since the app last looked
/// (`pipeline.finished`). This is the only honest end-of-run signal: a
/// publish is per-asset and coincidental, and a run that fails publishes
/// nothing at all. Drained by the RUNS chip, which re-reads on it
/// instead of waiting out its poll interval.
finished_pipelines: Vec<PipelineId>,
/// In-flight `RetireAsset`/`RetireRevision` requests, tracked only to
/// surface a failure (or a mismatched output) honestly — success is
/// applied locally via [`AssetStore::on_retired`] the moment the
@ -649,6 +654,22 @@ impl AssetStore {
}
}
}
// Fleet-built generation profiles landing from their worker thread.
if let Some(rx) = &self.profiles_rx {
match rx.try_recv() {
Ok(profiles) => {
self.profiles_rx = None;
self.profiles = Remote::Ready(profiles);
changed = true;
}
Err(std::sync::mpsc::TryRecvError::Empty) => {}
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
self.profiles_rx = None;
self.profiles = Remote::Failed("fleet probe thread died".to_string());
changed = true;
}
}
}
let mut catalog_events = Vec::new();
let mut feed_events = Vec::new();
if let Some(handles) = &mut self.handles {
@ -787,7 +808,6 @@ impl AssetStore {
if self.host_loops == HostLoops::Run {
self.publish =
start_publish_loop(&server, token, self.library_dir.clone());
self.jobs = start_job_loop(&server, token);
// LIVECODING: observed origin directories, catalogued in place.
// Only the HOST runs this — reference admission is loopback
// privilege, so an attached client never observes for somebody
@ -802,7 +822,7 @@ impl AssetStore {
///
/// The subscriber only reports FAILED polls: a poll that succeeds with
/// nothing to report sends no event at all, so a blip that healed would
/// otherwise look like a death forever. A profiles fetch is a plain
/// otherwise look like a death forever. A GC status fetch is a plain
/// authenticated GET that changes nothing, and its answer — success, or
/// even a refusal, which is still a server talking — clears the loss.
fn submit_probe(&mut self) {
@ -810,10 +830,7 @@ impl AssetStore {
return;
}
let Some(handles) = &mut self.handles else { return };
if let Ok(id) = handles
.catalog
.submit(ClientRequest::FetchJobProfiles { domain: None })
{
if let Ok(id) = handles.catalog.submit(ClientRequest::GcStatus) {
self.probe_req = Some(id);
}
}
@ -839,7 +856,7 @@ impl AssetStore {
self.search_continuation = false;
self.next_cursor = None;
self.detail_req = None;
self.profiles_req = None;
self.profiles_rx = None;
self.probe_req = None;
self.gc_req = None;
self.gc_cancel_req = None;
@ -1006,6 +1023,11 @@ impl AssetStore {
}
/// Take the assets catalog events touched since the last call.
/// Runs the server announced as finished since the last call.
pub fn take_finished_pipelines(&mut self) -> Vec<PipelineId> {
std::mem::take(&mut self.finished_pipelines)
}
pub fn take_changed_assets(&mut self) -> Vec<AssetId> {
std::mem::take(&mut self.changed_assets)
}
@ -1123,18 +1145,23 @@ impl AssetStore {
}
}
/// Build the generation-profile list from the LIVE fleet: probe the
/// boxes the LAN announces and let the shared profile builder say what
/// they can execute right now. Runs on its own thread — LAN probes must
/// never stall a frame — and lands through `profiles_rx` in [`Self::poll`].
fn submit_profiles(&mut self) {
let Some(handles) = &mut self.handles else { return };
match handles
.catalog
.submit(ClientRequest::FetchJobProfiles { domain: None })
{
Ok(id) => {
self.profiles_req = Some(id);
self.profiles = Remote::Loading;
}
Err(error) => self.profiles = Remote::Failed(error.to_string()),
}
let (tx, rx) = std::sync::mpsc::channel();
self.profiles_rx = Some(rx);
self.profiles = Remote::Loading;
let _ = std::thread::Builder::new()
.name("asset-ui-profiles".to_string())
.spawn(move || {
let snapshots = makepad_asset_creator::runner::fleet_snapshots();
let profiles = makepad_asset_importer::gen_profiles::build_profiles(
&snapshots, "gen",
);
let _ = tx.send(profiles);
});
}
fn on_catalog_event(&mut self, event: ClientEvent) -> bool {
@ -1224,8 +1251,6 @@ impl AssetStore {
0
} else if Some(id) == self.detail_req {
1
} else if Some(id) == self.profiles_req {
2
} else {
return false;
};
@ -1272,10 +1297,6 @@ impl AssetStore {
self.detail_req = None;
self.detail = Remote::Ready(detail);
}
(2, ClientOutput::JobProfiles(profiles)) => {
self.profiles_req = None;
self.profiles = Remote::Ready(profiles);
}
// A mismatched output shape for a tracked id is a
// protocol-level surprise — surface it, don't guess.
(0, other) => {
@ -1284,14 +1305,10 @@ impl AssetStore {
self.next_cursor = None;
self.search = Remote::Failed(format!("unexpected output {other:?}"));
}
(1, other) => {
(_, other) => {
self.detail_req = None;
self.detail = Remote::Failed(format!("unexpected output {other:?}"));
}
(_, other) => {
self.profiles_req = None;
self.profiles = Remote::Failed(format!("unexpected output {other:?}"));
}
}
true
}
@ -1306,14 +1323,10 @@ impl AssetStore {
self.next_cursor = None;
self.search = Remote::Failed(error.to_string());
}
1 => {
_ => {
self.detail_req = None;
self.detail = Remote::Failed(error.to_string());
}
_ => {
self.profiles_req = None;
self.profiles = Remote::Failed(error.to_string());
}
}
true
}
@ -1376,6 +1389,11 @@ impl AssetStore {
self.changed_assets.push(asset_id);
}
}
if let Some(pipeline) = event.pipeline {
if !self.finished_pipelines.contains(&pipeline) {
self.finished_pipelines.push(pipeline);
}
}
self.events.push_front(event);
}
self.events.truncate(EVENT_LOG_CAP);
@ -1451,7 +1469,7 @@ fn now_ms() -> u64 {
.unwrap_or(0)
}
fn checkout_root() -> PathBuf {
pub(crate) fn checkout_root() -> PathBuf {
if let Ok(root) = std::env::var("MAKEPAD_ROOT") {
return PathBuf::from(root);
}
@ -1655,76 +1673,6 @@ fn start_embedded_asset_server_at(
Ok((server, token))
}
/// Stop flag for the single in-process job coordinator (see PUBLISH_STOP).
static JOBS_STOP: AtomicBool = AtomicBool::new(false);
/// Owns the job-coordinator thread; dropping the store stops and joins it.
struct JobLoop {
join: Option<std::thread::JoinHandle<()>>,
}
impl Drop for JobLoop {
fn drop(&mut self) {
JOBS_STOP.store(true, Ordering::Release);
if let Some(join) = self.join.take() {
let _ = join.join();
}
}
}
/// Claim + dispatch generation jobs from the hosted server to the fleet the
/// LAN announces (the same boxes the asset-ui's own pipelines use).
///
/// This runs the SHARED generation service, so the embedded server gets the
/// same behaviour as a standalone worker: one claim loop per fleet box (N
/// queued jobs of a kind drain across the N boxes that serve it), every
/// wired kind rather than video alone, and a live advertisement on
/// `GET /v1/job-profiles` of what those boxes can actually execute — which
/// is what stops a client enqueueing a tier whose weights are on no box.
fn start_job_loop(
server: &makepad_asset_store::AssetServer,
token: &str,
) -> Option<JobLoop> {
let endpoints = localized_endpoints(server);
let server_id = server.server_id();
let token = token.to_string();
let cache = asset_ui_home().join("jobs-cache");
JOBS_STOP.store(false, Ordering::Release);
let join = std::thread::Builder::new()
.name("asset-ui-jobs".to_string())
.spawn(move || {
use makepad_asset_importer::gen_service::{FleetSource, GenServiceConfig};
log!(
"job loop: coordinating jobs on {}/{} → LAN fleet",
endpoints.control,
endpoints.data
);
makepad_asset_importer::gen_service::run(
&GenServiceConfig {
servers: vec![endpoints],
server_id: Some(server_id),
token,
cache_root: cache,
namespace: "gen".to_string(),
suffix: "asset-ui".to_string(),
rights: makepad_asset_client::PublishRights::generated_cc0(),
fleet: FleetSource::Lan,
announce: true,
log: true,
},
&JOBS_STOP,
);
log!("job loop: stopped");
});
match join {
Ok(join) => Some(JobLoop { join: Some(join) }),
Err(error) => {
log!("job loop: could not spawn: {error}");
None
}
}
}
/// Stop flag for the single in-process publish loop. A `static` (not an
/// `Arc`) because `watch::run` borrows it for the thread's whole life and
/// there is at most one loop per process.
@ -2090,6 +2038,9 @@ mod tests {
game_id: None,
game_revision: None,
alias: Some(format!("game/asset-{seq}")),
model_preview: None,
pipeline: None,
pipeline_state: None,
content_kind: None,
ts_ms: seq,
};
@ -2205,6 +2156,9 @@ mod tests {
game_id: None,
game_revision: None,
alias: None,
model_preview: None,
pipeline: None,
pipeline_state: None,
content_kind: None,
ts_ms: 1,
}]);
@ -2228,6 +2182,9 @@ mod tests {
game_id: None,
game_revision: None,
alias: None,
model_preview: None,
pipeline: None,
pipeline_state: None,
content_kind: None,
ts_ms: 2,
}]);

View file

@ -601,7 +601,7 @@ pub fn waveform_thumbnail_png(pcm: &WavPcm) -> Option<Vec<u8>> {
WAVEFORM_THUMB_W,
WAVEFORM_THUMB_H,
) {
return makepad_asset_ai::testpattern::encode_png_rgba(
return makepad_ai_hub::testpattern::encode_png_rgba(
&rgba,
WAVEFORM_THUMB_W,
WAVEFORM_THUMB_H,
@ -618,7 +618,7 @@ pub fn waveform_thumbnail_png(pcm: &WavPcm) -> Option<Vec<u8>> {
(pixel >> 24) as u8,
]);
}
makepad_asset_ai::testpattern::encode_png_rgba(&rgba, WAVEFORM_THUMB_W, WAVEFORM_THUMB_H)
makepad_ai_hub::testpattern::encode_png_rgba(&rgba, WAVEFORM_THUMB_W, WAVEFORM_THUMB_H)
.ok()
}
@ -675,7 +675,7 @@ mod tests {
// Round-trip against the service's own encoder shape: a 100-sample
// 24kHz mono ramp.
let samples: Vec<f32> = (0..100).map(|i| i as f32 / 100.0 - 0.5).collect();
let wav = makepad_asset_ai::wav::encode_wav_pcm16_mono(&samples, 24_000);
let wav = makepad_ai_hub::wav::encode_wav_pcm16_mono(&samples, 24_000);
let pcm = parse_wav(&wav).unwrap();
assert_eq!(pcm.sample_rate, 24_000);
assert_eq!(pcm.channels, 1);

View file

@ -2,7 +2,7 @@
//!
//! The mechanics — the session, the worker thread on a channel, the
//! transcript with its tool chips and rate meter, cancel and clear — are the
//! shared component in [`makepad_asset_chat_ui`], the same one the game
//! shared component in [`makepad_chat_ui`], the same one the game
//! sandbox runs. This file is what makes it the ASSET UI's chat:
//!
//! - it opens the session as `("gen", "gen")`, so the broker assembles the
@ -22,11 +22,11 @@ use crate::pipeline::{
IMAGE_SIZES, IMAGE_STEPS, MESH_FACE_COUNTS, MESH_TEXTURE_SIZES, MUSIC_DEFAULT_SECONDS, MUSIC_LENGTHS,
VIDEO_LENGTHS, VIDEO_SIZES,
};
use makepad_asset_ai::fleet::BoxSnapshot;
use makepad_ai_hub::fleet::BoxSnapshot;
use makepad_asset_chat::tools::{ContentToolCall, GenerateThen};
use makepad_asset_chat::wire::ToolOutcome;
use makepad_asset_chat_ui::feed::{default_call_title, default_outcome_summary, ellipsis};
use makepad_asset_chat_ui::{ChatFeed, ClientTools, FeedConfig};
use makepad_chat_ui::feed::{default_call_title, default_outcome_summary, ellipsis};
use makepad_chat_ui::{ChatFeed, ClientTools, FeedConfig};
use makepad_asset_client::dto::ChatToolOutcomeDto;
use makepad_asset_client::json::{self, Value};
use makepad_asset_client::{ApiEndpoints, ChatAttachment};
@ -36,10 +36,10 @@ use std::sync::{Arc, Mutex};
/// The transcript and its rate meter are the shared component's; this app
/// only reads them.
pub use makepad_asset_chat_ui::{ChatData, ChatRole};
pub use makepad_chat_ui::{ChatData, ChatRole};
// Test-only: the module tests below read the shared transcript directly.
#[cfg(test)]
use makepad_asset_chat_ui::CHAT;
use makepad_chat_ui::CHAT;
// ---------------------------------------------------------------------------
// mutable generation defaults
@ -452,6 +452,10 @@ pub struct ChatJob {
pub video_steps: u32,
pub seconds: u32,
pub voice: Option<String>,
/// The sung words for a music job, as their own field — an empty one is
/// what makes a music model generate an instrumental.
pub lyrics: Option<String>,
pub seed: Option<u64>,
}
// ---------------------------------------------------------------------------
@ -644,6 +648,17 @@ impl AppTools {
}
ChatJobKind::Music => {
pairs.push(("seconds", Value::Int(job.seconds as i64)));
// The lyric script is its own field all the way down: the
// caption never carries the sung words.
if let Some(text) = &job.lyrics {
pairs.push(("lyrics", json::s(text.clone())));
}
if let Some(steps) = job.steps {
pairs.push(("steps", Value::Int(steps as i64)));
}
if let Some(seed) = job.seed {
pairs.push(("seed", Value::Int(seed as i64)));
}
}
ChatJobKind::Speech => {
if let Some(v) = &job.voice {
@ -685,6 +700,8 @@ impl AppTools {
video_steps: VIDEO_LENGTHS[0].1,
seconds: MUSIC_DEFAULT_SECONDS,
voice: None,
lyrics: None,
seed: None,
})
}
@ -742,6 +759,8 @@ impl AppTools {
video_steps: steps.unwrap_or(VIDEO_LENGTHS[0].1),
seconds: MUSIC_DEFAULT_SECONDS,
voice: None,
lyrics: None,
seed: None,
}),
ContentToolCall::AudioGenerate { prompt, model } => self.queue_job(ChatJob {
prompt,
@ -755,6 +774,8 @@ impl AppTools {
video_steps: 0,
seconds: 0,
voice: None,
lyrics: None,
seed: None,
}),
ContentToolCall::SpeechGenerate { prompt, model, voice } => self.queue_job(ChatJob {
prompt,
@ -768,20 +789,26 @@ impl AppTools {
video_steps: 0,
seconds: 0,
voice,
lyrics: None,
seed: None,
}),
ContentToolCall::MusicGenerate { prompt, model, seconds } => self.queue_job(ChatJob {
prompt,
kind: ChatJobKind::Music,
then: GenerateThen::None,
model,
width: 0,
height: 0,
steps: None,
frames: 0,
video_steps: 0,
seconds: seconds.unwrap_or(MUSIC_DEFAULT_SECONDS),
voice: None,
}),
ContentToolCall::MusicGenerate { prompt, model, seconds, lyrics, steps, seed } => {
self.queue_job(ChatJob {
prompt,
kind: ChatJobKind::Music,
then: GenerateThen::None,
model,
width: 0,
height: 0,
steps,
frames: 0,
video_steps: 0,
seconds: seconds.unwrap_or(MUSIC_DEFAULT_SECONDS),
voice: None,
lyrics,
seed,
})
}
ContentToolCall::MeshGenerate { prompt, model, width, height, steps } => {
match self.image_job(
ChatJobKind::Mesh,
@ -907,12 +934,12 @@ impl ClientTools for AppTools {
#[cfg(test)]
mod tests {
use super::*;
use makepad_asset_ai::protocol::{HealthJson, ModelInfoJson};
use makepad_ai_hub::protocol::{HealthJson, ModelInfoJson};
fn snap(url: &str, domain: &str, id: &str, state: &str) -> BoxSnapshot {
BoxSnapshot {
base_url: url.into(),
health: Some(HealthJson {
health: Some(HealthJson { realtime: None,
service: "makepad-asset-ai".into(),
version: "t".into(),
gpu: Some("RTX".into()),

View file

@ -1,34 +1,15 @@
//! Post-process pass: walk the local library and fill missing metadata.
//! Post-process pass: rewrite path-like library labels into catalog titles
//! (Doom lump → Imp, Quake MDL → Shambler). No model, no invented lore.
//!
//! Two layers, both fail-closed:
//! 1. Catalog names (Doom lump → Imp, Quake MDL → Shambler). No model.
//! 2. Optional vision caption into `<file>.vision.json` when a Qwen-VL
//! (or any fleet model with `vl` in the id) is actually ready. Until
//! that box is provisioned the pass records `skipped` and does not
//! invent descriptions.
//! This file also held a stub "vision caption" layer that wrote
//! `<file>.vision.json` sidecars saying `skipped`: nothing ever called it
//! and nothing ever read one. The real vision pass is
//! [`crate::annotate_queue`], which writes descriptions into the STORE,
//! where search can index them — a sidecar beside a local GLB was never
//! reachable by a catalog query.
use crate::library::{Library, LibraryMeta};
use makepad_asset_importer::stateful_billboard::{mesh_title, sprite_title, world_title};
use std::path::{Path, PathBuf};
pub const VISION_SIDECAR: &str = "vision.json";
#[derive(Clone, Debug, Default)]
pub struct EnhanceStats {
pub scanned: usize,
pub named: usize,
pub vision_wrote: usize,
pub vision_skipped: usize,
pub message: String,
}
pub fn sidecar_path(library_dir: &Path, file: &str) -> PathBuf {
library_dir.join(format!("{file}.{VISION_SIDECAR}"))
}
pub fn has_vision(library_dir: &Path, file: &str) -> bool {
sidecar_path(library_dir, file).is_file()
}
/// Rewrite path-like labels to catalog titles. Does not invent lore.
pub fn apply_catalog_names(library: &mut Library) -> usize {
@ -127,28 +108,6 @@ fn label_asset_stem(label: &str) -> Option<&str> {
}
}
/// Mark assets that have no vision blob. Does not call a model unless
/// `vision_ready` is true — then the caller supplies captions.
pub fn stamp_skipped_vision(library_dir: &Path, items: &[LibraryMeta], reason: &str) -> usize {
let mut n = 0usize;
for item in items {
let path = sidecar_path(library_dir, &item.file);
if path.is_file() {
continue;
}
let body = format!(
"{{\"status\":\"skipped\",\"reason\":\"{}\",\"label\":\"{}\",\"domain\":\"{}\"}}\n",
reason.replace('"', "'"),
item.label.replace('"', "'"),
item.domain
);
if std::fs::write(&path, body).is_ok() {
n += 1;
}
}
n
}
#[cfg(test)]
mod tests {
use super::*;
@ -206,10 +165,3 @@ mod tests {
assert_eq!(better_label(&duke).as_deref(), Some("TILE-0123"));
}
}
pub fn fleet_has_vision(model_ids: &[String]) -> bool {
model_ids.iter().any(|id| {
let l = id.to_ascii_lowercase();
l.contains("vl") || l.contains("vision") || l.contains("qwen2.5-vl") || l.contains("qwen3-vl")
})
}

View file

@ -202,6 +202,8 @@ pub fn apply_gen(saved: &SavedFastPreset) -> GenParams {
enhance_upscale: saved.enhance_upscale.unwrap_or(2),
enhance_interpolate: saved.enhance_interpolate.unwrap_or(2),
enhance_flow: saved.enhance_flow.unwrap_or(true),
// Loop-ness derives from the preset row at dispatch, never a spec.
video_loop: false,
}
}

View file

@ -1,6 +1,6 @@
//! Fleet discovery over `cx.http_request`: polls `GET /health` +
//! `GET /models` on every fleet endpoint and feeds the parsed JSON into
//! [`makepad_asset_ai::fleet::BoxSnapshot`]s — the scheduler
//! [`makepad_ai_hub::fleet::BoxSnapshot`]s — the scheduler
//! (`fleet::pick_box` / `pick_for_domain`) is pure over those snapshots.
//!
//! Endpoint lifecycle (the part that keeps the fleet free of duplicates):
@ -25,9 +25,9 @@
//! keyed by url, not row index — rows may be coalesced away mid-flight and
//! the late response is then dropped instead of updating the wrong box.
use makepad_asset_ai::discovery::DiscoveredNode;
use makepad_asset_ai::fleet::BoxSnapshot;
use makepad_asset_ai::protocol::{HealthJson, JobStatusJson, JobsJson, LorasJson, ModelsJson};
use makepad_ai_hub::discovery::DiscoveredNode;
use makepad_ai_hub::fleet::BoxSnapshot;
use makepad_ai_hub::protocol::{HealthJson, JobStatusJson, JobsJson, LorasJson, ModelsJson};
use makepad_micro_serde::DeJson;
use makepad_widgets::*;
use std::collections::HashMap;
@ -58,9 +58,47 @@ pub struct FleetPoll {
pub loras: Vec<Vec<String>>,
/// Endpoints with any request in flight (index-parallel to snapshots).
busy: Vec<bool>,
/// Consecutive failed `/health` probes per endpoint. A box is only
/// declared offline after [`OFFLINE_AFTER`] of them: a node denoising a
/// whole GPU or streaming 17 GB of weights answers late, and late is not
/// the same as gone. The panel used to flip such a box to "down" on the
/// first slow answer and back on the next, once a second.
health_fails: Vec<u32>,
/// A failing streak has already been reported for this endpoint, per
/// probe kind (health, models); cleared by that probe's next success.
/// Without it a busy node writes one log line per poll — the log filled
/// with thousands of them while every box in fact answered 200 to curl.
/// The two are separate because a box that answers `/health` and drops
/// `/models` would otherwise clear the latch every round and complain
/// every round.
health_quiet: Vec<bool>,
models_quiet: Vec<bool>,
in_flight: HashMap<LiveId, InFlight>,
}
/// Consecutive `/health` failures before a box is shown as offline.
/// Three misses at the poll cadence is several seconds of real silence, well
/// past any answer a loaded box is merely slow to give.
const OFFLINE_AFTER: u32 = 3;
/// What a failed `/health` probe means for a row.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ProbeVerdict {
/// Keep showing the box exactly as it was; `remaining` more consecutive
/// misses would declare it offline.
Hold { remaining: u32 },
/// Enough silence: the box is gone, not merely busy.
Offline,
}
fn verdict_after_misses(misses: u32) -> ProbeVerdict {
if misses >= OFFLINE_AFTER {
ProbeVerdict::Offline
} else {
ProbeVerdict::Hold { remaining: OFFLINE_AFTER - misses }
}
}
impl FleetPoll {
pub fn new() -> Self {
Self {
@ -69,6 +107,9 @@ impl FleetPoll {
jobs: Vec::new(),
loras: Vec::new(),
busy: Vec::new(),
health_fails: Vec::new(),
health_quiet: Vec::new(),
models_quiet: Vec::new(),
in_flight: HashMap::new(),
}
}
@ -79,6 +120,25 @@ impl FleetPoll {
self.jobs.remove(index);
self.loras.remove(index);
self.busy.remove(index);
self.health_fails.remove(index);
self.health_quiet.remove(index);
self.models_quiet.remove(index);
}
/// One `/health` answer landed: the row is live and its miss streak
/// (and the log-once latch) resets. Returns the streak it ended.
fn note_health_ok(&mut self, index: usize) -> u32 {
let missed = self.health_fails[index];
self.health_fails[index] = 0;
self.health_quiet[index] = false;
missed
}
/// One `/health` probe missed. The row is NOT touched until the verdict
/// says offline — a box mid-denoise or mid-download is still there.
fn note_health_miss(&mut self, index: usize) -> ProbeVerdict {
self.health_fails[index] = self.health_fails[index].saturating_add(1);
verdict_after_misses(self.health_fails[index])
}
fn row_by_url(&self, url: &str) -> Option<usize> {
@ -123,6 +183,9 @@ impl FleetPoll {
self.jobs.push(Vec::new());
self.loras.push(Vec::new());
self.busy.push(false);
self.health_fails.push(0);
self.health_quiet.push(false);
self.models_quiet.push(false);
changed = true;
}
}
@ -206,12 +269,14 @@ impl FleetPoll {
/// (caller redraws the fleet panel). A response for a row that was
/// coalesced or lease-expired while the request flew is dropped.
pub fn handle_response(&mut self, cx: &mut Cx, item: &NetworkResponse) -> bool {
let (request_id, response) = match item {
let (request_id, response, transport_error) = match item {
NetworkResponse::HttpResponse {
request_id,
response,
} => (*request_id, Some(response)),
NetworkResponse::HttpError { request_id, .. } => (*request_id, None),
} => (*request_id, Some(response), None),
NetworkResponse::HttpError { request_id, error } => {
(*request_id, None, Some(error.message.clone()))
}
_ => return false,
};
let Some(in_flight) = self.in_flight.remove(&request_id) else {
@ -222,15 +287,22 @@ impl FleetPoll {
};
match in_flight.pending {
Pending::Health => {
let status = response.map(|r| r.status_code);
let health = response
.filter(|r| r.status_code == 200)
.and_then(|r| r.get_string_body())
.and_then(|body| HealthJson::deserialize_json_lenient(&body).ok());
let up = health.is_some();
self.latency_ms[index] =
up.then(|| in_flight.sent_at.elapsed().as_millis() as u64);
self.snapshots[index].health = health;
if up {
let missed = self.note_health_ok(index);
if missed > 0 {
log!(
"fleet: {} answered again after {missed} missed probe(s)",
in_flight.base_url
);
}
self.latency_ms[index] = Some(in_flight.sent_at.elapsed().as_millis() as u64);
self.snapshots[index].health = health;
// A fresh health may prove this row aliases another
// (same node_key/node_id) — collapse before fetching
// models so the duplicate never renders.
@ -240,49 +312,114 @@ impl FleetPoll {
let url = format!("{base_url}/models");
self.get(cx, base_url, url, Pending::Models);
}
} else {
self.snapshots[index].models.clear();
self.busy[index] = false;
return true;
}
// SLOW IS NOT GONE. A missed probe costs the row its
// latency reading and nothing else until the misses pile up.
let verdict = self.note_health_miss(index);
self.latency_ms[index] = None;
self.busy[index] = false;
let reason = transport_error.unwrap_or_else(|| match status {
Some(code) => format!("HTTP {code}"),
None => "unreadable /health".to_string(),
});
let was_up = self.snapshots[index].is_up();
match verdict {
ProbeVerdict::Offline => {
if was_up {
// ONE line, naming the reason, at the moment the
// box actually goes offline.
log!(
"fleet: {} offline after {OFFLINE_AFTER} failed /health probes: \
{reason}",
in_flight.base_url
);
}
self.snapshots[index].health = None;
self.snapshots[index].models.clear();
was_up
}
ProbeVerdict::Hold { remaining } => {
if !self.health_quiet[index] {
self.health_quiet[index] = true;
log!(
"fleet: {} missed a /health probe ({reason}) — holding it up \
for {remaining} more",
in_flight.base_url
);
}
// The row keeps its last known health and models: a
// busy box is still the same box.
false
}
}
true
}
Pending::Models => {
self.busy[index] = false;
// One line per failing STREAK. A saturated box drops a
// model probe now and then; that is worth knowing once, not
// thirty times a minute.
let complain = |quiet: &mut bool, message: String| {
if !*quiet {
*quiet = true;
log!("{}", message);
}
};
let quiet = &mut self.models_quiet[index];
let models = match response {
Some(response) if response.status_code == 200 => {
match response.get_string_body() {
Some(body) => match ModelsJson::deserialize_json_lenient(&body) {
Ok(models) => Some(models),
Err(error) => {
log!(
"fleet: {} /models JSON rejected ({} bytes): {:?}",
in_flight.base_url,
body.len(),
error
complain(
quiet,
format!(
"fleet: {} /models JSON rejected ({} bytes): {:?}",
in_flight.base_url,
body.len(),
error
),
);
None
}
},
None => {
log!("fleet: {} /models returned no text body", in_flight.base_url);
complain(
quiet,
format!(
"fleet: {} /models returned no text body",
in_flight.base_url
),
);
None
}
}
}
Some(response) => {
log!(
"fleet: {} /models returned HTTP {}",
in_flight.base_url,
response.status_code
complain(
quiet,
format!(
"fleet: {} /models returned HTTP {}",
in_flight.base_url, response.status_code
),
);
None
}
None => {
log!("fleet: {} /models request failed", in_flight.base_url);
complain(
quiet,
format!(
"fleet: {} /models request failed ({}) — keeping its last model list",
in_flight.base_url,
transport_error.as_deref().unwrap_or("no response")
),
);
None
}
};
if let Some(models) = models {
*quiet = false;
self.snapshots[index].models = models.models;
}
// Live job list last (running + queued, other clients too).
@ -461,10 +598,70 @@ mod tests {
DiscoveredNode {
base_url: base_url.to_string(),
node_id,
fleet: makepad_asset_ai::discovery::DEFAULT_FLEET.to_string(),
fleet: makepad_ai_hub::discovery::DEFAULT_FLEET.to_string(),
}
}
/// SLOW IS NOT GONE. Under a full-GPU denoise or a 17 GB weight pull a
/// box answers late; the panel used to flip it to "down" on the first
/// miss and back on the next, once a second, while every box in fact
/// answered 200 to curl.
#[test]
fn a_busy_box_survives_missed_probes_and_only_then_goes_offline() {
let mut fleet = fleet(&["http://10.0.0.165:8123"]);
fleet.snapshots[0].health = Some(health(1000, "key-a"));
fleet.snapshots[0].models = ModelsJson::deserialize_json_lenient(
r#"{"models":[{"id":"minimax-h3","domain":"video","backend":"h3","available":true,"gated":false,"state":"ready"}]}"#,
)
.expect("test models json parses")
.models;
// Two misses: still up, still holding its model list.
assert_eq!(fleet.note_health_miss(0), ProbeVerdict::Hold { remaining: 2 });
assert!(fleet.snapshots[0].is_up());
assert_eq!(fleet.note_health_miss(0), ProbeVerdict::Hold { remaining: 1 });
assert!(fleet.snapshots[0].is_up());
assert_eq!(fleet.snapshots[0].models.len(), 1);
// One answer and the streak is gone — no flicker, no lost state.
assert_eq!(fleet.note_health_ok(0), 2);
assert_eq!(fleet.note_health_miss(0), ProbeVerdict::Hold { remaining: 2 });
// Sustained silence IS offline.
fleet.note_health_miss(0);
assert_eq!(fleet.note_health_miss(0), ProbeVerdict::Offline);
}
#[test]
fn the_offline_verdict_needs_a_full_streak() {
assert_eq!(verdict_after_misses(0), ProbeVerdict::Hold { remaining: 3 });
assert_eq!(verdict_after_misses(1), ProbeVerdict::Hold { remaining: 2 });
assert_eq!(verdict_after_misses(OFFLINE_AFTER - 1), ProbeVerdict::Hold { remaining: 1 });
assert_eq!(verdict_after_misses(OFFLINE_AFTER), ProbeVerdict::Offline);
assert_eq!(verdict_after_misses(99), ProbeVerdict::Offline);
}
/// The log-once latch: a failing streak says so once, not once per poll.
/// The old behaviour wrote 16-32 identical lines per box per minute.
#[test]
fn a_failing_streak_is_reported_once_not_once_per_poll() {
let mut fleet = fleet(&["http://10.0.0.165:8123"]);
fleet.snapshots[0].health = Some(health(1000, "key-a"));
assert!(!fleet.health_quiet[0], "a healthy row has nothing latched");
// The handler latches on the first complaint of a streak.
fleet.health_quiet[0] = true;
fleet.note_health_miss(0);
assert!(fleet.health_quiet[0], "still latched: no second line");
// A success unlatches it, so the NEXT streak is reported again.
fleet.note_health_ok(0);
assert!(!fleet.health_quiet[0]);
// A box that answers /health and drops /models keeps its OWN latch,
// so it does not complain once per poll forever.
fleet.models_quiet[0] = true;
fleet.note_health_ok(0);
assert!(fleet.models_quiet[0]);
}
#[test]
fn same_node_id_via_two_addresses_stays_one_row() {
// First beacon answered health with node_id 11; the same service

View file

@ -7,6 +7,7 @@
//! not-provisioned cards until they get the same local-folder path.
use makepad_asset_importer::ao_bake;
use makepad_asset_importer::classic_import::ClassicSource;
use makepad_asset_importer::pack_import::{
self, kenney_pack, kenney_spec, KenneyPack, IMPORT_MANIFEST_FILE, KENNEY_ASSETS_HOME,
KENNEY_CREDITS, KENNEY_GITHUB, KENNEY_HOME, KENNEY_LICENSE, KENNEY_PACKS, KENNEY_SOURCE_ID,
@ -54,6 +55,10 @@ pub enum ImportJob {
Duke3d {
path: String,
},
EaClassic {
source: ClassicSource,
path: String,
},
Quake2 {
path: String,
},
@ -82,6 +87,7 @@ impl ImportJob {
ImportJob::Doom { .. } => "Doom shareware".into(),
ImportJob::Quake { .. } => "Quake shareware".into(),
ImportJob::Duke3d { .. } => "Duke3D shareware".into(),
ImportJob::EaClassic { source, .. } => source.title().into(),
ImportJob::Quake2 { .. } => "Quake II shareware".into(),
ImportJob::Quake3 { .. } => "Quake III demo".into(),
ImportJob::DarkMod { .. } => "The Dark Mod".into(),
@ -100,6 +106,10 @@ impl ImportJob {
pub fn conflicts(&self, other: &ImportJob) -> bool {
match (self, other) {
(ImportJob::Kenney { pack: a, .. }, ImportJob::Kenney { pack: b, .. }) => a == b,
(
ImportJob::EaClassic { source: a, .. },
ImportJob::EaClassic { source: b, .. },
) => a == b,
(ImportJob::KenneyAll, ImportJob::KenneyAll)
| (ImportJob::KenneyAll, ImportJob::Kenney { .. })
| (ImportJob::Kenney { .. }, ImportJob::KenneyAll)
@ -4130,6 +4140,8 @@ fn kind_tag(kind: AssetKind) -> &'static str {
AssetKind::Billboard => "billboard",
AssetKind::Game => "game",
AssetKind::VjEffect => "vjeffect",
AssetKind::Data => "data",
AssetKind::ModelProgram => "model-program",
}
}

View file

@ -1,4 +1,4 @@
//! Additive Import cards for Freedoom, LibreQuake, and official shareware.
//! Additive Import cards for libre, freeware, and official shareware packs.
//!
//! Downloads go through platform `cx.http_request`. Bytes are unpacked by
//! [`makepad_asset_importer::classic_fetch`], then converted via
@ -107,6 +107,87 @@ pub const DUKE3D_MODULE: PackModule = PackModule {
import_wired: true,
};
const EA_CNC_HOME: &str = "https://www.ea.com/games/command-and-conquer";
const EA_CLASSIC_LICENSE: &str = "EA freeware — local use, not redistributable";
const EA_CLASSIC_LICENSE_BLURB: &str = "© Westwood Studios / Electronic Arts. EA freeware for local preview in this app only. Not a redistributable grant.";
const EA_CLASSIC_CREDITS: &str = "Westwood Studios / Electronic Arts";
pub const EA_MODULES: [PackModule; 4] = [
PackModule {
id: "cnc",
title: "Tiberian Dawn",
blurb: "EA's 1995 RTS, freeware since 2007 — terrain, units, structures, sounds and every campaign and multiplayer map convert into RTS maps for the sandbox.",
license: EA_CLASSIC_LICENSE,
license_blurb: EA_CLASSIC_LICENSE_BLURB,
homepage: EA_CNC_HOME,
terms_url: EA_CNC_HOME,
source_page: EA_CNC_HOME,
github: None,
credits: EA_CLASSIC_CREDITS,
import_wired: true,
},
PackModule {
id: "ra",
title: "Red Alert",
blurb: "The 1996 sequel, freeware since 2008 — Allied and Soviet arsenals, snow/temperate/interior maps.",
license: EA_CLASSIC_LICENSE,
license_blurb: EA_CLASSIC_LICENSE_BLURB,
homepage: EA_CNC_HOME,
terms_url: EA_CNC_HOME,
source_page: EA_CNC_HOME,
github: None,
credits: EA_CLASSIC_CREDITS,
import_wired: true,
},
PackModule {
id: "ts",
title: "Tiberian Sun",
blurb: "The 1999 isometric sequel, freeware since 2010 — GDI/Nod units and tilesets; maps are generated from the tilesets.",
license: EA_CLASSIC_LICENSE,
license_blurb: EA_CLASSIC_LICENSE_BLURB,
homepage: EA_CNC_HOME,
terms_url: EA_CNC_HOME,
source_page: EA_CNC_HOME,
github: None,
credits: EA_CLASSIC_CREDITS,
import_wired: true,
},
PackModule {
id: "d2k",
title: "Dune 2000",
blurb: "Westwood's 1998 Dune RTS — Atreides/Harkonnen/Ordos units and the Arrakis tilesets; maps are generated.",
license: EA_CLASSIC_LICENSE,
license_blurb: EA_CLASSIC_LICENSE_BLURB,
homepage: EA_CNC_HOME,
terms_url: EA_CNC_HOME,
source_page: EA_CNC_HOME,
github: None,
credits: EA_CLASSIC_CREDITS,
import_wired: true,
},
];
pub const EA_SOURCES: [ClassicSource; 4] = [
ClassicSource::Cnc,
ClassicSource::RedAlert,
ClassicSource::TiberianSun,
ClassicSource::Dune2000,
];
pub const EA_PACK_LABELS: [&str; 4] = ["Tiberian Dawn", "Red Alert", "Tiberian Sun", "Dune 2000"];
pub fn ea_source_for_index(index: usize) -> ClassicSource {
EA_SOURCES.get(index).copied().unwrap_or(ClassicSource::Cnc)
}
pub fn ea_index_for_source(source: ClassicSource) -> Option<usize> {
EA_SOURCES.iter().position(|candidate| *candidate == source)
}
fn is_ea_source(source: ClassicSource) -> bool {
ea_index_for_source(source).is_some()
}
pub const QUAKE2_MODULE: PackModule = PackModule {
id: QUAKE2_SOURCE_ID,
title: "Quake II shareware",
@ -157,6 +238,10 @@ pub const PACK_MODULES_WITH_CLASSIC: &[PackModule] = &[
LIBREQUAKE_MODULE,
QUAKE_MODULE,
DUKE3D_MODULE,
EA_MODULES[0],
EA_MODULES[1],
EA_MODULES[2],
EA_MODULES[3],
QUAKE2_MODULE,
QUAKE3_MODULE,
DARKMOD_MODULE,
@ -1813,8 +1898,10 @@ fn classic_library_landings(
let mut seen_icons = std::collections::BTreeSet::new();
let mut seen_titles = std::collections::BTreeSet::new();
for asset in assets {
let ea_source = is_ea_source(source);
if matches!(asset.kind, AssetKind::Texture)
&& !matches!(source, classic_import::ClassicSource::Quake3)
&& !(ea_source && asset.key.starts_with("icons/"))
{
continue;
}
@ -1839,11 +1926,30 @@ fn classic_library_landings(
AssetKind::Character => (path, "model/gltf-binary", "character"),
AssetKind::Weapon => (path, "model/gltf-binary", "weapon"),
AssetKind::Prop => (path, "model/gltf-binary", "prop"),
AssetKind::Texture => (path, "image/png", "image"),
AssetKind::Texture => (
path,
"image/png",
if ea_source && asset.key.starts_with("icons/") {
"texture"
} else {
"image"
},
),
AssetKind::Audio => {
let music = asset.key.starts_with("music/")
|| asset.tags.iter().any(|t| t.eq_ignore_ascii_case("music"));
(path, "audio/wav", if music { "music" } else { "sfx" })
let speech = asset.tags.iter().any(|t| t.eq_ignore_ascii_case("speech"));
(
path,
"audio/wav",
if speech {
"speech"
} else if music {
"music"
} else {
"sfx"
},
)
}
AssetKind::Billboard => {
let ext = path.extension().and_then(|e| e.to_str()).unwrap_or("");
@ -1908,12 +2014,17 @@ fn classic_library_landings(
{
continue;
}
let dedupe_key = if ea_source {
asset.key.clone()
} else {
title.clone()
};
if !matches!(source, classic_import::ClassicSource::Quake3)
&& matches!(
asset.kind,
AssetKind::Billboard | AssetKind::Audio | AssetKind::Texture
)
&& !seen_titles.insert(title.clone())
&& !seen_titles.insert(dedupe_key)
{
continue;
}
@ -2191,6 +2302,10 @@ pub struct ClassicImportPage {
pub librequake: ClassicImportCard,
pub quake: ClassicImportCard,
pub duke3d: ClassicImportCard,
pub cnc: ClassicImportCard,
pub ra: ClassicImportCard,
pub ts: ClassicImportCard,
pub d2k: ClassicImportCard,
pub quake2: ClassicImportCard,
pub quake3: ClassicImportCard,
pub darkmod: ClassicImportCard,
@ -2204,6 +2319,10 @@ impl Default for ClassicImportPage {
librequake: ClassicImportCard::new(ClassicSource::LibreQuake),
quake: ClassicImportCard::new(ClassicSource::Quake),
duke3d: ClassicImportCard::new(ClassicSource::Duke3d),
cnc: ClassicImportCard::new(ClassicSource::Cnc),
ra: ClassicImportCard::new(ClassicSource::RedAlert),
ts: ClassicImportCard::new(ClassicSource::TiberianSun),
d2k: ClassicImportCard::new(ClassicSource::Dune2000),
quake2: ClassicImportCard::new(ClassicSource::Quake2),
quake3: ClassicImportCard::new(ClassicSource::Quake3),
darkmod: ClassicImportCard::new(ClassicSource::DarkMod),
@ -2219,6 +2338,10 @@ impl ClassicImportPage {
ClassicSource::LibreQuake => &self.librequake,
ClassicSource::Quake => &self.quake,
ClassicSource::Duke3d => &self.duke3d,
ClassicSource::Cnc => &self.cnc,
ClassicSource::RedAlert => &self.ra,
ClassicSource::TiberianSun => &self.ts,
ClassicSource::Dune2000 => &self.d2k,
ClassicSource::Quake2 => &self.quake2,
ClassicSource::Quake3 => &self.quake3,
ClassicSource::DarkMod => &self.darkmod,
@ -2232,6 +2355,10 @@ impl ClassicImportPage {
ClassicSource::LibreQuake => &mut self.librequake,
ClassicSource::Quake => &mut self.quake,
ClassicSource::Duke3d => &mut self.duke3d,
ClassicSource::Cnc => &mut self.cnc,
ClassicSource::RedAlert => &mut self.ra,
ClassicSource::TiberianSun => &mut self.ts,
ClassicSource::Dune2000 => &mut self.d2k,
ClassicSource::Quake2 => &mut self.quake2,
ClassicSource::Quake3 => &mut self.quake3,
ClassicSource::DarkMod => &mut self.darkmod,
@ -2244,6 +2371,10 @@ impl ClassicImportPage {
|| self.librequake.compiling()
|| self.quake.compiling()
|| self.duke3d.compiling()
|| self.cnc.compiling()
|| self.ra.compiling()
|| self.ts.compiling()
|| self.d2k.compiling()
|| self.quake2.compiling()
|| self.quake3.compiling()
|| self.darkmod.compiling()
@ -2293,6 +2424,10 @@ impl ClassicImportPage {
&mut self.librequake,
&mut self.quake,
&mut self.duke3d,
&mut self.cnc,
&mut self.ra,
&mut self.ts,
&mut self.d2k,
&mut self.quake2,
&mut self.quake3,
&mut self.darkmod,
@ -2306,10 +2441,14 @@ impl ClassicImportPage {
let c = self.librequake.poll();
let d = self.quake.poll();
let e = self.duke3d.poll();
let f = self.quake2.poll();
let g = self.quake3.poll();
let h = self.darkmod.poll();
a || b || c || d || e || f || g || h
let f = self.cnc.poll();
let g = self.ra.poll();
let h = self.ts.poll();
let i = self.d2k.poll();
let j = self.quake2.poll();
let k = self.quake3.poll();
let l = self.darkmod.poll();
a || b || c || d || e || f || g || h || i || j || k || l
}
pub fn take_all_landings(&mut self) -> Vec<LibraryLanding> {
@ -2318,6 +2457,10 @@ impl ClassicImportPage {
out.extend(self.librequake.take_library_landings());
out.extend(self.quake.take_library_landings());
out.extend(self.duke3d.take_library_landings());
out.extend(self.cnc.take_library_landings());
out.extend(self.ra.take_library_landings());
out.extend(self.ts.take_library_landings());
out.extend(self.d2k.take_library_landings());
out.extend(self.quake2.take_library_landings());
out.extend(self.quake3.take_library_landings());
out.extend(self.darkmod.take_library_landings());
@ -2330,6 +2473,10 @@ impl ClassicImportPage {
out.extend(self.librequake.take_previews());
out.extend(self.quake.take_previews());
out.extend(self.duke3d.take_previews());
out.extend(self.cnc.take_previews());
out.extend(self.ra.take_previews());
out.extend(self.ts.take_previews());
out.extend(self.d2k.take_previews());
out.extend(self.quake2.take_previews());
out.extend(self.quake3.take_previews());
out.extend(self.darkmod.take_previews());
@ -2399,6 +2546,10 @@ mod tests {
DOOM_MODULE,
QUAKE_MODULE,
DUKE3D_MODULE,
EA_MODULES[0],
EA_MODULES[1],
EA_MODULES[2],
EA_MODULES[3],
QUAKE2_MODULE,
QUAKE3_MODULE,
] {
@ -2416,6 +2567,62 @@ mod tests {
}
}
#[test]
fn ea_classics_modules_dropdown_and_library_domains_are_wired() {
let ids: Vec<_> = PACK_MODULES_WITH_CLASSIC.iter().map(|module| module.id).collect();
for id in ["cnc", "ra", "ts", "d2k"] {
assert!(ids.contains(&id), "missing EA classic module {id}");
}
for (index, source) in EA_SOURCES.into_iter().enumerate() {
assert_eq!(ea_source_for_index(index), source);
assert_eq!(ea_index_for_source(source), Some(index));
}
let staged = std::env::temp_dir().join(format!(
"asset-ui-ea-landings-{}",
std::process::id()
));
let _ = std::fs::remove_dir_all(&staged);
std::fs::create_dir_all(staged.join("billboards/cnc")).unwrap();
std::fs::create_dir_all(staged.join("worlds")).unwrap();
std::fs::write(staged.join("billboards/cnc/mtnk.billboard"), b"manifest").unwrap();
std::fs::write(staged.join("billboards/cnc/mtnk_thumb.png"), b"png").unwrap();
std::fs::write(staged.join("worlds/scm01ea.glb"), b"glb").unwrap();
std::fs::write(staged.join("worlds/scm01ea.png"), b"png").unwrap();
let assets = [
classic_import::ClassicAsset {
key: "billboards/cnc/mtnk".into(),
kind: AssetKind::Billboard,
rel_path: "billboards/cnc/mtnk.billboard".into(),
tags: vec!["unit".into()],
icon_rel: Some("billboards/cnc/mtnk_thumb.png".into()),
},
classic_import::ClassicAsset {
key: "worlds/scm01ea".into(),
kind: AssetKind::World,
rel_path: "worlds/scm01ea.glb".into(),
tags: vec!["map".into()],
icon_rel: Some("worlds/scm01ea.png".into()),
},
];
let landings = classic_library_landings(&staged, ClassicSource::Cnc, "cnc", &assets);
assert_eq!(
landings
.iter()
.find(|landing| landing.path.ends_with("mtnk.billboard"))
.map(|landing| landing.domain),
Some("billboard")
);
assert_eq!(
landings
.iter()
.find(|landing| landing.path.ends_with("scm01ea.glb"))
.map(|landing| landing.domain),
Some("map")
);
let _ = std::fs::remove_dir_all(staged);
}
#[test]
fn darkmod_card_is_explicitly_nc_sa() {
let text = format!(

View file

@ -1219,7 +1219,15 @@ pub fn infer_import_tags(
fn push_prompt_source_tag(tags: &mut Vec<String>, prompt: &str) {
let p = prompt.to_ascii_lowercase();
let source = if p.starts_with("the dark mod") {
let source = if p == "cnc" || p.starts_with("cnc ") || p.starts_with("cnc:") {
"cnc"
} else if p == "ra" || p.starts_with("ra ") || p.starts_with("ra:") {
"ra"
} else if p == "ts" || p.starts_with("ts ") || p.starts_with("ts:") {
"ts"
} else if p == "d2k" || p.starts_with("d2k ") || p.starts_with("d2k:") {
"d2k"
} else if p.starts_with("the dark mod") {
"darkmod"
} else if p.starts_with("freedoom") {
"freedoom"
@ -2409,7 +2417,7 @@ frame 1 A 2 64 64 trooper_a2.png
let dir = TestDir::new("audio-provenance");
let mut library = Library::open(&dir.0);
let samples: Vec<f32> = (0..64).map(|i| (i as f32 / 8.0).sin() * 0.5).collect();
let wav_bytes = makepad_asset_ai::wav::encode_wav_pcm16_mono(&samples, 24_000);
let wav_bytes = makepad_ai_hub::wav::encode_wav_pcm16_mono(&samples, 24_000);
// A poisoned caller thumbnail (e.g. the upstream pipeline image) is
// DISCARDED; the sidecar is the payload's own waveform strip.

File diff suppressed because it is too large Load diff

View file

@ -221,7 +221,7 @@ impl MaskPaint {
return None;
}
let rgba = bgra_to_rgba8(&self.canvas);
makepad_asset_ai::testpattern::encode_png_rgba(&rgba, self.width, self.height).ok()
makepad_ai_hub::testpattern::encode_png_rgba(&rgba, self.width, self.height).ok()
}
/// The mask as an opaque gray PNG (white = repaint).
@ -233,7 +233,7 @@ impl MaskPaint {
for &m in &self.mask {
rgba.extend_from_slice(&[m, m, m, 255]);
}
makepad_asset_ai::testpattern::encode_png_rgba(&rgba, self.width, self.height).ok()
makepad_ai_hub::testpattern::encode_png_rgba(&rgba, self.width, self.height).ok()
}
fn ensure_textures(&mut self, cx: &mut Cx) {

View file

@ -1020,6 +1020,8 @@ impl MeshView {
self.extra_instances.push(ModelInstance {
model: id,
transform: trs_yaw(spec.pos, spec.yaw, 1.0),
tint: vec4(1.0, 1.0, 1.0, 1.0),
color_adjust: vec4(0.0, 1.0, 1.0, 0.0),
dynamic: true,
depth_order: 0.0,
part_poses: Vec::new(),
@ -1493,6 +1495,8 @@ impl MeshView {
self.instance = Some(ModelInstance {
model: id,
transform: Mat4f::identity(),
tint: vec4(1.0, 1.0, 1.0, 1.0),
color_adjust: vec4(0.0, 1.0, 1.0, 0.0),
dynamic: true,
depth_order: 0.0,
part_poses: Vec::new(),
@ -1511,6 +1515,8 @@ impl MeshView {
0.35,
scale,
),
tint: vec4(1.0, 1.0, 1.0, 1.0),
color_adjust: vec4(0.0, 1.0, 1.0, 0.0),
// Realtime CSM only collects `dynamic` movers.
dynamic: true,
depth_order: 0.0,
@ -1914,6 +1920,8 @@ impl Widget for MeshView {
pos: vec4(s.pos.x, s.pos.y, s.pos.z, self.orbit_yaw),
size: vec4(s.width, s.height, 0.0, 0.0),
uv: vec4(0.0, 0.0, 1.0, 1.0),
tint: vec4(1.0, 1.0, 1.0, 1.0),
color_adjust: vec4(0.0, 1.0, 1.0, 0.0),
})
})
.collect();

View file

@ -772,7 +772,7 @@ pub fn studio_equirect_png() -> Vec<u8> {
rgba[i + 3] = 255;
}
}
makepad_asset_ai::testpattern::encode_png_rgba(&rgba, W, H).expect("studio equirect encodes")
makepad_ai_hub::testpattern::encode_png_rgba(&rgba, W, H).expect("studio equirect encodes")
}
#[cfg(test)]

View file

@ -9,14 +9,14 @@
//! - every fetched artifact also routes to the matching viewer.
//!
//! Box choice per stage = the fleet affinity scheduler
//! (`makepad_asset_ai::fleet`): loaded > ready > downloading > absent,
//! (`makepad_ai_hub::fleet`): loaded > ready > downloading > absent,
//! tiebreak queue depth, evaluated at stage START (a chain's later stages
//! see fresh snapshots). Text expansion has one deliberate policy layer:
//! a ready Qwen3.8-27B outranks the smaller fallback, but an absent or still
//! downloading 3.8 never displaces the already-ready qwen3.5-9b lane.
use makepad_asset_ai::fleet::{self, BoxSnapshot};
use makepad_asset_ai::protocol::{
use makepad_ai_hub::fleet::{self, BoxSnapshot};
use makepad_ai_hub::protocol::{
ArtifactRefJson, GenerateRequestJson, GenerateResponseJson, JobStatusJson, NamedInputJson, LoraRefJson};
use makepad_micro_serde::{DeJson, SerJson};
use makepad_widgets::*;
@ -26,44 +26,12 @@ use std::collections::{HashMap, HashSet};
// Presets
// ---------------------------------------------------------------------------
/// Image canvas presets; entry 0 is the chain default. Flux wants /16 dims.
pub const IMAGE_SIZES: &[(u32, u32)] = &[
(512, 512),
(768, 768),
(1024, 1024),
(768, 512),
(512, 768),
(1024, 576),
];
/// Image step presets; the dropdown's extra first entry means "model
/// default" (schnell 4, dev-class ~20).
pub const IMAGE_STEPS: &[u32] = &[4, 8, 12, 20, 28, 50];
/// img2img strength choices for edit chains: 1.0 = a full instruction edit
/// (reference tokens only); lower = the sampler starts from the VAE-encoded
/// input at sigma index floor((1-strength)*steps), keeping more of it.
pub const EDIT_STRENGTHS: &[f32] = &[1.0, 0.85, 0.7, 0.55, 0.4, 0.25];
/// LoRA strength choices for the image stage.
pub const LORA_STRENGTHS: &[f32] = &[1.0, 0.8, 0.6, 0.4, 1.2];
/// RIFE interpolation factors offered for video (1 = off).
pub const VIDEO_INTERPOLATE: &[u32] = &[1, 2, 4];
/// Enhance-stage factor choices, shared by the uprez and tween pickers.
pub const ENHANCE_FACTORS: &[u32] = &[1, 2, 4];
/// TRELLIS UV-atlas presets. 1024 preserves the current fast default; the
/// larger atlases trade bake time and device memory for sharper materials.
pub const MESH_TEXTURE_SIZES: &[u32] = &[1024, 2048, 4096];
/// QEM face-count presets. Index 0 is Auto (12k objects / 20k characters).
pub const MESH_FACE_COUNTS: &[u32] = &[0, 12_000, 20_000, 40_000, 80_000, 160_000];
/// Video canvas presets; entry 0 is the small default.
pub const VIDEO_SIZES: &[(u32, u32)] = &[(640, 352), (864, 480), (960, 544)];
/// Video (frames, steps) presets at 16 fps; entry 0 is the default.
pub const VIDEO_LENGTHS: &[(u32, u32)] = &[(39, 30), (65, 30), (97, 40), (129, 50)];
/// Full-song targets offered by the UI. Music3 accepts any duration from
/// five seconds through five minutes; these minute-aligned presets keep the
/// common choice legible and make a three-minute song the honest default.
pub const MUSIC_LENGTHS: &[u32] = &[60, 120, 180, 240, 300];
pub const MUSIC_DEFAULT_SECONDS: u32 = 180;
pub const MUSIC_MIN_SECONDS: u32 = 5;
pub const MUSIC_MAX_SECONDS: u32 = 300;
pub use makepad_asset_creator::presets::{
EDIT_STRENGTHS, ENHANCE_FACTORS, EXPAND_FALLBACK_NOTE, IMAGE_SIZES, IMAGE_STEPS,
LORA_STRENGTHS, MESH_FACE_COUNTS, MESH_TEXTURE_SIZES, MUSIC_DEFAULT_SECONDS,
MUSIC_LENGTHS, MUSIC_MAX_SECONDS, MUSIC_MIN_SECONDS, VIDEO_INTERPOLATE, VIDEO_LENGTHS,
VIDEO_SIZES,
};
/// Human-facing clock label used by the duration picker and run details.
pub fn format_music_duration(seconds: u32) -> String {
@ -148,6 +116,12 @@ pub struct GenParams {
/// Enhance stage: append the motion-vector `mkfl` box for
/// arbitrary-rate GPU playback.
pub enhance_flow: bool,
/// Video stage: send the keyframe as BOTH the first frame (`input_b64`)
/// and the H3 wire's `last_frame` named input, so the FL2VA conditioning
/// lands the clip back on its opening image — a seamless loop (verified
/// on fasth3-4step: endpoint delta ≈ adjacent-frame noise). Derived from
/// the preset row at dispatch, never persisted in saved specs.
pub video_loop: bool,
}
impl Default for GenParams {
@ -170,6 +144,7 @@ impl Default for GenParams {
enhance_upscale: 2,
enhance_interpolate: 2,
enhance_flow: true,
video_loop: false,
}
}
}
@ -185,6 +160,9 @@ pub struct Preset {
/// fan-out followed by an explicit human choice gate. The chosen
/// artifact is the only output promoted into the linear chain.
pub fan_out_stage: Option<usize>,
/// The video stage of this chain loops: its keyframe rides as first AND
/// last frame (see [`GenParams::video_loop`]).
pub video_loop: bool,
}
impl Preset {
@ -198,6 +176,22 @@ impl Preset {
domains,
pins,
fan_out_stage: None,
video_loop: false,
}
}
/// A linear chain whose video stage is a seamless loop.
const fn looped(
name: &'static str,
domains: &'static [&'static str],
pins: &'static [(&'static str, &'static str)],
) -> Self {
Self {
name,
domains,
pins,
fan_out_stage: None,
video_loop: true,
}
}
@ -212,6 +206,7 @@ impl Preset {
domains,
pins,
fan_out_stage: Some(stage),
video_loop: false,
}
}
}
@ -235,6 +230,18 @@ const CHARACTER_RIG_MODEL: &str = "skintokens";
const CHARACTER_MOTION_MODEL: &str = "hy-motion";
/// Instruction image editing (reference image + "change …" prompt).
const EDIT_MODEL: &str = "flux2-klein-4b";
/// Sprite enhancement runs on the 32B dev DiT, NOT the 4-step distilled
/// klein that `EDIT_MODEL` pins for interactive edits. Measured on the Doom
/// imp hero frame (2026-08-31): klein 4-step renders a smoothed version of
/// the original; dev at 20-30 steps redraws it with real anatomy, claws and
/// teeth. The distillation, not the prompt, was the ceiling.
///
/// `flux2-dev-q4-24g` is the same DiT quantized for the 24GB class. It is a
/// DIFFERENT numerics class ("expect its own look at the same seed"), so a
/// single asset must be enhanced entirely on one tier or its cells will not
/// match each other.
const SPRITE_ENHANCE_MODEL: &str = "flux2-dev";
pub const SPRITE_ENHANCE_MODEL_24G: &str = "flux2-dev-q4-24g";
/// General image 4x upscaling (RealESRGAN x4plus). Pinned — the domain has
/// exactly one model, so no dropdown.
const UPSCALE_MODEL: &str = "realesrgan-x4plus";
@ -310,8 +317,8 @@ fn pick_ready_model_target(
model.available
&& matches!(
model.state.as_str(),
makepad_asset_ai::protocol::MODEL_STATE_READY
| makepad_asset_ai::protocol::MODEL_STATE_LOADED
makepad_ai_hub::protocol::MODEL_STATE_READY
| makepad_ai_hub::protocol::MODEL_STATE_LOADED
)
})
})
@ -492,6 +499,14 @@ pub const PRESETS: &[Preset] = &[
&[("enhance", "video-enhance")],
),
Preset::linear("edit selected image (instruction)", &["edit"], &[("edit", EDIT_MODEL)]),
// Classic sprite enhancement: re-render an old game's artwork at modern
// quality. Pinned to the 32B dev tier because the distilled klein only
// smooths (see SPRITE_ENHANCE_MODEL).
Preset::linear(
"sprite → enhance (hi-res)",
&["edit"],
&[("edit", SPRITE_ENHANCE_MODEL)],
),
// Native RealESRGAN x4plus: select a picture, get it back at 4x
// resolution. Consumer-only like `edit` — no prompt-only mode, refused
// without a selected image.
@ -589,6 +604,8 @@ pub const PRESETS: &[Preset] = &[
("motion", CHARACTER_MOTION_MODEL),
],
),
Preset::looped("image → video loop", &["image", "video"], &[]),
Preset::looped("expand → image → video loop", &["text", "image", "video"], &[]),
];
/// Which upstream payload class a stage's request relays as its binary
@ -645,6 +662,67 @@ pub fn seed_replaces_prefix(domains: &[&str], seed_content_type: &str) -> Option
/// Human-facing stage name. In particular, call the text stage what it is:
/// a local model inference step, rather than making it look like string
/// templating in the UI.
/// The parameters a submitted request carries besides its prompt, one
/// `key=value` per line — read off the request that is actually being sent,
/// so an opened run shows what the box got rather than what the UI meant.
/// The prompt is kept separately and in full; an input payload is named by
/// size and type, never by its base64.
pub fn sent_params(request: &GenerateRequestJson) -> String {
let mut lines: Vec<String> = vec![format!("model={}", request.model)];
let mut put = |key: &str, value: String| lines.push(format!("{key}={value}"));
if let Some(text) = &request.negative_prompt {
put("negative_prompt", text.clone());
}
if let Some(text) = &request.lyrics {
put("lyrics", text.clone());
}
if let Some(text) = &request.text {
put("text", text.clone());
}
if let Some(voice) = &request.voice {
put("voice", voice.clone());
}
for (key, value) in [
("width", request.width),
("height", request.height),
("steps", request.steps),
("frames", request.frames),
("interpolate", request.interpolate),
("upscale", request.upscale),
("max_tokens", request.max_tokens),
] {
if let Some(value) = value {
put(key, value.to_string());
}
}
if let Some(seed) = request.seed {
put("seed", seed.to_string());
}
for (key, value) in [
("guidance", request.guidance),
("seconds", request.seconds),
("speed", request.speed),
] {
if let Some(value) = value {
put(key, format!("{value}"));
}
}
if let Some(strength) = request.strength {
put("strength", format!("{strength}"));
}
if let Some(bytes) = &request.input_b64 {
put(
"input",
format!(
"{} b64 chars {}",
bytes.len(),
request.input_content_type.as_deref().unwrap_or("?")
),
);
}
lines.join("\n")
}
pub fn stage_display_name(domain: &str) -> &str {
match domain {
"text" => "LLM prompt expansion",
@ -797,6 +875,16 @@ pub struct StageRun {
/// always have one; other pipelines retain the backend's existing seed
/// behavior.
pub seed: Option<u64>,
/// THE TEXT THIS STAGE ACTUALLY SENT, captured at submit, in full.
///
/// Not `Pipeline::prompt`: an expansion stage rewrites what the next
/// model sees, a music stage carries its lyrics, and a character chain
/// composes its own brief — so the only honest answer to "what did the
/// model get?" is the string that went on the wire. Kept so the run can
/// be opened and read.
pub sent_prompt: String,
/// The parameters that went with it, `key=value` per line.
pub sent_params: String,
pub started: Option<std::time::Instant>,
pub finished: Option<std::time::Instant>,
/// Fetched artifacts: (content_type, bytes).
@ -949,6 +1037,8 @@ impl Pipeline {
reason: String::new(),
service_state: String::new(),
seed: None,
sent_prompt: String::new(),
sent_params: String::new(),
started: None,
finished: None,
outputs: Vec::new(),
@ -1092,6 +1182,17 @@ impl Pipeline {
/// Service job ids this run has in flight on `base_url` (linear stage
/// + fan-out candidates) — lets the fleet panel tell "ours" from other
/// clients' jobs.
/// The text a job of this pipeline was handed, by the box's own job id.
/// Lets a fleet-box view show WHAT a job it is running was asked for —
/// the box itself never reports the prompt back.
pub fn sent_prompt_for_job(&self, job_id: &str) -> Option<&str> {
self.stages
.iter()
.find(|stage| stage.job_id == job_id)
.map(|stage| stage.sent_prompt.as_str())
.filter(|text| !text.is_empty())
}
pub fn job_ids_on(&self, base_url: &str) -> Vec<String> {
let mut ids: Vec<String> = self
.candidate_sets
@ -1197,27 +1298,35 @@ impl Pipeline {
/// identity anchor supplied on its request: `yoshi` can be elaborated,
/// never replaced.
fn prompt_for_stage(&self, stage: usize) -> Result<String, String> {
for earlier in self.stages[..stage].iter().rev() {
for (index, earlier) in self.stages[..stage].iter().enumerate().rev() {
if earlier.domain == "text" {
// An expansion that came back with nothing usable is a
// missing improvement, not a missing input: the person's own
// prompt still says what they want. A CHARACTER chain is the
// exception — its later stages are gated on the brief, so
// there the refusal stands (see `expander_is_optional`).
let optional = self.expander_is_optional(index);
let unusable = |reason: &str| -> Result<String, String> {
if optional {
log!("pipeline: {EXPAND_FALLBACK_NOTE} ({reason})");
Ok(self.prompt.clone())
} else {
Err(format!("LLM prompt expansion {reason}; refusing terse-prompt fallback"))
}
};
let Some((_, bytes)) = earlier
.outputs
.iter()
.find(|(ct, _)| ct.starts_with("text/plain"))
else {
return Err(
"LLM prompt expansion produced no text/plain artifact; refusing terse-prompt fallback"
.to_string(),
);
return unusable("produced no text/plain artifact");
};
let Ok(text) = std::str::from_utf8(bytes) else {
return unusable("artifact is not UTF-8");
};
let text = std::str::from_utf8(bytes).map_err(|_| {
"LLM prompt expansion artifact is not UTF-8; refusing terse-prompt fallback"
.to_string()
})?;
let text = text.trim();
if text.is_empty() {
return Err(
"LLM prompt expansion was empty; refusing terse-prompt fallback".to_string(),
);
return unusable("was empty");
}
if self.is_character_pipeline() {
let words = text.split_whitespace().count();
@ -1495,7 +1604,7 @@ impl Pipeline {
// Music3 can stop earlier when it emits its end-of-audio token.
"music" => {
let (description, lyrics) =
makepad_asset_ai::music3_backend::split_music_prompt(&prompt);
makepad_ai_hub::music3_backend::split_music_prompt(&prompt);
// An expansion stage promises the template's `Lyrics:`
// section (instrumental requests still carry it, holding
// only [Instrumental]). Its absence means the expander
@ -1596,6 +1705,20 @@ impl Pipeline {
request.input_b64 = Some(b64);
request.input_content_type = Some(content_type);
}
// Loop chains: the SAME keyframe rides as both the first frame
// (input_b64, above) and the H3 wire's `last_frame` named input, so
// the clip ends where it began. No keyframe = a hard error, never a
// silent non-looping clip.
if domain == "video" && self.gen.video_loop {
let (b64, content_type) = self.input_for_stage(stage).ok_or_else(|| {
"video loop needs a keyframe image from an earlier stage or seed".to_string()
})?;
request.inputs = Some(vec![NamedInputJson {
name: "last_frame".to_string(),
content_type,
data_b64: b64,
}]);
}
if domain == "inpaint" {
let (b64, content_type) = self
.input_for_stage(stage)
@ -2156,6 +2279,11 @@ impl Pipeline {
Ok(request) => request,
Err(error) => return self.fail_stage(stage, error, events),
};
// Remember what is about to go on the wire, before it goes: this is
// what an opened run shows, and it is the only place the composed
// text still exists as one string.
self.stages[stage].sent_prompt = request_json.prompt.clone().unwrap_or_default();
self.stages[stage].sent_params = sent_params(&request_json);
let url = format!("{}/generate", self.stages[stage].box_url);
let mut request = crate::http::request(url, HttpMethod::POST);
request.set_header("Content-Type".to_string(), "application/json".to_string());
@ -2325,10 +2453,54 @@ impl Pipeline {
if let Some(retry_stage) = self.prepare_character_mesh_retry(stage, &error) {
events.push(PipelineEvent::Changed);
events.extend(self.start_stage(cx, retry_stage, snapshots, avoid));
events
} else {
self.fail_stage(stage, error, events)
return events;
}
self.fail_stage_or_skip_expander(cx, stage, error, snapshots, avoid, events)
}
/// Is `stage` an expansion the run can do WITHOUT?
///
/// A `text` stage in front of other stages is a rewording courtesy: its
/// product is a better prompt, and the person already supplied a
/// perfectly usable one. Two cases are NOT optional and stay hard
/// failures: a chain whose only stage is the expansion (the text IS the
/// product), and a character chain, whose later stages are gated on the
/// brief keeping the named identity — see `prompt_for_stage`.
fn expander_is_optional(&self, stage: usize) -> bool {
self.stages[stage].domain == "text"
&& stage + 1 < self.stages.len()
&& !self.is_character_pipeline()
}
/// End the run — UNLESS the stage that failed was an optional expander,
/// in which case the run carries on from the prompt the person typed.
///
/// A lost expansion used to lose the whole run: the text box hiccuped
/// (busy, evicted, timed out) and a queued video that had nothing to do
/// with the expander simply never happened, with the reason buried in a
/// failed stage nobody was looking at.
fn fail_stage_or_skip_expander(
&mut self,
cx: &mut Cx,
stage: usize,
error: String,
snapshots: &[BoxSnapshot],
avoid: &[String],
mut events: Vec<PipelineEvent>,
) -> Vec<PipelineEvent> {
if !self.expander_is_optional(stage) {
return self.fail_stage(stage, error, events);
}
log!("pipeline: {EXPAND_FALLBACK_NOTE} ({error})");
self.stages[stage].state =
StageState::Failed(format!("{EXPAND_FALLBACK_NOTE} ({error})"));
self.stages[stage].detail = EXPAND_FALLBACK_NOTE.to_string();
self.stages[stage].finished = Some(std::time::Instant::now());
self.stages[stage].progress = 0.0;
events.push(PipelineEvent::StageFailed { stage });
events.push(PipelineEvent::Changed);
events.extend(self.start_stage(cx, stage + 1, snapshots, avoid));
events
}
/// Issue the next /job poll if the current stage is waiting on one.
@ -3222,7 +3394,7 @@ impl Pipeline {
);
};
if let Err(error) =
makepad_asset_ai::client::verify_artifact_bytes(&bytes, &artifact)
makepad_ai_hub::client::verify_artifact_bytes(&bytes, &artifact)
{
return self.candidate_failed(
cx,
@ -3342,7 +3514,8 @@ impl Pipeline {
if Self::is_vram_admission_error(&message) {
return self.wait_after_vram_rejection(stage, &message, events);
}
return self.fail_stage(stage, message, events);
return self
.fail_stage_or_skip_expander(cx, stage, message, snapshots, avoid, events);
}
}
}
@ -3352,9 +3525,12 @@ impl Pipeline {
.and_then(|r| r.get_string_body())
.is_some_and(|body| body.contains("no such job"))
{
return self.fail_stage(
return self.fail_stage_or_skip_expander(
cx,
stage,
"box lost the job (service restarted or the job expired)".to_string(),
snapshots,
avoid,
events,
);
}
@ -3415,9 +3591,12 @@ impl Pipeline {
.filter(|r| !failed && r.status_code == 200)
.and_then(|r| r.body.clone());
let Some(bytes) = bytes else {
return self.fail_stage(
return self.fail_stage_or_skip_expander(
cx,
stage,
format!("artifact {} fetch failed", artifact.id),
snapshots,
avoid,
events,
);
};
@ -3642,6 +3821,22 @@ impl Pipeline {
state,
elapsed
));
// WHAT THIS STAGE WAS HANDED, in full, indented under it. The
// music stage's prompt carries its lyrics and a video brief is a
// paragraph, so this is the one place the composed text can
// actually be read — the panel it lives in scrolls.
if !stage.sent_prompt.is_empty() {
out.push_str(" prompt sent:\n");
for line in stage.sent_prompt.lines() {
out.push_str(&format!(" {line}\n"));
}
}
if !stage.sent_params.is_empty() {
out.push_str(" params:\n");
for line in stage.sent_params.lines() {
out.push_str(&format!(" {line}\n"));
}
}
}
out
}
@ -3752,10 +3947,10 @@ mod tests {
}
fn image_snapshot(url: &str, node_key: &str) -> BoxSnapshot {
use makepad_asset_ai::protocol::{HealthJson, ModelInfoJson, MODEL_STATE_LOADED};
use makepad_ai_hub::protocol::{HealthJson, ModelInfoJson, MODEL_STATE_LOADED};
BoxSnapshot {
base_url: url.to_string(),
health: Some(HealthJson {
health: Some(HealthJson { realtime: None,
service: "test".to_string(),
version: "1".to_string(),
gpu: Some("GPU".to_string()),
@ -3797,10 +3992,10 @@ mod tests {
}
fn text_snapshot(url: &str, models: &[(&str, &str)]) -> BoxSnapshot {
use makepad_asset_ai::protocol::{HealthJson, ModelInfoJson, MODEL_STATE_LOADED};
use makepad_ai_hub::protocol::{HealthJson, ModelInfoJson, MODEL_STATE_LOADED};
BoxSnapshot {
base_url: url.to_string(),
health: Some(HealthJson {
health: Some(HealthJson { realtime: None,
service: "test".to_string(),
version: "1".to_string(),
gpu: Some("24 GB GPU".to_string()),
@ -3854,7 +4049,7 @@ mod tests {
#[test]
fn qwen38_expand_preference_is_ready_gated_and_falls_back() {
use makepad_asset_ai::protocol::{
use makepad_ai_hub::protocol::{
MODEL_STATE_ABSENT, MODEL_STATE_DOWNLOADING, MODEL_STATE_LOADED, MODEL_STATE_READY,
};
@ -4174,6 +4369,78 @@ mod tests {
}));
}
/// AN EXPANSION CAN NEVER LOSE A RUN. The expander is a rewording
/// courtesy; when it comes back with nothing the video still gets made,
/// from the words the person typed.
#[test]
fn a_useless_expansion_falls_back_to_the_prompt_the_person_typed() {
let mut pipeline = Pipeline::new(
"scanning electron microscope art",
&["text", "video"],
&[],
vec![],
None,
None,
GenParams::default(),
);
assert!(pipeline.expander_is_optional(0));
// Answered with nothing at all.
assert_eq!(
pipeline.request_for_stage(1).unwrap().prompt.as_deref(),
Some("scanning electron microscope art")
);
// Answered with whitespace.
put_output(&mut pipeline, 0, "text/plain; charset=utf-8", b" \n ");
assert_eq!(
pipeline.request_for_stage(1).unwrap().prompt.as_deref(),
Some("scanning electron microscope art")
);
// A real expansion is still what wins when there is one.
pipeline.stages[0].outputs.clear();
put_output(
&mut pipeline,
0,
"text/plain; charset=utf-8",
b"a false-colour scanning electron micrograph of a pollen grain",
);
assert_eq!(
pipeline.request_for_stage(1).unwrap().prompt.as_deref(),
Some("a false-colour scanning electron micrograph of a pollen grain")
);
}
/// The two chains where the expansion is NOT optional keep refusing: a
/// text-only run has no other product, and a character chain's later
/// stages are gated on the brief holding the named identity.
#[test]
fn an_expansion_that_is_the_product_still_refuses_to_be_skipped() {
let text_only = Pipeline::new(
"scanning electron microscope art",
&["text"],
&[],
vec![],
None,
None,
GenParams::default(),
);
assert!(!text_only.expander_is_optional(0));
let mut character = Pipeline::new(
"Boba Fett",
&["text", "image", "mesh", "rig", "motion"],
&[],
vec![],
None,
None,
GenParams::default(),
);
assert!(character.is_character_pipeline());
assert!(!character.expander_is_optional(0));
put_output(&mut character, 0, "text/plain; charset=utf-8", b" ");
assert!(character.request_for_stage(1).is_err());
}
#[test]
fn expanded_music_routes_description_and_lyrics_to_distinct_fields() {
let mut pipeline = Pipeline::new(
@ -4633,13 +4900,13 @@ Arrangement: Pulsing bass, gated drums and widening analog pads."
.contains("dropped identity anchor"));
}
fn registry() -> makepad_asset_ai::registry::Registry {
fn registry() -> makepad_ai_hub::registry::Registry {
let text = std::fs::read_to_string(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../libs/asset/ai/registry.json"
"/../../libs/ai/hub/registry.json"
))
.expect("registry.json readable");
makepad_asset_ai::registry::Registry::parse(&text).expect("registry parses")
makepad_ai_hub::registry::Registry::parse(&text).expect("registry parses")
}
#[test]
@ -4717,6 +4984,9 @@ Arrangement: Pulsing bass, gated drums and widening analog pads."
("speech", "kokoro"),
("text", "qwen3.8-27b"),
("upscale", "realesrgan-x4plus"),
("vision", "qwen3.8-27b-vision"),
("video", "fasth3-4step"),
("video", "fasth3-4step-q4-24g"),
("video", "minimax-h3"),
("video", "minimax-h3-bf16-96g"),
("video", "minimax-h3-nvfp4-32g"),

View file

@ -0,0 +1,531 @@
//! The RUNS chip, the panel behind it, and the ONE card every spawned unit
//! of work is drawn as.
//!
//! All generation runs in THIS app now (aicore §9): the engine in
//! `pipeline.rs` talks to fleet boxes directly, and the store no longer has
//! a queue to poll. So this module owns [`RunCard`] — the card grammar of
//! F1 §5.7 (a title row, ONE aggregate bar, one compact stage strip, and a
//! fold holding the whole truth: sent prompts, params, box tags, errors) —
//! built from the local engine's runs and its waiting queue.
//!
//! The bar is never computed here: [`aggregate_permille`] is the client
//! crate's one implementation. What IS held here is the per-card high-water
//! mark — a stage retry legitimately re-starts one stage's bar, and a bar
//! that goes backwards reads as a bug even when it is honest.
use makepad_asset_client::{aggregate_permille, default_stage_weight};
use std::collections::HashMap;
use crate::pipeline::{format_clock, stage_display_name, Pipeline, StageState};
/// Longest prompt excerpt a card title carries.
const EXCERPT: usize = 60;
// ---------------------------------------------------------------------------
// The card grammar
// ---------------------------------------------------------------------------
/// Which spawned unit a card is; the panel's fold/cancel state keys on it.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum CardKey {
/// A run of this app's own engine, by run id.
Local(u64),
/// A run waiting in this app's queue, by queue position.
LocalQueued(usize),
}
impl CardKey {
pub fn as_text(&self) -> String {
match self {
Self::Local(id) => format!("local:{id}"),
Self::LocalQueued(index) => format!("queued:{index}"),
}
}
}
/// The five states a spawned unit reads as, in the order a list sorts them.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum CardState {
Running,
Queued,
Done,
Failed,
Cancelled,
}
impl CardState {
/// Running work first, then what is waiting, then what is over.
fn rank(self) -> u8 {
match self {
Self::Running => 0,
Self::Queued => 1,
Self::Done | Self::Failed | Self::Cancelled => 2,
}
}
pub fn is_terminal(self) -> bool {
matches!(self, Self::Done | Self::Failed | Self::Cancelled)
}
/// The state dot's colour. ONE accent (#3d9bf0) and it belongs to the
/// thing that is alive; everything else is grey, and only a failure
/// earns red. The state WORD is always on the card too (row 2), so the
/// dot is never the only signal.
pub fn dot(self) -> [f32; 4] {
match self {
Self::Running => [0.239, 0.608, 0.941, 1.0],
Self::Queued => [0.29, 0.32, 0.36, 1.0],
Self::Done => [0.35, 0.42, 0.48, 1.0],
Self::Failed => [0.851, 0.345, 0.310, 1.0],
Self::Cancelled => [0.29, 0.32, 0.36, 1.0],
}
}
/// The one bar's fill. Same hue family as the dot, for the same reason.
pub fn fill(self) -> [f32; 4] {
match self {
Self::Running => [0.239, 0.608, 0.941, 1.0],
Self::Queued => [0.20, 0.28, 0.36, 1.0],
Self::Done => [0.173, 0.373, 0.533, 1.0],
Self::Failed => [0.851, 0.345, 0.310, 1.0],
Self::Cancelled => [0.29, 0.32, 0.36, 1.0],
}
}
}
/// One chip of the stage strip. No bar — row 2 is the only bar on the card.
#[derive(Clone, Debug, PartialEq)]
pub struct StageChip {
/// `expand · 15s`, `music · 62%`, `publish`.
pub text: String,
pub tone: StageTone,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum StageTone {
Pending,
Running,
Done,
Failed,
/// An `on_fail: skip` expander that failed — the run went on with the
/// raw prompt, and that is a fact worth its own colour.
Skipped,
}
impl StageTone {
pub fn color(self) -> [f32; 4] {
match self {
Self::Pending => [0.35, 0.38, 0.42, 1.0],
Self::Running => [0.239, 0.608, 0.941, 1.0],
Self::Done => [0.60, 0.64, 0.69, 1.0],
Self::Failed => [0.851, 0.345, 0.310, 1.0],
Self::Skipped => [0.85, 0.65, 0.35, 1.0],
}
}
}
/// ONE spawned unit of work, however it was spawned. F1 §5.7 verbatim:
/// row 1 title, row 2 the only bar, row 3 the stage strip, row 4 the fold.
#[derive(Clone, Debug, PartialEq)]
pub struct RunCard {
pub key: CardKey,
pub state: CardState,
/// Where this ran: the store, or this app's own engine.
pub origin: &'static str,
/// The preset / pipeline label.
pub label: String,
/// The words the PERSON typed, quoted and truncated. Never the
/// expanded prompt — that lives in the fold.
pub excerpt: String,
/// `m:ss`, live while running, frozen when terminal. This REPLACES the
/// "Done in 538.3s" banner.
pub elapsed: String,
/// The aggregate, 0..=1000, already high-water clamped.
pub permille: u16,
/// The humanized current-stage word, or the failure reason.
pub status: String,
/// Empty for a single-stage unit — a heading with nothing under it does
/// not render, and neither does a strip of one.
pub stages: Vec<StageChip>,
/// The whole truth, only when the card is open.
pub fold: String,
/// Whether the × is offered (pending / running / publishing).
pub can_cancel: bool,
/// Whether this queued LOCAL run can still be moved up the queue.
pub can_promote: bool,
pub open: bool,
/// Newest first inside a state bucket.
created_ms: u64,
}
impl RunCard {
pub fn percent(&self) -> u32 {
(self.permille as u32 + 5) / 10
}
pub fn fraction(&self) -> f32 {
(self.permille as f32 / 1000.0).clamp(0.0, 1.0)
}
/// `RUNS · 3 running · 61%` needs to know which cards are still work.
pub fn is_active(&self) -> bool {
!self.state.is_terminal()
}
}
/// A run of this app's own engine, lent to the card builder.
pub struct LocalRun<'a> {
pub id: u64,
pub label: &'a str,
pub prompt: &'a str,
pub created_ms: u64,
pub pipeline: &'a Pipeline,
}
/// A run waiting in this app's queue — spawned, visible, cancellable, but
/// nothing has been sent yet.
pub struct LocalQueued<'a> {
pub index: usize,
pub label: &'a str,
pub prompt: &'a str,
}
// ---------------------------------------------------------------------------
// Text
// ---------------------------------------------------------------------------
fn excerpt(prompt: &str) -> String {
let one_line = prompt.replace('\n', " ");
let trimmed = one_line.trim();
if trimmed.is_empty() {
return String::new();
}
format!("\u{201c}{}\u{201d}", crate::store_views::truncate(trimmed, EXCERPT))
}
/// The job-kind spelling a local stage domain corresponds to, so the ONE
/// shared weight table (`default_stage_weight`) covers local runs too and a
/// local card's bar cannot mean something different from a store card's.
pub fn local_stage_kind(domain: &str) -> &'static str {
match domain {
"text" => "text.expand",
"image" => "image.generate",
"edit" => "image.edit",
"inpaint" => "image.inpaint",
"control" => "image.control",
"upscale" => "image.upscale",
"video" => "video.generate",
"enhance" => "video.enhance",
"music" | "speech" | "sfx" => "music.generate",
"mesh" | "splat" | "paint" | "rig" => "mesh.generate",
_ => "",
}
}
fn local_stage_weight(domain: &str) -> u16 {
default_stage_weight(local_stage_kind(domain))
}
// ---------------------------------------------------------------------------
// Cards from this app's own engine
// ---------------------------------------------------------------------------
/// The local engine's per-stage share of the bar, in the same 0..=1000 the
/// store speaks — so `aggregate_permille` can weigh them together.
///
/// A FAILED stage keeps the fraction it died at. The old rendering filled it
/// to 1.0 and drew "100% · FAILED" beside it, which is the exact lie this
/// design exists to remove.
fn local_stage_permille(state: &StageState, progress: f64) -> u16 {
match state {
StageState::Done | StageState::AwaitingChoice => 1000,
_ => (progress.clamp(0.0, 1.0) * 1000.0) as u16,
}
}
fn local_chip(pipeline: &Pipeline, index: usize) -> StageChip {
let stage = &pipeline.stages[index];
let name = stage.domain.clone();
let (tone, tail) = match &stage.state {
StageState::Waiting => (StageTone::Pending, String::new()),
StageState::Failed(_) => (StageTone::Failed, String::new()),
StageState::Done | StageState::AwaitingChoice => (
StageTone::Done,
match (stage.started, stage.finished) {
(Some(t0), Some(t1)) => format!(" \u{b7} {}", format_clock((t1 - t0).as_secs_f64())),
_ => String::new(),
},
),
_ => (
StageTone::Running,
format!(" \u{b7} {}%", (stage.progress.clamp(0.0, 1.0) * 100.0).round() as u32),
),
};
StageChip { text: format!("{name}{tail}"), tone }
}
fn local_status(pipeline: &Pipeline) -> String {
if let Some((index, error)) = pipeline.stages.iter().enumerate().find_map(|(i, s)| match &s.state
{
StageState::Failed(error) => Some((i, error.clone())),
_ => None,
}) {
return format!(
"failed at {} \u{2014} {}",
pipeline.stages[index].domain,
crate::store_views::truncate(error.trim(), 120)
);
}
if !pipeline.is_running() {
return "done".to_string();
}
let stage = &pipeline.stages[pipeline.current.min(pipeline.stages.len() - 1)];
match &stage.state {
StageState::Waiting if !stage.detail.is_empty() => stage.detail.clone(),
StageState::Waiting => "queued".to_string(),
StageState::FanOut | StageState::AwaitingChoice => stage.detail.clone(),
StageState::Submitting => "submitting".to_string(),
StageState::Polling if !stage.detail.is_empty() => stage.detail.clone(),
StageState::Polling if !stage.service_state.is_empty() => stage.service_state.clone(),
StageState::Polling => "rendering".to_string(),
StageState::Fetching => "fetching artifacts".to_string(),
StageState::Done => "done".to_string(),
StageState::Failed(error) => error.clone(),
}
}
/// The fold of a local run: every stage's inspect block, the SAME text the
/// RUNS surface's opened stage rows already show (`store_views::stage_detail`
/// is the one implementation), with the routing reasoning this engine knows
/// and nothing else does.
fn local_fold(run: &LocalRun) -> String {
let mut out = format!("PROMPT\n{}\n", run.prompt);
for (index, stage) in run.pipeline.stages.iter().enumerate() {
out.push_str(&format!(
"\n\u{2500}\u{2500} {} \u{b7} {} \u{b7} {}\n",
index + 1,
stage_display_name(&stage.domain),
match &stage.state {
StageState::Failed(_) => "failed",
StageState::Done => "succeeded",
StageState::Waiting => "pending",
_ => "running",
}
));
if !stage.box_url.is_empty() {
out.push_str(&format!(
"MODEL\n{} @ {}\n\n",
stage.model,
stage.box_url.trim_start_matches("http://")
));
}
if !stage.reason.is_empty() {
out.push_str(&format!("ROUTED\n{}\n\n", stage.reason));
}
out.push_str(&crate::store_views::stage_detail(stage));
out.push_str("\n\n");
if let StageState::Failed(error) = &stage.state {
out.push_str(&format!("ERROR\n{error}\n\n"));
}
}
while out.ends_with('\n') {
out.pop();
}
out
}
fn local_card(run: &LocalRun, open: bool) -> RunCard {
let failed = run
.pipeline
.stages
.iter()
.any(|s| matches!(s.state, StageState::Failed(_)));
let state = if failed {
CardState::Failed
} else if run.pipeline.is_running() {
if run
.pipeline
.stages
.iter()
.all(|s| s.state == StageState::Waiting)
{
CardState::Queued
} else {
CardState::Running
}
} else {
CardState::Done
};
let permille = aggregate_permille(run.pipeline.stages.iter().map(|stage| {
(
local_stage_weight(&stage.domain),
local_stage_permille(&stage.state, stage.progress),
)
}));
let elapsed: f64 = run
.pipeline
.stages
.iter()
.filter_map(|s| match (s.started, s.finished) {
(Some(t0), Some(t1)) => Some((t1 - t0).as_secs_f64()),
(Some(t0), None) => Some(t0.elapsed().as_secs_f64()),
_ => None,
})
.sum();
RunCard {
key: CardKey::Local(run.id),
state,
origin: "LOCAL",
label: run.label.to_string(),
excerpt: excerpt(run.prompt),
elapsed: format_clock(elapsed),
permille,
status: local_status(run.pipeline),
stages: if run.pipeline.stages.len() > 1 {
(0..run.pipeline.stages.len())
.map(|index| local_chip(run.pipeline, index))
.collect()
} else {
Vec::new()
},
fold: if open { local_fold(run) } else { String::new() },
can_cancel: run.pipeline.is_running(),
can_promote: false,
open,
created_ms: run.created_ms,
}
}
fn queued_card(queued: &LocalQueued, open: bool) -> RunCard {
RunCard {
key: CardKey::LocalQueued(queued.index),
state: CardState::Queued,
origin: "LOCAL",
label: queued.label.to_string(),
excerpt: excerpt(queued.prompt),
elapsed: String::new(),
permille: 0,
status: format!("waiting for a free slot \u{b7} #{}", queued.index + 1),
stages: Vec::new(),
fold: if open {
format!(
"not sent yet \u{2014} this run has not started\n\nPROMPT\n{}",
queued.prompt
)
} else {
String::new()
},
can_cancel: true,
can_promote: queued.index > 0,
open,
created_ms: 0,
}
}
// ---------------------------------------------------------------------------
// The chip line
// ---------------------------------------------------------------------------
/// The header chip beside SEARCHABLE. Always present: "nothing is running"
/// is itself an answer, and a chip that disappears cannot be clicked to see
/// what just finished.
pub fn chip_text(cards: &[RunCard]) -> String {
let running = cards
.iter()
.filter(|card| card.state == CardState::Running)
.count();
let queued = cards
.iter()
.filter(|card| card.state == CardState::Queued)
.count();
if running == 0 && queued == 0 {
return "RUNS \u{b7} idle".to_string();
}
// The percent covers everything still owed, queued runs included at 0 —
// "how far is the work I fired off", not "how far is the busiest box".
let permille = aggregate_permille(
cards
.iter()
.filter(|card| card.is_active())
.map(|card| (1u16, card.permille)),
);
if running == 0 {
return format!("RUNS \u{b7} {queued} queued");
}
let mut line = format!("RUNS \u{b7} {running} running");
if queued > 0 {
line.push_str(&format!(" \u{b7} {queued} queued"));
}
line.push_str(&format!(" \u{b7} {}%", (permille as u32 + 5) / 10));
line
}
// ---------------------------------------------------------------------------
// The chip state
// ---------------------------------------------------------------------------
/// Everything the app knows about work in flight — all of it local now.
#[derive(Default)]
pub struct RunsChip {
/// The cards the person has unfolded.
open: Vec<CardKey>,
/// Per-card high-water mark. A stage retry honestly restarts one
/// stage's bar; a bar that walks backwards reads as a broken app.
high_water: HashMap<CardKey, u16>,
pub panel_open: bool,
}
impl RunsChip {
pub fn set_panel_open(&mut self, open: bool) {
self.panel_open = open;
}
pub fn is_open(&self, key: &CardKey) -> bool {
self.open.contains(key)
}
/// Unfold / refold one card.
pub fn toggle_open(&mut self, key: &CardKey) {
match self.open.iter().position(|held| held == key) {
Some(at) => {
self.open.remove(at);
}
None => self.open.push(key.clone()),
}
}
/// Every spawned unit, one card each, in the one order: what is running,
/// what is waiting, what is over — newest first inside each.
pub fn cards(&mut self, local: &[LocalRun], queued: &[LocalQueued]) -> Vec<RunCard> {
let mut cards = Vec::new();
for run in local {
let key = CardKey::Local(run.id);
cards.push(local_card(run, self.is_open(&key)));
}
for entry in queued {
let key = CardKey::LocalQueued(entry.index);
cards.push(queued_card(entry, self.is_open(&key)));
}
// The high-water mark, held per card: a stage that retries restarts
// its own bar honestly, and the aggregate must still not walk back.
for card in &mut cards {
let seen = self.high_water.entry(card.key.clone()).or_insert(0);
*seen = (*seen).max(card.permille);
card.permille = *seen;
}
let live: Vec<CardKey> = cards.iter().map(|card| card.key.clone()).collect();
self.high_water.retain(|key, _| live.contains(key));
cards.sort_by(|a, b| {
a.state
.rank()
.cmp(&b.state.rank())
.then(b.created_ms.cmp(&a.created_ms))
});
cards
}
}

View file

@ -15,7 +15,8 @@ use crate::library::LibraryMeta;
use crate::pipeline::{
format_clock, format_music_duration, stage_display_name, CandidateSet, Pipeline, StageState,
};
use makepad_asset_ai::fleet::BoxSnapshot;
use crate::runs_chip::{CardKey, RunCard};
use makepad_ai_hub::fleet::BoxSnapshot;
use makepad_asset_widgets::{AssetThumb, ThumbMedia};
use makepad_widgets::*;
use std::collections::{HashMap, HashSet};
@ -1541,6 +1542,18 @@ pub enum RowAction {
StopImport,
/// Drop a waiting import by queue id.
RemoveQueuedImport(u64),
/// Open (or close) stage `index` of run `id` to read what it was given.
ToggleStage(u64, usize),
/// Put that stage's full input on the clipboard.
CopyStage(u64, usize),
/// Unfold (or refold) one run card.
ToggleCard(CardKey),
/// Stop the spawned unit this card is, whichever engine holds it.
CancelCard(CardKey),
/// Put the card's whole fold on the clipboard.
CopyCard(CardKey),
/// Move a queued local run one place up its queue.
PromoteCard(CardKey),
}
/// One concurrent pipeline run as the Runs surface renders it.
@ -1563,6 +1576,14 @@ pub enum StoreRow {
progress: f32,
failed: bool,
cancel: Option<RowAction>,
/// What this stage was HANDED — the full prompt text and the
/// parameters beside it — shown while the row is open. Empty when
/// the row is closed, so a closed list stays a list.
detail: String,
/// Opening/closing this row, when there is something to open.
expand: Option<RowAction>,
/// Copying `detail` to the clipboard, when the row is open.
copy: Option<RowAction>,
},
/// App-side queued run (waiting for the active pipeline to finish).
Queued { title: String, cancel: RowAction },
@ -1585,6 +1606,12 @@ pub enum StoreRow {
},
/// Honest big empty-state block for missing server data.
Disconnected { title: String, detail: String },
/// ONE spawned unit of work in the one card grammar (`runs_chip.rs`) —
/// a store pipeline, a standalone store job, or a run of this app's own
/// engine. Boxed: a card is much larger than every other row, and a
/// `Vec<StoreRow>` should not pay for that on rows that are three
/// strings.
Card(Box<RunCard>),
}
/// PortalList renderer over a caller-provided `Vec<StoreRow>`. One widget
@ -1657,10 +1684,21 @@ impl Widget for StoreListPanel {
progress,
failed,
cancel,
detail,
copy,
..
} => {
let item = list.item(cx, item_id, id!(StageR));
item.label(cx, ids!(stage_title)).set_text(cx, &title);
item.button(cx, ids!(stage_title)).set_text(cx, &title);
item.label(cx, ids!(stage_meta)).set_text(cx, &meta);
// The detail block only exists while the row is
// open; a closed row is exactly the row it was.
let detail_label = item.label(cx, ids!(stage_detail));
detail_label.set_visible(cx, !detail.is_empty());
if !detail.is_empty() {
detail_label.set_text(cx, &detail);
}
item.button(cx, ids!(stage_copy)).set_visible(cx, copy.is_some());
let bar = item.view(cx, ids!(stage_bar));
bar.set_uniform(cx, live_id!(progress), &[progress]);
let fill: [f32; 4] = if failed {
@ -1729,6 +1767,11 @@ impl Widget for StoreListPanel {
item.label(cx, ids!(disc_detail)).set_text(cx, &detail);
item
}
StoreRow::Card(card) => {
let item = list.item(cx, item_id, id!(CardR));
paint_card(cx, &item, &card);
item
}
};
item.draw_all_unscoped(cx);
}
@ -1760,6 +1803,76 @@ pub fn format_bytes(bytes: u64) -> String {
}
}
/// The stage-strip chips a card body holds, and the label inside each. Eight
/// is the store's own stage ceiling; a longer local chain folds its tail
/// into the last chip rather than growing the row.
const CARD_CHIPS: [&[LiveId]; 8] = [
ids!(cs0),
ids!(cs1),
ids!(cs2),
ids!(cs3),
ids!(cs4),
ids!(cs5),
ids!(cs6),
ids!(cs7),
];
const CARD_CHIP_LABELS: [&[LiveId]; 8] = [
ids!(cs0.cs_label),
ids!(cs1.cs_label),
ids!(cs2.cs_label),
ids!(cs3.cs_label),
ids!(cs4.cs_label),
ids!(cs5.cs_label),
ids!(cs6.cs_label),
ids!(cs7.cs_label),
];
/// Draw ONE card into one `RunCardBody` instance — the same function for a
/// row of the RUNS panel's list and for a slot on the Create surface, which
/// is what makes them the same card rather than two things that look alike.
pub fn paint_card(cx: &mut Cx, item: &WidgetRef, card: &RunCard) {
item.label(cx, ids!(card_label)).set_text(cx, &card.label);
item.label(cx, ids!(card_excerpt)).set_text(cx, &card.excerpt);
item.label(cx, ids!(card_time)).set_text(cx, &card.elapsed);
item.label(cx, ids!(card_pct))
.set_text(cx, &format!("{}%", card.percent()));
item.label(cx, ids!(card_status)).set_text(cx, &card.status);
item.view(cx, ids!(card_dot))
.set_uniform(cx, live_id!(tone), &card.state.dot());
// THE bar — the only one on the card.
let bar = item.view(cx, ids!(card_bar));
bar.set_uniform(cx, live_id!(progress), &[card.fraction()]);
bar.set_uniform(cx, live_id!(color_fill), &card.state.fill());
item.button(cx, ids!(card_cancel))
.set_visible(cx, card.can_cancel);
item.button(cx, ids!(card_up))
.set_visible(cx, card.can_promote);
// The stage strip: one chip per stage, one style, never a bar. A single
// stage has no strip — a strip of one is decoration.
item.view(cx, ids!(card_strip))
.set_visible(cx, !card.stages.is_empty());
for (index, chip_id) in CARD_CHIPS.iter().enumerate() {
let chip = item.view(cx, *chip_id);
let Some(stage) = card.stages.get(index) else {
chip.set_visible(cx, false);
continue;
};
chip.set_visible(cx, true);
chip.set_uniform(cx, live_id!(tone), &stage.tone.color());
item.label(cx, CARD_CHIP_LABELS[index])
.set_text(cx, &stage.text);
}
// The fold only EXISTS while the card is open: a closed list stays a
// list, and nothing diagnostic is ever on the face of a card.
let fold = item.label(cx, ids!(card_fold));
fold.set_visible(cx, !card.fold.is_empty());
if !card.fold.is_empty() {
fold.set_text(cx, &card.fold);
}
item.button(cx, ids!(card_copy))
.set_visible(cx, !card.fold.is_empty());
}
pub fn truncate(text: &str, max: usize) -> String {
if text.chars().count() <= max {
text.to_string()
@ -1769,7 +1882,32 @@ pub fn truncate(text: &str, max: usize) -> String {
}
}
fn stage_row(run_id: u64, pipeline: &Pipeline, index: usize) -> StoreRow {
/// What one stage was given, as a person reads it. The prompt first and in
/// FULL — it is the reason the row opens — then the parameters that rode
/// with it.
pub(crate) fn stage_detail(stage: &crate::pipeline::StageRun) -> String {
if stage.sent_prompt.is_empty() && stage.sent_params.is_empty() {
return "not sent yet — this stage has not started".to_string();
}
let mut out = String::new();
if !stage.sent_prompt.is_empty() {
out.push_str("PROMPT SENT\n");
out.push_str(&stage.sent_prompt);
}
if !stage.sent_params.is_empty() {
if !out.is_empty() {
out.push_str("\n\n");
}
out.push_str("PARAMS\n");
out.push_str(&stage.sent_params);
}
if !stage.job_id.is_empty() {
out.push_str(&format!("\n\nJOB\n{}", stage.job_id));
}
out
}
fn stage_row(run_id: u64, pipeline: &Pipeline, index: usize, open: bool) -> StoreRow {
let stage = &pipeline.stages[index];
let where_ = if stage.box_url.is_empty() {
String::new()
@ -1828,12 +1966,22 @@ fn stage_row(run_id: u64, pipeline: &Pipeline, index: usize) -> StoreRow {
.collect::<Vec<_>>()
.join(" · ");
StoreRow::Stage {
title: format!("{} · {}", index + 1, stage_display_name(&stage.domain)),
// The marker says the row has more to say — the same [+]/[] the
// chat tool chips use, so it reads as the same gesture.
title: format!(
"{} {} · {}",
if open { "[-]" } else { "[+]" },
index + 1,
stage_display_name(&stage.domain)
),
meta,
progress: progress as f32,
failed,
cancel: (index == pipeline.current && pipeline.can_cancel_current())
.then_some(RowAction::CancelRun(run_id)),
detail: if open { stage_detail(stage) } else { String::new() },
expand: Some(RowAction::ToggleStage(run_id, index)),
copy: open.then_some(RowAction::CopyStage(run_id, index)),
}
}
@ -1846,6 +1994,11 @@ pub fn runs_rows(
fleet: &[BoxSnapshot],
latency_ms: &[Option<u64>],
store: &AssetStore,
// `open_stages` is `(run id, stage index)` of every stage the person has
// opened. It is held by the APP, not the widget: these rows are rebuilt
// from scratch several times a second, so an open row has to be
// remembered by something that outlives them.
open_stages: &[(u64, usize)],
) -> Vec<StoreRow> {
let mut rows = vec![StoreRow::Section("THIS APP · LOCAL RUNS".into())];
if runs.is_empty() {
@ -1865,7 +2018,8 @@ pub fn runs_rows(
truncate(&run.pipeline.prompt, 72)
)));
for index in 0..run.pipeline.stages.len() {
rows.push(stage_row(run.id, run.pipeline, index));
let open = open_stages.contains(&(run.id, index));
rows.push(stage_row(run.id, run.pipeline, index, open));
}
}
if !queued.is_empty() {
@ -1881,7 +2035,7 @@ pub fn runs_rows(
rows.push(StoreRow::Section("FLEET WORKERS · LAN".into()));
if fleet.is_empty() {
rows.push(StoreRow::Note(
"No GPU boxes on the LAN — start a makepad-asset-ai fleet service.".into(),
"No GPU boxes on the LAN — start a makepad-ai-hub fleet service.".into(),
));
}
// One row per PHYSICAL host; several service instances on one box list
@ -2042,6 +2196,15 @@ pub fn admin_rows(store: &AssetStore) -> Vec<StoreRow> {
.clone()
.or_else(|| event.asset_id.map(|id| id.to_string()))
.or_else(|| event.game_id.map(|id| id.to_string()))
// A finished run's subject is the RUN, and how it ended — the
// namespace alone would say nothing about which one it was.
.or_else(|| {
let pipeline = event.pipeline?;
Some(match event.pipeline_state {
Some(state) => format!("{pipeline} · {}", state.as_str()),
None => pipeline.to_string(),
})
})
.unwrap_or_else(|| event.namespace.clone());
rows.push(StoreRow::Record {
title: format!("#{} · {}", event.seq, event.kind.as_str()),
@ -2211,6 +2374,61 @@ mod tests {
)
}
/// The user's question — "what text went into the music model?" — has
/// an answer in the RUNS list: press the stage, read the whole thing.
#[test]
fn an_opened_stage_shows_the_whole_text_it_sent() {
let mut pipeline = local_pipeline();
let prompt = "warm analog house, 120 bpm\n\n[verse]\nthe city hums at dusk";
pipeline.stages[0].sent_prompt = prompt.to_string();
pipeline.stages[0].sent_params = "model=minimax-music3\nseconds=60".to_string();
let runs = [RunView { id: 7, label: "music", pipeline: &pipeline }];
let store = AssetStore::default();
// Closed, the list is still a list: no row carries the detail.
let closed = runs_rows(&runs, &[], &[], &[], &store, &[]);
assert!(closed.iter().all(|row| !matches!(
row,
StoreRow::Stage { detail, .. } if !detail.is_empty()
)));
assert!(closed.iter().any(|row| matches!(
row,
StoreRow::Stage { title, copy: None, .. } if title.starts_with("[+]")
)));
// Opened, the row carries the text EXACTLY as it was sent — line
// breaks and all, never truncated — plus what rode with it.
let open = runs_rows(&runs, &[], &[], &[], &store, &[(7, 0)]);
let detail = open
.iter()
.find_map(|row| match row {
StoreRow::Stage { detail, .. } if !detail.is_empty() => Some(detail.clone()),
_ => None,
})
.expect("the opened stage");
assert!(detail.contains(prompt), "{detail}");
assert!(detail.contains("seconds=60"), "{detail}");
assert!(open.iter().any(|row| matches!(
row,
StoreRow::Stage { title, copy: Some(RowAction::CopyStage(7, 0)), .. }
if title.starts_with("[-]")
)));
// Only the stage that was opened: opening one row is not opening
// the run.
let opened = open
.iter()
.filter(|row| matches!(row, StoreRow::Stage { detail, .. } if !detail.is_empty()))
.count();
assert_eq!(opened, 1);
// A stage that has not run yet says so instead of showing nothing.
let other = runs_rows(&runs, &[], &[], &[], &store, &[(7, 1)]);
assert!(other.iter().any(|row| matches!(
row,
StoreRow::Stage { detail, .. } if detail.contains("not sent yet")
)));
}
#[test]
fn runs_rows_show_each_concurrent_run_and_honest_server_state() {
let first = local_pipeline();
@ -2234,6 +2452,7 @@ mod tests {
&fleet,
&[None],
&AssetStore::default(),
&[],
);
// One Stage row per stage PER RUN — concurrent runs itemize
// independently instead of collapsing into one "active" pipeline.

View file

@ -819,6 +819,8 @@ impl ThumbnailRenderer {
ThumbnailSubject::Statue(ModelInstance {
model: id,
transform: frame.transform,
tint: vec4(1.0, 1.0, 1.0, 1.0),
color_adjust: vec4(0.0, 1.0, 1.0, 0.0),
dynamic: true,
depth_order: 0.0,
part_poses: Vec::new(),
@ -945,7 +947,7 @@ impl ThumbnailRenderer {
}
let encoded = rgba.and_then(|rgba| {
makepad_asset_ai::testpattern::encode_png_rgba(&rgba, THUMBNAIL_SIZE, THUMBNAIL_SIZE)
makepad_ai_hub::testpattern::encode_png_rgba(&rgba, THUMBNAIL_SIZE, THUMBNAIL_SIZE)
.map_err(|error| log!("thumbnail {file}: PNG encode failed: {error}"))
.ok()
});

20
apps/finance/Cargo.toml Normal file
View file

@ -0,0 +1,20 @@
# Personal finance: a ledger, budgets and reports over a SQLite file, with
# bank-CSV import. The database is the file format; everything the screens
# read is an in-memory cache rebuilt from it.
[package]
name = "makepad-app-finance"
version = "0.1.0"
edition = "2021"
description = "Personal finance: ledger, budgets, reports and CSV import"
license = "MIT OR Apache-2.0"
default-run = "finance"
[[bin]]
name = "finance"
path = "src/main.rs"
[dependencies]
makepad-widgets = { path = "../../widgets" }
mp-theme = { path = "../../libs/mp_theme" }
makepad-sqlite = { path = "../../libs/sqlite_query" }

601
apps/finance/src/chart.rs Normal file
View file

@ -0,0 +1,601 @@
//! The charts, drawn as shaders.
//!
//! One widget, three forms, because a finance dashboard only ever needs
//! three: a series over time (net worth, balance), a comparison across a
//! handful of periods (income against spending), and the same series
//! shrunk into a table row (a sparkline). Each is a handful of instanced
//! quads with an SDF in the pixel shader — no geometry pass, no texture,
//! and the whole chart batches into a few draw calls, which is why a
//! sparkline per row of a scrolling ledger costs nothing.
//!
//! The visual rules come from the dataviz guidance and are deliberate:
//!
//! * **No gridlines by default.** A line over a dark surface reads on its
//! own; a grid competes with it. Two faint rules mark the extremes, and
//! that is all the scale anyone reads off a trend.
//! * **Selective labels.** The first, last and extreme values are labelled
//! — never every point.
//! * **The fill is a gradient to nothing.** A flat fill under a line reads
//! as a solid shape and hides the line; the fade keeps the line the
//! subject.
//! * **Bars sit on the baseline with a rounded top and a 2px gap.** The
//! gap is the surface showing through, which is what separates adjacent
//! bars without a stroke around each one.
use makepad_widgets::*;
/// The area under a series: one quad per sample interval, each shading the
/// slice between the curve and the baseline.
///
/// The top edge is interpolated ACROSS the quad (`top_left` → `top_right`),
/// so a slice is a trapezoid rather than a staircase, and the edge is
/// antialiased against the fill rather than left to the rasteriser.
#[derive(Script, ScriptHook)]
#[repr(C)]
pub struct DrawAreaFill {
#[deref]
draw_super: DrawQuad,
/// Height of the curve at this quad's left edge, 0 = top of the plot.
#[live]
pub top_left: f32,
#[live]
pub top_right: f32,
/// Colour at the curve, fading to fully transparent at the baseline.
#[live]
pub color_top: Vec4f,
/// How far down the fade reaches: 1.0 fades across the whole plot.
#[live(1.0)]
pub fade: f32,
}
/// One segment of the line, as an SDF capsule so the joins are round and
/// the edges are antialiased at any angle.
#[derive(Script, ScriptHook)]
#[repr(C)]
pub struct DrawLineSeg {
#[deref]
draw_super: DrawQuad,
#[live]
pub y0: f32,
#[live]
pub y1: f32,
#[live]
pub color_line: Vec4f,
#[live(2.0)]
pub thickness: f32,
/// A soft outer glow, which is what stops a 2px line looking thin on a
/// dark surface without having to thicken it.
#[live(0.0)]
pub glow: f32,
}
/// A bar, anchored to the baseline with a rounded top.
#[derive(Script, ScriptHook)]
#[repr(C)]
pub struct DrawBar {
#[deref]
draw_super: DrawQuad,
#[live]
pub color_bar: Vec4f,
/// 0 at the baseline, 1 at the top of the plot.
#[live]
pub height_frac: f32,
#[live(4.0)]
pub radius: f32,
/// Bars that hang below the baseline round the other way.
#[live(0.0)]
pub downward: f32,
}
/// A proportion bar — track and fill in one quad.
///
/// Drawn rather than laid out on purpose: a fill sized by layout needs the
/// track's measured width, which does not exist until after a layout pass,
/// so the first frame draws empty bars and every resize needs a re-measure.
/// A shader that takes the fraction as an instance has neither problem, and
/// it is one quad instead of three widgets per row.
#[derive(Script, ScriptHook)]
#[repr(C)]
pub struct DrawMeter {
#[deref]
draw_super: DrawQuad,
/// 0..1 of the track.
#[live]
pub fraction: f32,
#[live]
pub color_track: Vec4f,
#[live]
pub color_fill: Vec4f,
/// A second, dimmer mark on the same track — what was budgeted, or the
/// same period last year. Negative hides it.
#[live(-1.0)]
pub marker: f32,
#[live]
pub color_marker: Vec4f,
}
/// The point at the end of a series — "you are here".
#[derive(Script, ScriptHook)]
#[repr(C)]
pub struct DrawDot {
#[deref]
draw_super: DrawQuad,
#[live]
pub color_dot: Vec4f,
#[live]
pub color_ring: Vec4f,
}
script_mod! {
use mod.prelude.widgets_internal.*
use mod.widgets.*
// The `#[live]` fields on each struct become the shader's instances;
// the script block only carries the pixel program.
set_type_default() do #(DrawAreaFill::script_shader(vm)){
..mod.draw.DrawQuad
pixel: fn() {
// Where the curve sits at this pixel's x, 0 = top of the plot.
let top = mix(self.top_left, self.top_right, self.pos.x)
let y = self.pos.y
let feather = 1.0 / max(self.rect_size.y, 1.0)
let inside = smoothstep(top - feather, top + feather, y)
if inside <= 0.0 {
return #0000
}
// Fade to nothing on the way down, so the fill never reads as
// a solid block and the line stays the subject.
let depth = (y - top) / max(self.fade * (1.0 - top), 0.001)
let strength = clamp(1.0 - depth, 0.0, 1.0)
let alpha = inside * strength * strength * self.color_top.a
return vec4(self.color_top.rgb * alpha, alpha)
}
}
set_type_default() do #(DrawLineSeg::script_shader(vm)){
..mod.draw.DrawQuad
pixel: fn() {
let p = self.pos * self.rect_size
let a = vec2(0.0, self.y0 * self.rect_size.y)
let b = vec2(self.rect_size.x, self.y1 * self.rect_size.y)
// Distance to the segment: the capsule SDF, so joins are round
// and the edge is antialiased at any angle.
let pa = p - a
let ba = b - a
let h = clamp(dot(pa, ba) / max(dot(ba, ba), 0.0001), 0.0, 1.0)
let d = length(pa - ba * h)
let half = self.thickness * 0.5
let line = clamp(1.0 - smoothstep(half - 0.75, half + 0.75, d), 0.0, 1.0)
let halo = clamp(1.0 - smoothstep(half, half + max(self.glow, 0.001), d), 0.0, 1.0)
let alpha = clamp(line + halo * 0.3 * (1.0 - line), 0.0, 1.0) * self.color_line.a
return vec4(self.color_line.rgb * alpha, alpha)
}
}
set_type_default() do #(DrawBar::script_shader(vm)){
..mod.draw.DrawQuad
pixel: fn() {
let size = self.rect_size
let sdf = Sdf2d.viewport(self.pos * size)
// Round the far end only. A bar rounded at both ends reads as a
// floating pill instead of a measurement standing on an axis,
// so the baseline end is pushed outside the quad and clipped
// square. (Sdf2d takes the DIAMETER as its corner argument.)
let r = min(self.radius, size.x * 0.5)
if self.downward > 0.5 {
sdf.box(0.0, 0.0 - r * 2.0, size.x, size.y + r * 2.0, r * 0.5)
} else {
sdf.box(0.0, 0.0, size.x, size.y + r * 2.0, r * 0.5)
}
sdf.fill(self.color_bar)
return sdf.result
}
}
set_type_default() do #(DrawMeter::script_shader(vm)){
..mod.draw.DrawQuad
pixel: fn() {
let size = self.rect_size
let sdf = Sdf2d.viewport(self.pos * size)
let r = size.y * 0.25
sdf.box(0.0, 0.0, size.x, size.y, r)
sdf.fill(self.color_track)
let w = max(self.fraction * size.x, size.y)
sdf.box(0.0, 0.0, w, size.y, r)
sdf.fill(self.color_fill)
if self.marker >= 0.0 {
// A hairline where the target sits, drawn over the fill so
// it reads whether you are under or over it.
let x = self.marker * size.x
sdf.box(x - 1.0, 0.0 - 1.0, 2.0, size.y + 2.0, 0.0)
sdf.fill(self.color_marker)
}
return sdf.result
}
}
set_type_default() do #(DrawDot::script_shader(vm)){
..mod.draw.DrawQuad
pixel: fn() {
let size = self.rect_size
let sdf = Sdf2d.viewport(self.pos * size)
let r = min(size.x, size.y) * 0.5
sdf.circle(size.x * 0.5, size.y * 0.5, r)
sdf.fill(self.color_ring)
sdf.circle(size.x * 0.5, size.y * 0.5, r * 0.42)
sdf.fill(self.color_dot)
return sdf.result
}
}
mod.widgets.MeterBase = #(Meter::register_widget(vm))
mod.widgets.Meter = set_type_default() do mod.widgets.MeterBase{
width: Fill
height: 6
draw_meter +: {
color_track: #x272a35
color_fill: #x5e6ad2
color_marker: #xa2a8b8
}
}
mod.widgets.FinanceChartBase = #(FinanceChart::register_widget(vm))
mod.widgets.FinanceChart = set_type_default() do mod.widgets.FinanceChartBase{
width: Fill
height: Fill
color_line: #x3987e5
color_fill: #x3987e5
color_second: #xd95926
color_axis: #x6b7784
color_rule: #x2a323d
draw_text +: {
color: #x9aa7b4
text_style: theme.font_regular{font_size: 7.5}
}
}
}
/// What a chart is drawing.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum Form {
/// A series over time, filled to the baseline.
Area,
/// Two series compared per period, as paired bars.
Bars,
/// A series with no chrome at all, for a table cell.
Spark,
}
impl Default for Form {
fn default() -> Form {
Form::Area
}
}
/// Padding inside the plot, so a line at the maximum is not clipped by the
/// widget's own edge and labels have somewhere to sit.
const PAD_TOP: f64 = 14.0;
const PAD_BOTTOM: f64 = 18.0;
const PAD_RIGHT: f64 = 54.0;
#[derive(Script, ScriptHook, Widget)]
pub struct FinanceChart {
#[uid]
uid: WidgetUid,
#[walk]
walk: Walk,
#[redraw]
#[live]
draw_area: DrawAreaFill,
#[live]
draw_line: DrawLineSeg,
#[live]
draw_bar: DrawBar,
#[live]
draw_dot: DrawDot,
#[live]
draw_rule: DrawColor,
#[live]
draw_text: DrawText,
#[live]
color_line: Vec4f,
#[live]
color_fill: Vec4f,
#[live]
color_second: Vec4f,
#[live]
color_axis: Vec4f,
#[live]
color_rule: Vec4f,
#[rust]
form: Form,
#[rust]
series: Vec<f64>,
#[rust]
second: Vec<f64>,
/// Labels under the bars, and the value labels' formatter output.
#[rust]
labels: Vec<String>,
#[rust]
value_labels: Vec<(f64, String)>,
#[rust]
area: Area,
}
impl FinanceChart {
pub fn set_area(&mut self, values: &[f64], marks: Vec<(f64, String)>) {
self.form = Form::Area;
self.series = values.to_vec();
self.value_labels = marks;
}
pub fn set_spark(&mut self, values: &[f64]) {
self.form = Form::Spark;
self.series = values.to_vec();
}
pub fn set_bars(&mut self, first: &[f64], second: &[f64], labels: Vec<String>) {
self.form = Form::Bars;
self.series = first.to_vec();
self.second = second.to_vec();
self.labels = labels;
}
/// The value range to draw against.
///
/// Zero is included for bars (a bar chart that does not start at zero
/// lies about proportion) but NOT for a trend line, where the story is
/// the change and a forced zero flattens it into a straight line.
fn bounds(&self) -> (f64, f64) {
let mut low = f64::MAX;
let mut high = f64::MIN;
for value in self.series.iter().chain(self.second.iter()) {
low = low.min(*value);
high = high.max(*value);
}
if low > high {
return (0.0, 1.0);
}
if self.form == Form::Bars {
low = low.min(0.0);
high = high.max(0.0);
}
if (high - low).abs() < 1e-9 {
high = low + 1.0;
}
// A little headroom, so the peak is not welded to the top edge.
let margin = (high - low) * 0.08;
(low - margin, high + margin)
}
}
impl Widget for FinanceChart {
fn draw_walk(&mut self, cx: &mut Cx2d, _scope: &mut Scope, walk: Walk) -> DrawStep {
let rect = cx.walk_turtle(walk);
self.area = Area::Empty;
if rect.size.x < 2.0 || rect.size.y < 2.0 || self.series.len() < 2 {
return DrawStep::done();
}
let spark = self.form == Form::Spark;
let plot = Rect {
pos: dvec2(rect.pos.x, rect.pos.y + if spark { 1.0 } else { PAD_TOP }),
size: dvec2(
rect.size.x - if spark { 0.0 } else { PAD_RIGHT },
rect.size.y - if spark { 2.0 } else { PAD_TOP + PAD_BOTTOM },
),
};
if plot.size.x < 2.0 || plot.size.y < 2.0 {
return DrawStep::done();
}
let (low, high) = self.bounds();
let span = high - low;
let y_of = |value: f64| -> f64 { ((high - value) / span).clamp(0.0, 1.0) };
match self.form {
Form::Area | Form::Spark => {
let count = self.series.len();
let step = plot.size.x / (count - 1) as f64;
if !spark {
// Two faint rules at the extremes: enough scale to read
// against, far less than a grid.
self.draw_rule.color = self.color_rule;
for fraction in [0.0, 1.0] {
self.draw_rule.draw_abs(
cx,
Rect {
pos: dvec2(plot.pos.x, plot.pos.y + plot.size.y * fraction),
size: dvec2(plot.size.x, 1.0),
},
);
}
}
// The fill, one slice per interval.
self.draw_area.color_top = Vec4f {
w: if spark { 0.35 } else { 0.55 },
..self.color_fill
};
self.draw_area.fade = 1.0;
for index in 0..count - 1 {
self.draw_area.top_left = y_of(self.series[index]) as f32;
self.draw_area.top_right = y_of(self.series[index + 1]) as f32;
self.draw_area.draw_abs(
cx,
Rect {
pos: dvec2(plot.pos.x + step * index as f64, plot.pos.y),
size: dvec2(step + 0.5, plot.size.y),
},
);
}
// The line on top.
self.draw_line.color_line = self.color_line;
self.draw_line.thickness = if spark { 1.5 } else { 2.0 };
self.draw_line.glow = if spark { 0.0 } else { 5.0 };
for index in 0..count - 1 {
self.draw_line.y0 = y_of(self.series[index]) as f32;
self.draw_line.y1 = y_of(self.series[index + 1]) as f32;
self.draw_line.draw_abs(
cx,
Rect {
pos: dvec2(plot.pos.x + step * index as f64, plot.pos.y),
size: dvec2(step, plot.size.y),
},
);
}
if !spark {
// "You are here", and the only labelled points: the
// last value, and whatever the caller marked.
let last = *self.series.last().unwrap();
let dot = dvec2(
plot.pos.x + plot.size.x,
plot.pos.y + y_of(last) * plot.size.y,
);
self.draw_dot.color_ring = self.color_line;
self.draw_dot.color_dot = Vec4f { x: 1.0, y: 1.0, z: 1.0, w: 1.0 };
self.draw_dot.draw_abs(
cx,
Rect { pos: dot - dvec2(4.0, 4.0), size: dvec2(8.0, 8.0) },
);
self.draw_text.color = self.color_axis;
for (value, text) in &self.value_labels {
let y = plot.pos.y + y_of(*value) * plot.size.y;
self.draw_text.draw_abs(
cx,
dvec2(plot.pos.x + plot.size.x + 8.0, y - 6.0),
text,
);
}
}
}
Form::Bars => {
let count = self.series.len().max(1);
let slot = plot.size.x / count as f64;
// Two bars per period with a 2px gap between them, and a
// wider gap between periods so the pairs read as pairs.
let gap = 2.0;
let group = (slot - 6.0).max(4.0);
let bar = ((group - gap) * 0.5).max(2.0);
let zero = y_of(0.0);
self.draw_rule.color = self.color_rule;
self.draw_rule.draw_abs(
cx,
Rect {
pos: dvec2(plot.pos.x, plot.pos.y + zero * plot.size.y),
size: dvec2(plot.size.x, 1.0),
},
);
for index in 0..count {
let left = plot.pos.x + slot * index as f64 + (slot - group) * 0.5;
for (offset, value, color) in [
(0.0, self.series.get(index).copied().unwrap_or(0.0), self.color_line),
(
bar + gap,
self.second.get(index).copied().unwrap_or(0.0),
self.color_second,
),
] {
let top = y_of(value);
let baseline = plot.pos.y + zero * plot.size.y;
let height = ((zero - top).abs() * plot.size.y).max(2.0);
let downward = value < 0.0;
// Both directions start ON the baseline, so the two
// series of a pair meet exactly at the axis.
let y = if downward { baseline } else { baseline - height };
self.draw_bar.color_bar = color;
self.draw_bar.downward = if downward { 1.0 } else { 0.0 };
self.draw_bar.draw_abs(
cx,
Rect { pos: dvec2(left + offset, y), size: dvec2(bar, height) },
);
}
// Period labels, thinned out so they never collide.
if let Some(label) = self.labels.get(index) {
let every = ((count as f64 * 42.0) / plot.size.x).ceil().max(1.0) as usize;
if index % every == 0 {
self.draw_text.color = self.color_axis;
self.draw_text.draw_abs(
cx,
dvec2(left, plot.pos.y + plot.size.y + 4.0),
label,
);
}
}
}
}
}
cx.add_aligned_rect_area(&mut self.area, rect);
DrawStep::done()
}
fn handle_event(&mut self, _cx: &mut Cx, _event: &Event, _scope: &mut Scope) {}
}
impl FinanceChartRef {
pub fn set_area(&self, cx: &mut Cx, values: &[f64], marks: Vec<(f64, String)>) {
if let Some(mut inner) = self.borrow_mut() {
inner.set_area(values, marks);
inner.redraw(cx);
}
}
pub fn set_spark(&self, cx: &mut Cx, values: &[f64]) {
if let Some(mut inner) = self.borrow_mut() {
inner.set_spark(values);
inner.redraw(cx);
}
}
pub fn set_bars(&self, cx: &mut Cx, first: &[f64], second: &[f64], labels: Vec<String>) {
if let Some(mut inner) = self.borrow_mut() {
inner.set_bars(first, second, labels);
inner.redraw(cx);
}
}
}
/// A proportion bar. One quad, one instance value.
#[derive(Script, ScriptHook, Widget)]
pub struct Meter {
#[uid]
uid: WidgetUid,
#[walk]
walk: Walk,
#[redraw]
#[live]
draw_meter: DrawMeter,
#[rust]
area: Area,
}
impl Widget for Meter {
fn draw_walk(&mut self, cx: &mut Cx2d, _scope: &mut Scope, walk: Walk) -> DrawStep {
let rect = cx.walk_turtle(walk);
self.draw_meter.draw_abs(cx, rect);
cx.add_aligned_rect_area(&mut self.area, rect);
DrawStep::done()
}
fn handle_event(&mut self, _cx: &mut Cx, _event: &Event, _scope: &mut Scope) {}
}
impl MeterRef {
/// `fraction` is 0..1; `color` overrides the fill (for a status colour);
/// `marker` places the target hairline, or hides it when negative.
pub fn set(&self, cx: &mut Cx, fraction: f64, color: Option<Vec4f>, marker: f64) {
if let Some(mut inner) = self.borrow_mut() {
inner.draw_meter.fraction = fraction.clamp(0.0, 1.0) as f32;
inner.draw_meter.marker = marker as f32;
if let Some(color) = color {
inner.draw_meter.color_fill = color;
}
inner.redraw(cx);
}
}
}

339
apps/finance/src/csv.rs Normal file
View file

@ -0,0 +1,339 @@
//! A CSV reader that survives what banks actually export.
//!
//! RFC 4180 is a page long and describes maybe half of the files a bank
//! will hand you. The rest of this module is the other half: a UTF-8 BOM
//! that would otherwise make the first header `\u{feff}Date`; semicolons
//! because the country uses a comma for decimals; tabs; CRLF; quoted fields
//! with embedded newlines; `""` escapes inside quotes; a preamble of
//! account-header junk above the real header row; and ragged rows.
//!
//! Nothing here allocates per field beyond the field itself, and the whole
//! file is read into memory on purpose — the largest statement export
//! anyone has is a few megabytes, and random access to the rows is what the
//! import preview needs.
/// The delimiter a file turned out to use.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Delimiter {
Comma,
Semicolon,
Tab,
Pipe,
}
impl Delimiter {
pub fn byte(self) -> u8 {
match self {
Delimiter::Comma => b',',
Delimiter::Semicolon => b';',
Delimiter::Tab => b'\t',
Delimiter::Pipe => b'|',
}
}
pub fn label(self) -> &'static str {
match self {
Delimiter::Comma => "Comma",
Delimiter::Semicolon => "Semicolon",
Delimiter::Tab => "Tab",
Delimiter::Pipe => "Pipe",
}
}
pub const ALL: [Delimiter; 4] =
[Delimiter::Comma, Delimiter::Semicolon, Delimiter::Tab, Delimiter::Pipe];
}
/// A parsed file: rows of fields, plus what we had to work out to read it.
#[derive(Clone, Debug)]
pub struct Csv {
pub rows: Vec<Vec<String>>,
pub delimiter: Delimiter,
/// Rows skipped above the header (bank preamble).
pub preamble: usize,
}
impl Csv {
/// The row we believe holds column names.
pub fn header(&self) -> &[String] {
self.rows.first().map(|r| r.as_slice()).unwrap_or(&[])
}
/// Everything below the header.
pub fn records(&self) -> &[Vec<String>] {
self.rows.get(1..).unwrap_or(&[])
}
/// One column of the records, for sniffing formats. Short rows yield
/// an empty string rather than being skipped, so the row index and the
/// value index stay in step.
pub fn column(&self, index: usize) -> impl Iterator<Item = &str> {
self.records()
.iter()
.map(move |row| row.get(index).map(|s| s.as_str()).unwrap_or(""))
}
pub fn width(&self) -> usize {
self.rows.iter().map(|r| r.len()).max().unwrap_or(0)
}
}
/// Split text into rows of fields with a known delimiter.
///
/// The state machine is RFC 4180's, with the tolerances real files need: a
/// quote inside an unquoted field is a literal quote (not an error), a
/// field that never closes its quote ends at end of input, and CRLF, LF and
/// a lone CR all end a row.
pub fn parse_with(text: &str, delimiter: Delimiter) -> Vec<Vec<String>> {
let delim = delimiter.byte() as char;
let mut rows: Vec<Vec<String>> = Vec::new();
let mut row: Vec<String> = Vec::new();
let mut field = String::new();
let mut in_quotes = false;
let mut chars = text.chars().peekable();
// A BOM would otherwise become part of the first header name.
if text.starts_with('\u{feff}') {
chars.next();
}
let mut any = false;
while let Some(ch) = chars.next() {
any = true;
if in_quotes {
if ch == '"' {
if chars.peek() == Some(&'"') {
chars.next();
field.push('"'); // "" is one literal quote
} else {
in_quotes = false;
}
} else {
field.push(ch);
}
continue;
}
match ch {
'"' if field.is_empty() => in_quotes = true,
c if c == delim => {
row.push(std::mem::take(&mut field));
}
'\r' => {
if chars.peek() == Some(&'\n') {
chars.next();
}
row.push(std::mem::take(&mut field));
rows.push(std::mem::take(&mut row));
}
'\n' => {
row.push(std::mem::take(&mut field));
rows.push(std::mem::take(&mut row));
}
c => field.push(c),
}
}
if any && (!field.is_empty() || !row.is_empty()) {
row.push(field);
rows.push(row);
}
// A trailing newline leaves one empty row; so does a blank line in the
// middle of a bank's preamble. Neither is a record.
rows.retain(|row| !(row.len() == 1 && row[0].trim().is_empty()));
rows
}
/// Work out the delimiter by trying each one and asking which gives the
/// most consistent row width.
///
/// Counting occurrences is not enough: a file full of `"Smith, John"`
/// payees has plenty of commas inside quotes, and a German file has both
/// semicolons and commas. Parsing under each candidate and scoring the
/// result is what tells them apart — the right delimiter yields many
/// columns AND the same number in nearly every row.
pub fn sniff_delimiter(text: &str) -> Delimiter {
let sample: String = text.lines().take(30).collect::<Vec<_>>().join("\n");
let mut best = (Delimiter::Comma, -1.0f64);
for candidate in Delimiter::ALL {
let rows = parse_with(&sample, candidate);
if rows.len() < 2 {
continue;
}
let widths: Vec<usize> = rows.iter().map(|r| r.len()).collect();
let modal = modal_width(&widths);
if modal < 2 {
continue; // one column means this character is not the delimiter
}
let consistent =
widths.iter().filter(|w| **w == modal).count() as f64 / widths.len() as f64;
// Consistency first, then column count as the tie-break: a file
// read under the wrong delimiter is ragged, and a file read under
// the right one usually has more columns than a partial split.
let score = consistent * 100.0 + modal as f64;
if score > best.1 {
best = (candidate, score);
}
}
best.0
}
fn modal_width(widths: &[usize]) -> usize {
let mut counts: Vec<(usize, usize)> = Vec::new();
for width in widths {
match counts.iter_mut().find(|(w, _)| w == width) {
Some((_, n)) => *n += 1,
None => counts.push((*width, 1)),
}
}
counts.sort_by_key(|(width, count)| (std::cmp::Reverse(*count), *width));
counts.first().map(|(w, _)| *w).unwrap_or(0)
}
/// Read a file: sniff the delimiter, drop any preamble above the header,
/// and pad ragged rows to the header's width.
///
/// The preamble is the reason this is not two lines. Plenty of banks print
/// "Account: 1234", a blank line and a date range above the actual table;
/// the header is the first row whose width matches the width most rows
/// have. Everything above it is dropped, and remembered so the import
/// screen can say so.
pub fn parse(text: &str) -> Csv {
let delimiter = sniff_delimiter(text);
let mut rows = parse_with(text, delimiter);
let widths: Vec<usize> = rows.iter().map(|r| r.len()).collect();
let modal = modal_width(&widths);
let preamble = rows
.iter()
.position(|row| row.len() == modal && row.iter().any(|f| !f.trim().is_empty()))
.unwrap_or(0);
if preamble > 0 {
rows.drain(..preamble);
}
for row in rows.iter_mut() {
while row.len() < modal {
row.push(String::new());
}
}
Csv { rows, delimiter, preamble }
}
/// Quote a field for writing: only when it has to be, the way every other
/// tool does it, so a round trip through this module is a no-op.
pub fn escape_field(field: &str, delimiter: Delimiter) -> String {
let needs = field.contains(delimiter.byte() as char)
|| field.contains('"')
|| field.contains('\n')
|| field.contains('\r')
|| field.starts_with(' ')
|| field.ends_with(' ');
if !needs {
return field.to_string();
}
let mut out = String::with_capacity(field.len() + 2);
out.push('"');
for ch in field.chars() {
if ch == '"' {
out.push('"');
}
out.push(ch);
}
out.push('"');
out
}
/// Write rows back out as RFC 4180 (CRLF, as the spec says).
pub fn write(rows: &[Vec<String>], delimiter: Delimiter) -> String {
let mut out = String::new();
for row in rows {
for (i, field) in row.iter().enumerate() {
if i > 0 {
out.push(delimiter.byte() as char);
}
out.push_str(&escape_field(field, delimiter));
}
out.push_str("\r\n");
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reads_rfc4180_including_the_awkward_parts() {
let text = "a,b,c\r\n1,\"two, with comma\",3\r\n4,\"say \"\"hi\"\"\",6\r\n";
let csv = parse(text);
assert_eq!(csv.delimiter, Delimiter::Comma);
assert_eq!(csv.header(), ["a", "b", "c"]);
assert_eq!(csv.records()[0][1], "two, with comma");
assert_eq!(csv.records()[1][1], "say \"hi\"");
}
#[test]
fn a_quoted_field_may_contain_a_newline() {
let text = "date,memo\n2024-01-01,\"line one\nline two\"\n";
let csv = parse(text);
assert_eq!(csv.records().len(), 1);
assert_eq!(csv.records()[0][1], "line one\nline two");
}
#[test]
fn strips_the_bom_so_the_first_header_is_usable() {
let text = "\u{feff}Date,Amount\n2024-01-01,10.00\n";
let csv = parse(text);
assert_eq!(csv.header()[0], "Date");
}
#[test]
fn tells_a_semicolon_file_from_a_comma_one() {
// German: semicolon delimited, commas INSIDE the numbers.
let german = "Datum;Beschreibung;Betrag\n04.03.2024;Miete;-1.234,56\n05.03.2024;Lohn;2.500,00\n";
assert_eq!(parse(german).delimiter, Delimiter::Semicolon);
assert_eq!(parse(german).records()[0][2], "-1.234,56");
// Comma delimited with commas inside quoted payees.
let anglo = "Date,Payee,Amount\n2024-03-04,\"Smith, John\",-25.00\n2024-03-05,\"Doe, Jane\",30.00\n";
assert_eq!(parse(anglo).delimiter, Delimiter::Comma);
assert_eq!(parse(anglo).records()[0][1], "Smith, John");
let tabbed = "Date\tPayee\tAmount\n2024-03-04\tRent\t-100\n";
assert_eq!(parse(tabbed).delimiter, Delimiter::Tab);
}
#[test]
fn drops_the_junk_a_bank_prints_above_the_table() {
let text = "Account Statement\n\nAccount:,1234567890\nPeriod:,Jan 2024\n\n\
Date,Description,Amount,Balance\n\
2024-01-02,Coffee,-4.50,995.50\n\
2024-01-03,Salary,2000.00,2995.50\n";
let csv = parse(text);
assert_eq!(csv.header(), ["Date", "Description", "Amount", "Balance"]);
assert_eq!(csv.records().len(), 2);
assert!(csv.preamble > 0);
}
#[test]
fn ragged_rows_are_padded_so_column_access_never_panics() {
let text = "a,b,c\n1,2,3\n4,5\n";
let csv = parse(text);
assert_eq!(csv.records()[1].len(), 3);
assert_eq!(csv.column(2).collect::<Vec<_>>(), ["3", ""]);
}
#[test]
fn round_trips_through_write() {
let rows = vec![
vec!["Date".into(), "Payee".into(), "Amount".into()],
vec!["2024-03-04".into(), "Smith, John".into(), "-25.00".into()],
vec!["2024-03-05".into(), "say \"hi\"".into(), "30.00".into()],
];
let text = write(&rows, Delimiter::Comma);
let back = parse(&text);
assert_eq!(back.rows, rows);
}
#[test]
fn an_empty_or_single_line_file_does_not_panic() {
assert!(parse("").rows.is_empty());
assert_eq!(parse("just one line\n").rows.len(), 1);
}
}

506
apps/finance/src/date.rs Normal file
View file

@ -0,0 +1,506 @@
//! Civil dates as a day number, and the fight to read the ones banks write.
//!
//! A ledger only ever needs whole days: no clocks, no zones, no leap
//! seconds. So a date is an `i32` count of days from 1970-01-01, which
//! sorts, subtracts and indexes into a month bucket without a calendar
//! library, and is four bytes in a row of a hundred thousand.
//!
//! The hard part is not arithmetic, it is `03/04/2024`. That is the 3rd of
//! April in Europe and the 4th of March in America, and the file rarely
//! says which. Guessing per row silently scatters transactions across
//! months. So [`sniff_date_format`] reads the WHOLE column and only then
//! decides — a single row with a day above 12 settles it for every other
//! row, and when nothing settles it the caller is told, so the import
//! screen can ask instead of inventing an answer.
use std::fmt;
/// Days since 1970-01-01. Negative reaches back before it.
pub type Day = i32;
/// Days from the civil date. Howard Hinnant's `days_from_civil`, which is
/// exact for the whole proleptic Gregorian calendar.
pub fn from_ymd(year: i32, month: u32, day: u32) -> Day {
let y = if month <= 2 { year - 1 } else { year };
let era = if y >= 0 { y } else { y - 399 } / 400;
let yoe = (y - era * 400) as i64; // [0, 399]
let mp = ((month as i64 + 9) % 12) as i64; // Mar = 0
let doy = (153 * mp + 2) / 5 + day as i64 - 1; // [0, 365]
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; // [0, 146096]
(era as i64 * 146097 + doe - 719468) as Day
}
/// The civil date of a day number.
pub fn to_ymd(day: Day) -> (i32, u32, u32) {
let z = day as i64 + 719468;
let era = if z >= 0 { z } else { z - 146096 } / 146097;
let doe = z - era * 146097; // [0, 146096]
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365; // [0, 399]
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
let mp = (5 * doy + 2) / 153; // [0, 11], Mar = 0
let d = doy - (153 * mp + 2) / 5 + 1; // [1, 31]
let m = if mp < 10 { mp + 3 } else { mp - 9 }; // [1, 12]
((if m <= 2 { y + 1 } else { y }) as i32, m as u32, d as u32)
}
pub fn year_of(day: Day) -> i32 {
to_ymd(day).0
}
pub fn month_of(day: Day) -> u32 {
to_ymd(day).1
}
/// 0 = Monday. (1970-01-01 was a Thursday.)
pub fn weekday(day: Day) -> u32 {
(day.rem_euclid(7) as u32 + 3) % 7
}
pub fn is_weekend(day: Day) -> bool {
weekday(day) >= 5
}
pub fn days_in_month(year: i32, month: u32) -> u32 {
match month {
1 | 3 | 5 | 7 | 8 | 10 | 12 => 31,
4 | 6 | 9 | 11 => 30,
2 if (year % 4 == 0 && year % 100 != 0) || year % 400 == 0 => 29,
2 => 28,
_ => 30,
}
}
/// First day of the month containing `day`.
pub fn month_start(day: Day) -> Day {
let (y, m, _) = to_ymd(day);
from_ymd(y, m, 1)
}
/// Last day of the month containing `day`.
pub fn month_end(day: Day) -> Day {
let (y, m, _) = to_ymd(day);
from_ymd(y, m, days_in_month(y, m))
}
/// Move whole months, clamping the day of month — 31 January plus one
/// month is 28 February, which is what a monthly bill on the 31st does.
pub fn add_months(day: Day, months: i32) -> Day {
let (y, m, d) = to_ymd(day);
let total = y * 12 + (m as i32 - 1) + months;
let (ny, nm) = (total.div_euclid(12), total.rem_euclid(12) as u32 + 1);
from_ymd(ny, nm, d.min(days_in_month(ny, nm)))
}
/// A month as a sortable integer key, `year * 12 + (month - 1)` — what
/// budgets and monthly rollups are keyed by.
pub type MonthKey = i32;
pub fn month_key(day: Day) -> MonthKey {
let (y, m, _) = to_ymd(day);
y * 12 + (m as i32 - 1)
}
pub fn month_key_start(key: MonthKey) -> Day {
from_ymd(key.div_euclid(12), key.rem_euclid(12) as u32 + 1, 1)
}
pub const MONTH_NAMES: [&str; 12] = [
"January", "February", "March", "April", "May", "June", "July", "August",
"September", "October", "November", "December",
];
pub const MONTH_ABBR: [&str; 12] = [
"Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec",
];
pub const WEEKDAY_ABBR: [&str; 7] = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"];
/// `2024-03-04` — the storage form, and the only unambiguous one.
pub fn format_iso(day: Day) -> String {
let (y, m, d) = to_ymd(day);
format!("{y:04}-{m:02}-{d:02}")
}
/// `4 Mar 2024` — the ledger form: unambiguous to a human, and short.
pub fn format_short(day: Day) -> String {
let (y, m, d) = to_ymd(day);
format!("{d} {} {y}", MONTH_ABBR[(m - 1) as usize])
}
/// `Mar 2024` — column headers on a budget.
pub fn format_month(key: MonthKey) -> String {
let year = key.div_euclid(12);
let month = key.rem_euclid(12) as usize;
format!("{} {year}", MONTH_ABBR[month])
}
/// How the dates in an imported column are written.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)]
pub struct DateFormat {
/// The order of the numeric fields.
pub order: FieldOrder,
/// Two-digit years, which need a century guess.
pub two_digit_year: bool,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)]
pub enum FieldOrder {
/// `2024-03-04`, ISO 8601. The default because it is the only order
/// that cannot be misread.
#[default]
Ymd,
/// `04/03/2024` — most of the world.
Dmy,
/// `03/04/2024` — the United States.
Mdy,
}
/// What a column of dates turned out to be, and whether we are sure.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct DateSniff {
pub format: DateFormat,
/// False when every row was ambiguous (no day above 12 anywhere), so
/// the order is a guess and the import screen must offer the choice.
pub certain: bool,
/// Rows that parsed under the chosen format.
pub parsed: usize,
/// Rows that did not parse at all.
pub failed: usize,
}
/// Read a whole column of dates and work out how it is written.
///
/// The rule that does the work: in `a/b/c`, if any row has `a > 12` the
/// first field must be a day, and if any row has `b > 12` the second must
/// be. One such row decides the column. If none does — a statement whose
/// every transaction lands in the first twelve days of a month — the order
/// stays a guess, `certain` is false, and the caller asks.
pub fn sniff_date_format<'a>(cells: impl Iterator<Item = &'a str>) -> DateSniff {
let mut first_over_12 = false;
let mut second_over_12 = false;
let mut iso = 0usize;
let mut two_digit = 0usize;
let mut numeric = 0usize;
let mut total = 0usize;
let mut samples: Vec<[u32; 3]> = Vec::new();
for cell in cells {
let cell = cell.trim();
if cell.is_empty() {
continue;
}
total += 1;
let Some((fields, year_digits)) = split_numeric_date(cell) else {
// Named-month forms ("4 Mar 2024") are self-describing and
// vote for nothing.
continue;
};
numeric += 1;
if year_digits == 2 {
two_digit += 1;
}
if fields[0] > 31 {
iso += 1; // a 4-digit year leading: 2024-03-04
} else {
if fields[0] > 12 {
first_over_12 = true;
}
if fields[1] > 12 {
second_over_12 = true;
}
samples.push(fields);
}
}
let order = if iso > 0 && iso >= numeric / 2 {
FieldOrder::Ymd
} else if first_over_12 {
FieldOrder::Dmy
} else if second_over_12 {
FieldOrder::Mdy
} else {
// Nothing decisive. Day-first is the world's convention and the
// safer default; `certain: false` is what actually matters here.
FieldOrder::Dmy
};
let format = DateFormat { order, two_digit_year: two_digit > numeric / 2 };
let certain = matches!(order, FieldOrder::Ymd) || first_over_12 || second_over_12;
DateSniff { format, certain, parsed: numeric, failed: total - numeric }
}
/// Split `04/03/2024`, `04-03-2024`, `04.03.2024` into its three numbers,
/// with the digit count of the field that looks like a year.
fn split_numeric_date(text: &str) -> Option<([u32; 3], usize)> {
let head: &str = text.split_whitespace().next().unwrap_or(text);
let mut fields = [0u32; 3];
let mut widths = [0usize; 3];
let mut index = 0usize;
let mut digits = 0usize;
let mut current = 0u32;
for ch in head.chars() {
if let Some(d) = ch.to_digit(10) {
current = current.checked_mul(10)?.checked_add(d)?;
digits += 1;
} else if matches!(ch, '/' | '-' | '.') {
if index >= 2 || digits == 0 {
return None;
}
fields[index] = current;
widths[index] = digits;
index += 1;
current = 0;
digits = 0;
} else {
return None;
}
}
if index != 2 || digits == 0 {
return None;
}
fields[2] = current;
widths[2] = digits;
let year_digits = if widths[0] == 4 { widths[0] } else { widths[2] };
Some((fields, year_digits))
}
/// Two digits to a century: the 69/70 split every system uses, biased so
/// that a statement from '99 is 1999 and one from '24 is 2024.
fn expand_year(year: u32) -> i32 {
if year >= 100 {
year as i32
} else if year >= 70 {
1900 + year as i32
} else {
2000 + year as i32
}
}
/// Parse one cell under a known format. Also understands ISO and named
/// months regardless of `format`, since those are unambiguous.
pub fn parse_date(text: &str, format: DateFormat) -> Option<Day> {
let text = text.trim();
if text.is_empty() {
return None;
}
if let Some((fields, _)) = split_numeric_date(text) {
let (y, m, d) = if fields[0] > 31 {
(fields[0], fields[1], fields[2]) // leading 4-digit year: ISO
} else {
match format.order {
FieldOrder::Ymd => (fields[0], fields[1], fields[2]),
FieldOrder::Dmy => (fields[2], fields[1], fields[0]),
FieldOrder::Mdy => (fields[2], fields[0], fields[1]),
}
};
return valid_ymd(expand_year(y), m, d);
}
parse_named_month(text)
}
/// `4 Mar 2024`, `Mar 4, 2024`, `4 March 2024`, `2024 Mar 4`.
fn parse_named_month(text: &str) -> Option<Day> {
let cleaned: String = text
.chars()
.map(|c| if c == ',' { ' ' } else { c })
.collect();
let mut month = None;
let mut numbers: Vec<u32> = Vec::new();
for word in cleaned.split_whitespace() {
let lower = word.to_ascii_lowercase();
if let Some(index) = MONTH_ABBR
.iter()
.position(|m| lower.starts_with(&m.to_ascii_lowercase()))
{
if month.is_none() {
month = Some(index as u32 + 1);
continue;
}
}
let digits: String = word.chars().filter(|c| c.is_ascii_digit()).collect();
if !digits.is_empty() {
if let Ok(value) = digits.parse::<u32>() {
numbers.push(value);
}
}
}
let month = month?;
if numbers.len() < 2 {
return None;
}
// Whichever number could not be a day is the year.
let (day, year) = if numbers[0] > 31 {
(numbers[1], numbers[0])
} else {
(numbers[0], numbers[1])
};
valid_ymd(expand_year(year), month, day)
}
fn valid_ymd(year: i32, month: u32, day: u32) -> Option<Day> {
if !(1..=12).contains(&month) || day == 0 || day > days_in_month(year, month) {
return None;
}
if !(1900..=2200).contains(&year) {
return None;
}
Some(from_ymd(year, month, day))
}
/// A closed range of days, which is what every report and filter is.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct DateRange {
pub start: Day,
pub end: Day,
}
impl DateRange {
pub fn contains(&self, day: Day) -> bool {
day >= self.start && day <= self.end
}
pub fn days(&self) -> i32 {
self.end - self.start + 1
}
pub fn month(key: MonthKey) -> DateRange {
let start = month_key_start(key);
DateRange { start, end: month_end(start) }
}
/// The last `n` whole months ending with the month of `day`.
pub fn last_months(day: Day, n: i32) -> DateRange {
let end = month_end(day);
let start = month_start(add_months(day, -(n - 1)));
DateRange { start, end }
}
pub fn year(year: i32) -> DateRange {
DateRange { start: from_ymd(year, 1, 1), end: from_ymd(year, 12, 31) }
}
}
impl fmt::Display for DateRange {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{} {}", format_short(self.start), format_short(self.end))
}
}
/// Today, from the system clock. The one place time enters the app.
pub fn today() -> Day {
let secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0);
(secs / 86_400) as Day
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn civil_conversion_round_trips_across_centuries() {
assert_eq!(from_ymd(1970, 1, 1), 0);
assert_eq!(to_ymd(0), (1970, 1, 1));
assert_eq!(from_ymd(2024, 2, 29), 19782);
assert_eq!(to_ymd(19782), (2024, 2, 29));
assert_eq!(from_ymd(1969, 12, 31), -1);
assert_eq!(to_ymd(-1), (1969, 12, 31));
// Every day of a leap year and a century year round trips.
for day in from_ymd(1999, 1, 1)..=from_ymd(2001, 12, 31) {
let (y, m, d) = to_ymd(day);
assert_eq!(from_ymd(y, m, d), day);
}
}
#[test]
fn weekday_and_month_edges() {
assert_eq!(weekday(from_ymd(1970, 1, 1)), 3); // Thursday
assert_eq!(weekday(from_ymd(2024, 3, 4)), 0); // Monday
assert!(is_weekend(from_ymd(2024, 3, 9)));
assert_eq!(month_start(from_ymd(2024, 3, 15)), from_ymd(2024, 3, 1));
assert_eq!(month_end(from_ymd(2024, 2, 15)), from_ymd(2024, 2, 29));
assert_eq!(days_in_month(2023, 2), 28);
assert_eq!(days_in_month(2000, 2), 29);
assert_eq!(days_in_month(1900, 2), 28);
}
#[test]
fn monthly_bills_clamp_to_the_end_of_short_months() {
let jan31 = from_ymd(2024, 1, 31);
assert_eq!(to_ymd(add_months(jan31, 1)), (2024, 2, 29));
assert_eq!(to_ymd(add_months(jan31, 13)), (2025, 2, 28));
assert_eq!(to_ymd(add_months(from_ymd(2024, 3, 15), -3)), (2023, 12, 15));
}
#[test]
fn month_keys_sort_and_invert() {
let key = month_key(from_ymd(2024, 3, 4));
assert_eq!(month_key_start(key), from_ymd(2024, 3, 1));
assert!(month_key(from_ymd(2024, 1, 1)) < month_key(from_ymd(2024, 2, 1)));
assert_eq!(format_month(month_key(from_ymd(2024, 3, 4))), "Mar 2024");
}
#[test]
fn the_ambiguous_column_is_settled_by_one_decisive_row() {
// 13 can only be a day: the whole column is day-first.
let eu = ["04/03/2024", "13/03/2024", "01/04/2024"];
let sniff = sniff_date_format(eu.iter().copied());
assert_eq!(sniff.format.order, FieldOrder::Dmy);
assert!(sniff.certain);
assert_eq!(parse_date("04/03/2024", sniff.format), Some(from_ymd(2024, 3, 4)));
// 13 in the second field: month-first.
let us = ["03/04/2024", "03/13/2024", "04/01/2024"];
let sniff = sniff_date_format(us.iter().copied());
assert_eq!(sniff.format.order, FieldOrder::Mdy);
assert!(sniff.certain);
// The same eight characters, read the other way round: month 03,
// day 04 — which is the whole reason the column has to vote.
assert_eq!(parse_date("03/04/2024", sniff.format), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("04/01/2024", sniff.format), Some(from_ymd(2024, 4, 1)));
// Nothing decisive: we guess, but we SAY we guessed.
let ambiguous = ["03/04/2024", "05/06/2024"];
let sniff = sniff_date_format(ambiguous.iter().copied());
assert!(!sniff.certain);
// ISO needs no guessing.
let iso = ["2024-03-04", "2024-03-13"];
let sniff = sniff_date_format(iso.iter().copied());
assert_eq!(sniff.format.order, FieldOrder::Ymd);
assert!(sniff.certain);
}
#[test]
fn parses_the_forms_banks_write() {
let dmy = DateFormat { order: FieldOrder::Dmy, two_digit_year: false };
assert_eq!(parse_date("04.03.2024", dmy), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("04-03-2024", dmy), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("04/03/24", dmy), Some(from_ymd(2024, 3, 4)));
// ISO and named months parse under any declared order.
assert_eq!(parse_date("2024-03-04", dmy), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("4 Mar 2024", dmy), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("Mar 4, 2024", dmy), Some(from_ymd(2024, 3, 4)));
assert_eq!(parse_date("4 March 2024", dmy), Some(from_ymd(2024, 3, 4)));
// Impossible dates are rejected, not clamped.
assert_eq!(parse_date("31/02/2024", dmy), None);
assert_eq!(parse_date("00/01/2024", dmy), None);
assert_eq!(parse_date("hello", dmy), None);
// Two-digit years split at 70.
assert_eq!(parse_date("01/01/99", dmy), Some(from_ymd(1999, 1, 1)));
assert_eq!(parse_date("01/01/24", dmy), Some(from_ymd(2024, 1, 1)));
}
#[test]
fn ranges_cover_what_reports_ask_for() {
let day = from_ymd(2024, 3, 15);
let last_3 = DateRange::last_months(day, 3);
assert_eq!(last_3.start, from_ymd(2024, 1, 1));
assert_eq!(last_3.end, from_ymd(2024, 3, 31));
assert!(last_3.contains(from_ymd(2024, 2, 29)));
assert!(!last_3.contains(from_ymd(2023, 12, 31)));
assert_eq!(DateRange::year(2024).days(), 366);
}
}

829
apps/finance/src/db.rs Normal file
View file

@ -0,0 +1,829 @@
//! The SQLite file, and the load that turns it into a [`Ledger`].
//!
//! The database is the file format — one `.finance` file you can copy,
//! back up, and open with any SQLite tool, which is the whole reason for
//! choosing it over a private binary. But no screen queries it. Everything
//! is read once into memory ([`Ledger`]) and written through on change,
//! because the queries a finance app makes — "every transaction of this
//! account with its running balance", "spend per category per month for
//! two years" — are passes over a few hundred thousand small rows, and
//! that is microseconds in RAM against milliseconds per round trip in SQL.
//! It is also why this app stays fast where the commercial ones famously
//! do not: the reports never touch the disk.
//!
//! Schema changes go in [`MIGRATIONS`], never by editing [`SCHEMA`]: an
//! existing file must survive an upgrade. `user_version` records how far a
//! file has come.
use crate::date::Day;
use crate::model::*;
use crate::money::{currency_by_code, Currency, USD};
use makepad_sqlite::{Connection, Value};
use std::path::Path;
use std::time::Duration;
/// Bumped whenever [`MIGRATIONS`] grows.
pub const SCHEMA_VERSION: i64 = 1;
const SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS accounts(
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
kind TEXT NOT NULL,
currency TEXT NOT NULL DEFAULT 'USD',
institution TEXT NOT NULL DEFAULT '',
opening_balance INTEGER NOT NULL DEFAULT 0,
opening_date INTEGER NOT NULL DEFAULT 0,
closed INTEGER NOT NULL DEFAULT 0,
off_budget INTEGER NOT NULL DEFAULT 0,
sort_order INTEGER NOT NULL DEFAULT 0,
note TEXT NOT NULL DEFAULT ''
);
CREATE TABLE IF NOT EXISTS categories(
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
parent INTEGER,
kind TEXT NOT NULL DEFAULT 'expense',
budgeted INTEGER NOT NULL DEFAULT 1,
rollover INTEGER NOT NULL DEFAULT 0,
sort_order INTEGER NOT NULL DEFAULT 0,
color INTEGER NOT NULL DEFAULT 0,
hidden INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS transactions(
id INTEGER PRIMARY KEY,
account INTEGER NOT NULL,
date INTEGER NOT NULL,
payee TEXT NOT NULL DEFAULT '',
memo TEXT NOT NULL DEFAULT '',
amount INTEGER NOT NULL,
category INTEGER,
transfer_group INTEGER,
cleared TEXT NOT NULL DEFAULT 'uncleared',
statement INTEGER,
import_hash INTEGER,
reference TEXT NOT NULL DEFAULT '',
flagged INTEGER NOT NULL DEFAULT 0,
notes TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS transactions_account_date ON transactions(account, date);
CREATE INDEX IF NOT EXISTS transactions_date ON transactions(date);
CREATE INDEX IF NOT EXISTS transactions_import_hash ON transactions(import_hash);
CREATE TABLE IF NOT EXISTS splits(
id INTEGER PRIMARY KEY,
txn INTEGER NOT NULL,
category INTEGER,
amount INTEGER NOT NULL,
memo TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS splits_txn ON splits(txn);
CREATE TABLE IF NOT EXISTS payees(
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
default_category INTEGER
);
CREATE TABLE IF NOT EXISTS rules(
id INTEGER PRIMARY KEY,
name TEXT NOT NULL DEFAULT '',
match_on TEXT NOT NULL DEFAULT 'raw',
how TEXT NOT NULL DEFAULT 'contains',
pattern TEXT NOT NULL DEFAULT '',
amount_min INTEGER NOT NULL DEFAULT 0,
amount_max INTEGER NOT NULL DEFAULT 0,
set_category INTEGER,
rename_payee TEXT,
set_memo TEXT,
flag INTEGER NOT NULL DEFAULT 0,
priority INTEGER NOT NULL DEFAULT 0,
enabled INTEGER NOT NULL DEFAULT 1,
hits INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS budgets(
category INTEGER NOT NULL,
month INTEGER NOT NULL,
assigned INTEGER NOT NULL DEFAULT 0,
rollover INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY(category, month)
);
CREATE TABLE IF NOT EXISTS scheduled(
id INTEGER PRIMARY KEY,
account INTEGER NOT NULL,
payee TEXT NOT NULL DEFAULT '',
amount INTEGER NOT NULL DEFAULT 0,
category INTEGER,
recurrence TEXT NOT NULL DEFAULT 'monthly',
next_due INTEGER NOT NULL DEFAULT 0,
last_posted INTEGER,
auto_post INTEGER NOT NULL DEFAULT 0,
enabled INTEGER NOT NULL DEFAULT 1,
detected INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS import_profiles(
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
account INTEGER,
mapping TEXT NOT NULL DEFAULT '',
date_order TEXT NOT NULL DEFAULT 'ymd',
decimal_comma INTEGER NOT NULL DEFAULT 0,
delimiter TEXT NOT NULL DEFAULT ',',
used INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS settings(
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
";
/// Each entry runs once, in order, on a file whose `user_version` is below
/// its index + 1. Append only — never edit one that has shipped.
const MIGRATIONS: [&str; 0] = [];
pub struct Db {
conn: Connection,
}
impl Db {
/// Open (creating if absent) and bring the schema up to date.
pub fn open(path: &Path) -> Result<Db, String> {
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir).map_err(|e| format!("create {}: {e}", dir.display()))?;
}
let mut conn = Connection::open(path, Duration::from_secs(5))
.map_err(|e| format!("open {}: {e:?}", path.display()))?;
// A ledger is one table scan wide, not a hundred: the default row
// budget would refuse a decade of transactions in one query.
conn.limits_mut().max_rows = 5_000_000;
conn.limits_mut().max_steps = 500_000_000;
conn.execute_batch(SCHEMA).map_err(|e| format!("schema: {e:?}"))?;
let mut db = Db { conn };
db.migrate()?;
Ok(db)
}
fn migrate(&mut self) -> Result<(), String> {
let from = self.conn.user_version().max(0) as usize;
for (index, sql) in MIGRATIONS.iter().enumerate().skip(from) {
self.conn
.execute_batch(sql)
.map_err(|e| format!("migration {}: {e:?}", index + 1))?;
}
if from < MIGRATIONS.len() {
self.conn
.execute(&format!("PRAGMA user_version = {}", MIGRATIONS.len()), &[])
.map_err(|e| format!("set user_version: {e:?}"))?;
}
Ok(())
}
/// True for a file with no accounts — a first run, which is what the
/// demo data offers to fill.
pub fn is_empty(&mut self) -> Result<bool, String> {
let result = self
.conn
.query("SELECT COUNT(*) FROM accounts", &[])
.map_err(|e| format!("count accounts: {e:?}"))?;
Ok(result.scalar().and_then(|v| v.as_integer()).unwrap_or(0) == 0)
}
/// Read the whole file into memory. Every screen reads the result;
/// nothing else queries.
pub fn load(&mut self) -> Result<Ledger, String> {
let mut ledger = Ledger { base_currency: self.base_currency(), ..Ledger::default() };
let rows = self
.conn
.query(
"SELECT id, name, kind, currency, institution, opening_balance, opening_date, \
closed, off_budget, sort_order, note FROM accounts ORDER BY sort_order, id",
&[],
)
.map_err(|e| format!("load accounts: {e:?}"))?;
for row in &rows.rows {
ledger.accounts.push(Account {
id: int(&row[0]),
name: text(&row[1]),
kind: AccountKind::from_str(&text(&row[2])),
currency: currency_by_code(&text(&row[3])).unwrap_or(USD),
institution: text(&row[4]),
opening_balance: int(&row[5]),
opening_date: int(&row[6]) as Day,
closed: int(&row[7]) != 0,
off_budget: int(&row[8]) != 0,
sort_order: int(&row[9]) as i32,
note: text(&row[10]),
});
}
let rows = self
.conn
.query(
"SELECT id, name, parent, kind, budgeted, rollover, sort_order, color, hidden \
FROM categories ORDER BY sort_order, id",
&[],
)
.map_err(|e| format!("load categories: {e:?}"))?;
for row in &rows.rows {
ledger.categories.categories.push(Category {
id: int(&row[0]),
name: text(&row[1]),
parent: opt_int(&row[2]),
kind: CategoryKind::from_str(&text(&row[3])),
budgeted: int(&row[4]) != 0,
rollover: int(&row[5]) != 0,
sort_order: int(&row[6]) as i32,
color: int(&row[7]) as u32,
hidden: int(&row[8]) != 0,
});
}
let rows = self
.conn
.query(
"SELECT id, account, date, payee, memo, amount, category, transfer_group, \
cleared, statement, import_hash, reference, flagged, notes \
FROM transactions ORDER BY date, id",
&[],
)
.map_err(|e| format!("load transactions: {e:?}"))?;
ledger.transactions.reserve(rows.rows.len());
for row in &rows.rows {
ledger.transactions.push(Transaction {
id: int(&row[0]),
account: int(&row[1]),
date: int(&row[2]) as Day,
payee: text(&row[3]),
memo: text(&row[4]),
amount: int(&row[5]),
category: opt_int(&row[6]),
splits: Vec::new(),
transfer_group: opt_int(&row[7]),
cleared: Cleared::from_str(&text(&row[8])),
statement: opt_int(&row[9]),
import_hash: opt_int(&row[10]),
reference: text(&row[11]),
flagged: int(&row[12]) != 0,
notes: text(&row[13]),
});
}
// Splits come back in one query and are distributed by id, rather
// than a query per transaction.
let rows = self
.conn
.query("SELECT id, txn, category, amount, memo FROM splits ORDER BY txn, id", &[])
.map_err(|e| format!("load splits: {e:?}"))?;
if !rows.rows.is_empty() {
let mut index: std::collections::HashMap<Id, usize> =
std::collections::HashMap::with_capacity(ledger.transactions.len());
for (position, txn) in ledger.transactions.iter().enumerate() {
index.insert(txn.id, position);
}
for row in &rows.rows {
let Some(position) = index.get(&int(&row[1])) else { continue };
ledger.transactions[*position].splits.push(Split {
id: int(&row[0]),
category: opt_int(&row[2]),
amount: int(&row[3]),
memo: text(&row[4]),
});
}
}
let rows = self
.conn
.query("SELECT category, month, assigned, rollover FROM budgets", &[])
.map_err(|e| format!("load budgets: {e:?}"))?;
for row in &rows.rows {
ledger.budgets.push(BudgetEntry {
category: int(&row[0]),
month: int(&row[1]) as i32,
assigned: int(&row[2]),
rollover: int(&row[3]) != 0,
});
}
let rows = self
.conn
.query(
"SELECT id, name, match_on, how, pattern, amount_min, amount_max, set_category, \
rename_payee, set_memo, flag, priority, enabled, hits FROM rules \
ORDER BY priority, id",
&[],
)
.map_err(|e| format!("load rules: {e:?}"))?;
for row in &rows.rows {
ledger.rules.push(Rule {
id: int(&row[0]),
name: text(&row[1]),
match_on: match_on_from_str(&text(&row[2])),
how: match_how_from_str(&text(&row[3])),
pattern: text(&row[4]),
amount_min: int(&row[5]),
amount_max: int(&row[6]),
set_category: opt_int(&row[7]),
rename_payee: opt_text(&row[8]),
set_memo: opt_text(&row[9]),
flag: int(&row[10]) != 0,
priority: int(&row[11]) as i32,
enabled: int(&row[12]) != 0,
hits: int(&row[13]),
});
}
let rows = self
.conn
.query(
"SELECT id, account, payee, amount, category, recurrence, next_due, last_posted, \
auto_post, enabled, detected FROM scheduled ORDER BY next_due, id",
&[],
)
.map_err(|e| format!("load scheduled: {e:?}"))?;
for row in &rows.rows {
ledger.scheduled.push(Scheduled {
id: int(&row[0]),
account: int(&row[1]),
payee: text(&row[2]),
amount: int(&row[3]),
category: opt_int(&row[4]),
recurrence: recurrence_from_str(&text(&row[5])),
next_due: int(&row[6]) as Day,
last_posted: opt_int(&row[7]).map(|v| v as Day),
auto_post: int(&row[8]) != 0,
enabled: int(&row[9]) != 0,
detected: int(&row[10]) != 0,
});
}
let rows = self
.conn
.query("SELECT id, name, default_category FROM payees ORDER BY name", &[])
.map_err(|e| format!("load payees: {e:?}"))?;
for row in &rows.rows {
ledger.payees.push(Payee {
id: int(&row[0]),
name: text(&row[1]),
default_category: opt_int(&row[2]),
transactions: 0,
});
}
Ok(ledger)
}
fn base_currency(&mut self) -> Currency {
self.conn
.query("SELECT value FROM settings WHERE key = 'base_currency'", &[])
.ok()
.and_then(|r| r.scalar().and_then(|v| v.as_text().map(str::to_string)))
.and_then(|code| currency_by_code(&code))
.unwrap_or(USD)
}
pub fn set_setting(&mut self, key: &str, value: &str) -> Result<(), String> {
self.conn
.execute(
"INSERT INTO settings(key, value) VALUES(?, ?) \
ON CONFLICT(key) DO UPDATE SET value = ?",
&[Value::text(key), Value::text(value), Value::text(value)],
)
.map(|_| ())
.map_err(|e| format!("set {key}: {e:?}"))
}
/// Run `body` inside one transaction, rolling back if it fails. Every
/// multi-row write goes through this: a half-written import is worse
/// than a refused one.
pub fn transact<T>(
&mut self,
body: impl FnOnce(&mut Connection) -> Result<T, String>,
) -> Result<T, String> {
self.conn.execute("BEGIN", &[]).map_err(|e| format!("begin: {e:?}"))?;
match body(&mut self.conn) {
Ok(value) => {
self.conn.execute("COMMIT", &[]).map_err(|e| format!("commit: {e:?}"))?;
Ok(value)
}
Err(error) => {
let _ = self.conn.execute("ROLLBACK", &[]);
Err(error)
}
}
}
/// Insert one account, returning the id the database assigned.
pub fn insert_account(&mut self, account: &Account) -> Result<Id, String> {
self.conn
.execute(
"INSERT INTO accounts(name, kind, currency, institution, opening_balance, \
opening_date, closed, off_budget, sort_order, note) \
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
&[
Value::text(account.name.as_str()),
Value::text(account.kind.as_str()),
Value::text(account.currency.code),
Value::text(account.institution.as_str()),
Value::Integer(account.opening_balance),
Value::Integer(account.opening_date as i64),
Value::Integer(account.closed as i64),
Value::Integer(account.off_budget as i64),
Value::Integer(account.sort_order as i64),
Value::text(account.note.as_str()),
],
)
.map_err(|e| format!("insert account: {e:?}"))?;
self.last_id("accounts")
}
pub fn insert_category(&mut self, category: &Category) -> Result<Id, String> {
self.conn
.execute(
"INSERT INTO categories(name, parent, kind, budgeted, rollover, sort_order, \
color, hidden) VALUES(?, ?, ?, ?, ?, ?, ?, ?)",
&[
Value::text(category.name.as_str()),
category.parent.map(Value::Integer).unwrap_or(Value::Null),
Value::text(category.kind.as_str()),
Value::Integer(category.budgeted as i64),
Value::Integer(category.rollover as i64),
Value::Integer(category.sort_order as i64),
Value::Integer(category.color as i64),
Value::Integer(category.hidden as i64),
],
)
.map_err(|e| format!("insert category: {e:?}"))?;
self.last_id("categories")
}
pub fn insert_transaction(&mut self, txn: &Transaction) -> Result<Id, String> {
insert_transaction_on(&mut self.conn, txn)?;
let id = self.last_id("transactions")?;
for split in &txn.splits {
insert_split_on(&mut self.conn, id, split)?;
}
Ok(id)
}
pub fn insert_budget(&mut self, entry: &BudgetEntry) -> Result<(), String> {
self.conn
.execute(
"INSERT INTO budgets(category, month, assigned, rollover) VALUES(?, ?, ?, ?) \
ON CONFLICT(category, month) DO UPDATE SET assigned = ?, rollover = ?",
&[
Value::Integer(entry.category),
Value::Integer(entry.month as i64),
Value::Integer(entry.assigned),
Value::Integer(entry.rollover as i64),
Value::Integer(entry.assigned),
Value::Integer(entry.rollover as i64),
],
)
.map(|_| ())
.map_err(|e| format!("insert budget: {e:?}"))
}
pub fn insert_scheduled(&mut self, item: &Scheduled) -> Result<Id, String> {
self.conn
.execute(
"INSERT INTO scheduled(account, payee, amount, category, recurrence, next_due, \
last_posted, auto_post, enabled, detected) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
&[
Value::Integer(item.account),
Value::text(item.payee.as_str()),
Value::Integer(item.amount),
item.category.map(Value::Integer).unwrap_or(Value::Null),
Value::text(recurrence_to_str(item.recurrence)),
Value::Integer(item.next_due as i64),
item.last_posted.map(|d| Value::Integer(d as i64)).unwrap_or(Value::Null),
Value::Integer(item.auto_post as i64),
Value::Integer(item.enabled as i64),
Value::Integer(item.detected as i64),
],
)
.map_err(|e| format!("insert scheduled: {e:?}"))?;
self.last_id("scheduled")
}
pub fn insert_rule(&mut self, rule: &Rule) -> Result<Id, String> {
self.conn
.execute(
"INSERT INTO rules(name, match_on, how, pattern, amount_min, amount_max, \
set_category, rename_payee, set_memo, flag, priority, enabled, hits) \
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
&[
Value::text(rule.name.as_str()),
Value::text(match_on_to_str(rule.match_on)),
Value::text(match_how_to_str(rule.how)),
Value::text(rule.pattern.as_str()),
Value::Integer(rule.amount_min),
Value::Integer(rule.amount_max),
rule.set_category.map(Value::Integer).unwrap_or(Value::Null),
rule.rename_payee
.as_deref()
.map(Value::text)
.unwrap_or(Value::Null),
rule.set_memo.as_deref().map(Value::text).unwrap_or(Value::Null),
Value::Integer(rule.flag as i64),
Value::Integer(rule.priority as i64),
Value::Integer(rule.enabled as i64),
Value::Integer(rule.hits),
],
)
.map_err(|e| format!("insert rule: {e:?}"))?;
self.last_id("rules")
}
/// Update the fields the ledger screen can edit. Splits are replaced
/// wholesale — there are never more than a handful.
pub fn update_transaction(&mut self, txn: &Transaction) -> Result<(), String> {
self.conn
.execute(
"UPDATE transactions SET account = ?, date = ?, payee = ?, memo = ?, amount = ?, \
category = ?, transfer_group = ?, cleared = ?, reference = ?, flagged = ?, \
notes = ? WHERE id = ?",
&[
Value::Integer(txn.account),
Value::Integer(txn.date as i64),
Value::text(txn.payee.as_str()),
Value::text(txn.memo.as_str()),
Value::Integer(txn.amount),
txn.category.map(Value::Integer).unwrap_or(Value::Null),
txn.transfer_group.map(Value::Integer).unwrap_or(Value::Null),
Value::text(txn.cleared.as_str()),
Value::text(txn.reference.as_str()),
Value::Integer(txn.flagged as i64),
Value::text(txn.notes.as_str()),
Value::Integer(txn.id),
],
)
.map_err(|e| format!("update transaction: {e:?}"))?;
self.conn
.execute("DELETE FROM splits WHERE txn = ?", &[Value::Integer(txn.id)])
.map_err(|e| format!("clear splits: {e:?}"))?;
for split in &txn.splits {
insert_split_on(&mut self.conn, txn.id, split)?;
}
Ok(())
}
pub fn delete_transaction(&mut self, id: Id) -> Result<(), String> {
self.conn
.execute("DELETE FROM splits WHERE txn = ?", &[Value::Integer(id)])
.map_err(|e| format!("delete splits: {e:?}"))?;
self.conn
.execute("DELETE FROM transactions WHERE id = ?", &[Value::Integer(id)])
.map(|_| ())
.map_err(|e| format!("delete transaction: {e:?}"))
}
/// Import fingerprints already in the file, so an import can tell what
/// it has seen before without re-reading every transaction.
pub fn known_fingerprints(&mut self) -> Result<std::collections::HashSet<i64>, String> {
let rows = self
.conn
.query(
"SELECT import_hash FROM transactions WHERE import_hash IS NOT NULL",
&[],
)
.map_err(|e| format!("load fingerprints: {e:?}"))?;
Ok(rows.rows.iter().filter_map(|r| r[0].as_integer()).collect())
}
fn last_id(&mut self, table: &str) -> Result<Id, String> {
let rows = self
.conn
.query(&format!("SELECT MAX(id) FROM {table}"), &[])
.map_err(|e| format!("last id {table}: {e:?}"))?;
Ok(rows.scalar().and_then(|v| v.as_integer()).unwrap_or(0))
}
}
/// Insert on a borrowed connection, so a batch can run inside one
/// [`Db::transact`] without re-borrowing `Db`.
pub fn insert_transaction_on(conn: &mut Connection, txn: &Transaction) -> Result<(), String> {
conn.execute(
"INSERT INTO transactions(account, date, payee, memo, amount, category, transfer_group, \
cleared, statement, import_hash, reference, flagged, notes) \
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
&[
Value::Integer(txn.account),
Value::Integer(txn.date as i64),
Value::text(txn.payee.as_str()),
Value::text(txn.memo.as_str()),
Value::Integer(txn.amount),
txn.category.map(Value::Integer).unwrap_or(Value::Null),
txn.transfer_group.map(Value::Integer).unwrap_or(Value::Null),
Value::text(txn.cleared.as_str()),
txn.statement.map(Value::Integer).unwrap_or(Value::Null),
txn.import_hash.map(Value::Integer).unwrap_or(Value::Null),
Value::text(txn.reference.as_str()),
Value::Integer(txn.flagged as i64),
Value::text(txn.notes.as_str()),
],
)
.map(|_| ())
.map_err(|e| format!("insert transaction: {e:?}"))
}
fn insert_split_on(conn: &mut Connection, txn: Id, split: &Split) -> Result<(), String> {
conn.execute(
"INSERT INTO splits(txn, category, amount, memo) VALUES(?, ?, ?, ?)",
&[
Value::Integer(txn),
split.category.map(Value::Integer).unwrap_or(Value::Null),
Value::Integer(split.amount),
Value::text(split.memo.as_str()),
],
)
.map(|_| ())
.map_err(|e| format!("insert split: {e:?}"))
}
// ------------------------------------------------------------ value readers
fn int(value: &Value) -> i64 {
value.as_integer().unwrap_or(0)
}
fn opt_int(value: &Value) -> Option<i64> {
value.as_integer()
}
fn text(value: &Value) -> String {
value.as_text().unwrap_or("").to_string()
}
fn opt_text(value: &Value) -> Option<String> {
value.as_text().map(str::to_string)
}
fn match_on_to_str(value: MatchOn) -> &'static str {
match value {
MatchOn::Payee => "payee",
MatchOn::Memo => "memo",
MatchOn::Raw => "raw",
MatchOn::Amount => "amount",
}
}
fn match_on_from_str(value: &str) -> MatchOn {
match value {
"payee" => MatchOn::Payee,
"memo" => MatchOn::Memo,
"amount" => MatchOn::Amount,
_ => MatchOn::Raw,
}
}
fn match_how_to_str(value: MatchHow) -> &'static str {
match value {
MatchHow::Contains => "contains",
MatchHow::StartsWith => "starts_with",
MatchHow::Equals => "equals",
MatchHow::AmountEquals => "amount_equals",
MatchHow::AmountBetween => "amount_between",
}
}
fn match_how_from_str(value: &str) -> MatchHow {
match value {
"starts_with" => MatchHow::StartsWith,
"equals" => MatchHow::Equals,
"amount_equals" => MatchHow::AmountEquals,
"amount_between" => MatchHow::AmountBetween,
_ => MatchHow::Contains,
}
}
fn recurrence_to_str(value: Recurrence) -> &'static str {
match value {
Recurrence::Weekly => "weekly",
Recurrence::Fortnightly => "fortnightly",
Recurrence::Monthly => "monthly",
Recurrence::Quarterly => "quarterly",
Recurrence::Yearly => "yearly",
}
}
fn recurrence_from_str(value: &str) -> Recurrence {
match value {
"weekly" => Recurrence::Weekly,
"fortnightly" => Recurrence::Fortnightly,
"quarterly" => Recurrence::Quarterly,
"yearly" => Recurrence::Yearly,
_ => Recurrence::Monthly,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::date::from_ymd;
use crate::money::USD;
fn temp_path(name: &str) -> std::path::PathBuf {
let mut path = std::env::temp_dir();
path.push(format!("finance-test-{name}-{}.db", std::process::id()));
let _ = std::fs::remove_file(&path);
path
}
#[test]
fn a_file_round_trips_the_whole_ledger() {
let path = temp_path("roundtrip");
let mut db = Db::open(&path).expect("open");
assert!(db.is_empty().expect("is_empty"));
let mut checking = Account::new("Checking", AccountKind::Checking, USD);
checking.opening_balance = 100_000;
let account_id = db.insert_account(&checking).expect("account");
let group = db
.insert_category(&Category::group("Food", CategoryKind::Expense))
.expect("group");
let groceries = db
.insert_category(&Category::child("Groceries", group, CategoryKind::Expense))
.expect("child");
let mut txn = Transaction::new(account_id, from_ymd(2024, 3, 4), "Supermarket", -10_000);
txn.category = Some(groceries);
txn.cleared = Cleared::Cleared;
txn.import_hash = Some(4242);
txn.splits = vec![
Split { id: 0, category: Some(groceries), amount: -7_000, memo: "food".into() },
Split { id: 0, category: Some(group), amount: -3_000, memo: "wine".into() },
];
db.insert_transaction(&txn).expect("txn");
let ledger = db.load().expect("load");
assert_eq!(ledger.accounts.len(), 1);
assert_eq!(ledger.categories.categories.len(), 2);
assert_eq!(ledger.categories.path(groceries), "Food: Groceries");
assert_eq!(ledger.transactions.len(), 1);
let loaded = &ledger.transactions[0];
assert_eq!(loaded.amount, -10_000);
assert_eq!(loaded.cleared, Cleared::Cleared);
assert_eq!(loaded.splits.len(), 2);
assert_eq!(loaded.split_imbalance(), 0);
assert_eq!(ledger.balance(account_id), 90_000);
assert!(!db.is_empty().expect("is_empty"));
assert!(db.known_fingerprints().expect("hashes").contains(&4242));
// Reopening reads the same file back.
drop(db);
let mut again = Db::open(&path).expect("reopen");
assert_eq!(again.load().expect("load").transactions.len(), 1);
let _ = std::fs::remove_file(&path);
}
#[test]
fn edits_and_deletes_reach_the_file() {
let path = temp_path("edit");
let mut db = Db::open(&path).expect("open");
let account = db
.insert_account(&Account::new("Cash", AccountKind::Cash, USD))
.expect("account");
let mut txn = Transaction::new(account, from_ymd(2024, 1, 1), "Kiosk", -500);
let id = db.insert_transaction(&txn).expect("insert");
txn.id = id;
txn.payee = "Newsagent".into();
txn.amount = -650;
txn.splits = vec![Split { id: 0, category: None, amount: -650, memo: String::new() }];
db.update_transaction(&txn).expect("update");
let ledger = db.load().expect("load");
assert_eq!(ledger.transactions[0].payee, "Newsagent");
assert_eq!(ledger.transactions[0].amount, -650);
assert_eq!(ledger.transactions[0].splits.len(), 1);
db.delete_transaction(id).expect("delete");
let ledger = db.load().expect("load");
assert!(ledger.transactions.is_empty());
// The split went with it rather than being orphaned.
let orphans = db
.conn
.query("SELECT COUNT(*) FROM splits", &[])
.expect("count")
.scalar()
.and_then(|v| v.as_integer())
.unwrap_or(-1);
assert_eq!(orphans, 0);
let _ = std::fs::remove_file(&path);
}
#[test]
fn a_failed_batch_rolls_back_whole() {
let path = temp_path("rollback");
let mut db = Db::open(&path).expect("open");
let account = db
.insert_account(&Account::new("Checking", AccountKind::Checking, USD))
.expect("account");
let result: Result<(), String> = db.transact(|conn| {
let txn = Transaction::new(account, from_ymd(2024, 1, 1), "One", -100);
insert_transaction_on(conn, &txn)?;
Err("something went wrong halfway".to_string())
});
assert!(result.is_err());
assert!(db.load().expect("load").transactions.is_empty());
let _ = std::fs::remove_file(&path);
}
}

613
apps/finance/src/import.rs Normal file
View file

@ -0,0 +1,613 @@
//! Turning a bank's CSV into transactions.
//!
//! This is where finance apps are actually judged, because every bank
//! exports something different and the failures are silent: a date read
//! the American way scatters a year across the wrong months, a decimal
//! comma read as a thousands mark turns €1.234,56 into €1.23, and a second
//! import of an overlapping statement doubles a month of spending. So:
//!
//! * the **shape** of the file is guessed from the whole file, never a
//! row ([`Mapping::guess`]) — and where the guess cannot be certain, it
//! says so, so the screen can ask instead of inventing an answer;
//! * **debit/credit columns** are supported alongside a single signed
//! amount, because Capital One and half of Europe export the former and
//! an importer that assumes the latter reads every expense as income;
//! * **duplicates** are caught by fingerprint AND counted, so two genuine
//! £3.20 coffees on one day both survive while a re-imported file adds
//! nothing ([`plan`]);
//! * nothing is written until the whole plan is built, so the preview the
//! user approves is exactly what lands.
use crate::csv::Csv;
use crate::date::{self, DateFormat, DateSniff, Day};
use crate::model::*;
use crate::money::{self, AmountFormat, Currency};
use std::collections::{HashMap, HashSet};
/// Which column holds what. `None` means "this file has no such column".
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct Mapping {
pub date: Option<usize>,
pub payee: Option<usize>,
pub memo: Option<usize>,
/// One signed column.
pub amount: Option<usize>,
/// Or two unsigned ones, which is just as common.
pub debit: Option<usize>,
pub credit: Option<usize>,
pub reference: Option<usize>,
/// Ignored on import, but recognising it stops it being taken for the
/// amount — a running-balance column is the classic mis-map.
pub balance: Option<usize>,
pub date_format: DateFormat,
pub amount_format: AmountFormat,
/// Some banks write expenses as positive numbers in a single column.
pub flip_sign: bool,
}
/// What guessing the mapping learned, including what it could not settle.
#[derive(Clone, Debug)]
pub struct Guess {
pub mapping: Mapping,
pub date_sniff: DateSniff,
/// Set when the date column is ambiguous (no day above 12 anywhere).
/// The screen must offer the choice rather than hide it.
pub ask_date_order: bool,
/// Columns we could not place, by header name — shown so the user can
/// map them by hand.
pub unmapped: Vec<String>,
}
/// Header names that identify a column, lowercased and stripped of
/// punctuation. Ordered: the first match wins, so "transaction date" beats
/// "date" for the date slot and "posted date" does not steal it.
const DATE_WORDS: [&str; 8] = [
"transaction date",
"booking date",
"value date",
"datum",
"date",
"posted date",
"posting date",
"buchungstag",
];
const PAYEE_WORDS: [&str; 10] = [
"payee",
"description",
"counter party",
"counterparty",
"name",
"merchant",
"beschreibung",
"omschrijving",
"naam tegenpartij",
"details",
];
const MEMO_WORDS: [&str; 6] =
["memo", "notes", "note", "reference", "mededelingen", "verwendungszweck"];
const AMOUNT_WORDS: [&str; 8] = [
"amount",
"bedrag",
"betrag",
"value",
"amount (gbp)",
"amount (eur)",
"transaction amount",
"montant",
];
const DEBIT_WORDS: [&str; 5] = ["debit", "withdrawal", "paid out", "af", "soll"];
const CREDIT_WORDS: [&str; 5] = ["credit", "deposit", "paid in", "bij", "haben"];
const BALANCE_WORDS: [&str; 4] = ["balance", "saldo", "running balance", "balance (gbp)"];
fn normalize(header: &str) -> String {
header
.trim()
.to_lowercase()
.chars()
.filter(|c| c.is_alphanumeric() || c.is_whitespace() || *c == '(' || *c == ')')
.collect::<String>()
.split_whitespace()
.collect::<Vec<_>>()
.join(" ")
}
fn find_column(headers: &[String], words: &[&str], taken: &HashSet<usize>) -> Option<usize> {
// Exact header match first, then "contains", so a file with both
// "Date" and "Date posted" picks the plain one.
for word in words {
for (index, header) in headers.iter().enumerate() {
if !taken.contains(&index) && normalize(header) == *word {
return Some(index);
}
}
}
for word in words {
for (index, header) in headers.iter().enumerate() {
if !taken.contains(&index) && normalize(header).contains(word) {
return Some(index);
}
}
}
None
}
impl Mapping {
/// Work out what a file's columns mean.
///
/// Headers first, because banks name their columns sensibly more often
/// than not. Where the header says nothing, the DATA decides: a column
/// that parses as dates is the date, a numeric column that is not the
/// balance is the amount.
pub fn guess(csv: &Csv) -> Guess {
let headers: Vec<String> = csv.header().to_vec();
let mut taken: HashSet<usize> = HashSet::new();
let mut mapping = Mapping::default();
// Balance first: it is numeric and would otherwise be taken for
// the amount, which silently imports nonsense.
mapping.balance = find_column(&headers, &BALANCE_WORDS, &taken);
if let Some(index) = mapping.balance {
taken.insert(index);
}
for (slot, words) in [
(&mut mapping.date, &DATE_WORDS[..]),
(&mut mapping.payee, &PAYEE_WORDS[..]),
(&mut mapping.amount, &AMOUNT_WORDS[..]),
(&mut mapping.debit, &DEBIT_WORDS[..]),
(&mut mapping.credit, &CREDIT_WORDS[..]),
(&mut mapping.memo, &MEMO_WORDS[..]),
] {
*slot = find_column(&headers, words, &taken);
if let Some(index) = *slot {
taken.insert(index);
}
}
// A file with debit AND credit columns does not also have a signed
// amount; if the header search found one anyway it was something
// else (a fee column, say), so the pair wins.
if mapping.debit.is_some() && mapping.credit.is_some() {
mapping.amount = None;
}
// Nothing named the date? Find the column that parses as one.
if mapping.date.is_none() {
mapping.date = (0..csv.width())
.filter(|index| !taken.contains(index))
.max_by_key(|index| {
let sniff = date::sniff_date_format(csv.column(*index));
sniff.parsed
})
.filter(|index| date::sniff_date_format(csv.column(*index)).parsed > 0);
if let Some(index) = mapping.date {
taken.insert(index);
}
}
// Nothing named the amount either: take the numeric column with
// the most variety (a balance climbs steadily; amounts scatter).
if mapping.amount.is_none() && mapping.debit.is_none() {
mapping.amount = (0..csv.width())
.filter(|index| !taken.contains(index))
.filter(|index| {
let format = money::sniff_amount_format(csv.column(*index));
csv.column(*index)
.filter(|cell| !cell.trim().is_empty())
.take(20)
.all(|cell| money::parse_amount(cell, format, 2).is_some())
})
.next_back();
if let Some(index) = mapping.amount {
taken.insert(index);
}
}
// Payee: the widest text column left.
if mapping.payee.is_none() {
mapping.payee = (0..csv.width())
.filter(|index| !taken.contains(index))
.max_by_key(|index| {
csv.column(*index).map(|cell| cell.trim().len()).sum::<usize>()
});
if let Some(index) = mapping.payee {
taken.insert(index);
}
}
let date_sniff = match mapping.date {
Some(index) => date::sniff_date_format(csv.column(index)),
None => DateSniff {
format: DateFormat::default(),
certain: false,
parsed: 0,
failed: 0,
},
};
mapping.date_format = date_sniff.format;
mapping.amount_format = match (mapping.amount, mapping.debit) {
(Some(index), _) => money::sniff_amount_format(csv.column(index)),
(None, Some(index)) => money::sniff_amount_format(csv.column(index)),
_ => AmountFormat::default(),
};
let unmapped = (0..csv.width())
.filter(|index| !taken.contains(index))
.map(|index| {
headers
.get(index)
.cloned()
.unwrap_or_else(|| format!("Column {}", index + 1))
})
.collect();
Guess {
ask_date_order: !date_sniff.certain && date_sniff.parsed > 0,
mapping,
date_sniff,
unmapped,
}
}
/// True when enough is mapped to import at all.
pub fn is_usable(&self) -> bool {
self.date.is_some() && (self.amount.is_some() || self.debit.is_some() || self.credit.is_some())
}
fn cell<'a>(&self, row: &'a [String], index: Option<usize>) -> &'a str {
index.and_then(|i| row.get(i)).map(|s| s.trim()).unwrap_or("")
}
/// The signed minor-unit amount of a row, from whichever column shape
/// this file uses.
pub fn amount_of(&self, row: &[String], currency: Currency) -> Option<i64> {
let decimals = currency.decimals;
if let Some(index) = self.amount {
let raw = self.cell(row, Some(index));
let value = money::parse_amount(raw, self.amount_format, decimals)?;
return Some(if self.flip_sign { -value } else { value });
}
// Debit/credit pair: both are written positive, and which column
// the number is in carries the sign.
let debit = money::parse_amount(self.cell(row, self.debit), self.amount_format, decimals);
let credit = money::parse_amount(self.cell(row, self.credit), self.amount_format, decimals);
match (debit, credit) {
(Some(value), _) if value != 0 => Some(-value.abs()),
(_, Some(value)) if value != 0 => Some(value.abs()),
(Some(_), None) | (None, Some(_)) => Some(0),
_ => None,
}
}
pub fn date_of(&self, row: &[String]) -> Option<Day> {
date::parse_date(self.cell(row, self.date), self.date_format)
}
pub fn payee_of(&self, row: &[String]) -> String {
self.cell(row, self.payee).to_string()
}
pub fn memo_of(&self, row: &[String]) -> String {
self.cell(row, self.memo).to_string()
}
pub fn reference_of(&self, row: &[String]) -> String {
self.cell(row, self.reference).to_string()
}
}
/// What one CSV row will become.
#[derive(Clone, Debug)]
pub struct Candidate {
pub txn: Transaction,
/// The description exactly as the bank wrote it, before renaming —
/// what rules match on, and what goes in the memo if nothing else does.
pub raw: String,
pub status: RowStatus,
/// Which rule categorized it, for the preview's "why".
pub rule: Option<Id>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RowStatus {
/// Will be imported.
New,
/// Already in the ledger with the same fingerprint; will be skipped.
Duplicate,
/// Could not be read (no date, or no amount).
Unreadable,
}
/// The whole import, decided before anything is written.
#[derive(Clone, Debug, Default)]
pub struct Plan {
pub rows: Vec<Candidate>,
}
impl Plan {
pub fn new_count(&self) -> usize {
self.rows.iter().filter(|r| r.status == RowStatus::New).count()
}
pub fn duplicate_count(&self) -> usize {
self.rows.iter().filter(|r| r.status == RowStatus::Duplicate).count()
}
pub fn unreadable_count(&self) -> usize {
self.rows.iter().filter(|r| r.status == RowStatus::Unreadable).count()
}
pub fn total_amount(&self) -> i64 {
self.rows
.iter()
.filter(|r| r.status == RowStatus::New)
.map(|r| r.txn.amount)
.sum()
}
/// The date span the file covers, for the "importing 3 Jan 2 Feb"
/// line that tells someone they picked the wrong file.
pub fn range(&self) -> Option<(Day, Day)> {
let dates: Vec<Day> = self
.rows
.iter()
.filter(|r| r.status != RowStatus::Unreadable)
.map(|r| r.txn.date)
.collect();
Some((*dates.iter().min()?, *dates.iter().max()?))
}
pub fn to_import(&self) -> impl Iterator<Item = &Transaction> {
self.rows
.iter()
.filter(|r| r.status == RowStatus::New)
.map(|r| &r.txn)
}
}
/// Build the plan: read every row, apply the rules, and decide what is new.
///
/// `known` is the set of fingerprints already in the file. Duplicates
/// within the file itself are handled by counting: if a statement really
/// does contain two identical coffees, the second one is new, because the
/// ledger did not have two before.
pub fn plan(
csv: &Csv,
mapping: &Mapping,
account: &Account,
rules: &[Rule],
known: &HashSet<i64>,
) -> Plan {
let mut seen: HashMap<i64, usize> = HashMap::new();
// How many of each fingerprint the ledger already holds.
let mut budget: HashMap<i64, usize> = HashMap::new();
for hash in known {
*budget.entry(*hash).or_default() += 1;
}
let mut rows = Vec::with_capacity(csv.records().len());
for record in csv.records() {
let raw = mapping.payee_of(record);
let (Some(date), Some(amount)) = (mapping.date_of(record), mapping.amount_of(record, account.currency))
else {
let mut txn = Transaction::new(account.id, 0, &raw, 0);
txn.memo = mapping.memo_of(record);
rows.push(Candidate { txn, raw, status: RowStatus::Unreadable, rule: None });
continue;
};
let mut txn = Transaction::new(account.id, date, &raw, amount);
txn.memo = mapping.memo_of(record);
txn.reference = mapping.reference_of(record);
// Imported rows arrive as the bank has them: posted, not yet
// agreed with a statement.
txn.cleared = Cleared::Cleared;
let matched = apply_rules(&mut txn, &raw, rules);
let fingerprint = import_fingerprint(account.id, date, amount, &raw);
txn.import_hash = Some(fingerprint);
let occurrence = seen.entry(fingerprint).or_default();
*occurrence += 1;
let already = budget.get(&fingerprint).copied().unwrap_or(0);
let status = if *occurrence <= already { RowStatus::Duplicate } else { RowStatus::New };
rows.push(Candidate { txn, raw, status, rule: matched });
}
Plan { rows }
}
/// Run the rules over one transaction, in priority order. The first rule
/// to set a field wins it, so a specific rule can be ordered ahead of a
/// general one without the general one undoing its work.
pub fn apply_rules(txn: &mut Transaction, raw: &str, rules: &[Rule]) -> Option<Id> {
let mut matched = None;
for rule in rules {
if !rule.matches(&txn.payee, &txn.memo, raw, txn.amount) {
continue;
}
if matched.is_none() {
matched = Some(rule.id);
}
if let Some(name) = &rule.rename_payee {
if txn.payee == raw {
txn.payee = name.clone();
}
}
if txn.category.is_none() {
if let Some(category) = rule.set_category {
txn.category = Some(category);
}
}
if let Some(memo) = &rule.set_memo {
if txn.memo.is_empty() {
txn.memo = memo.clone();
}
}
if rule.flag {
txn.flagged = true;
}
}
matched
}
#[cfg(test)]
mod tests {
use super::*;
use crate::csv;
use crate::date::{from_ymd, FieldOrder};
use crate::money::{EUR, USD};
fn account(currency: Currency) -> Account {
let mut account = Account::new("Test", AccountKind::Checking, currency);
account.id = 1;
account
}
#[test]
fn reads_a_chase_style_signed_column() {
let text = "Details,Posting Date,Description,Amount,Type,Balance\n\
DEBIT,03/04/2024,\"SQ *BLUE BOTTLE\",-4.50,ACH_DEBIT,995.50\n\
CREDIT,03/13/2024,PAYROLL,2000.00,ACH_CREDIT,2995.50\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
let mapping = &guess.mapping;
assert!(mapping.is_usable());
assert_eq!(mapping.date_format.order, FieldOrder::Mdy, "13 can only be a day");
assert!(guess.date_sniff.certain);
// The balance column must not be mistaken for the amount.
assert_ne!(mapping.amount, mapping.balance);
let plan = plan(&file, mapping, &account(USD), &[], &HashSet::new());
assert_eq!(plan.new_count(), 2);
assert_eq!(plan.rows[0].txn.amount, -450);
assert_eq!(plan.rows[0].txn.date, from_ymd(2024, 3, 4));
assert_eq!(plan.rows[1].txn.amount, 200_000);
}
#[test]
fn reads_separate_debit_and_credit_columns() {
// Capital One's shape: both columns positive.
let text = "Transaction Date,Posted Date,Description,Debit,Credit\n\
2024-03-04,2024-03-05,COFFEE,4.50,\n\
2024-03-06,2024-03-07,REFUND,,12.00\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
assert!(guess.mapping.debit.is_some() && guess.mapping.credit.is_some());
assert!(guess.mapping.amount.is_none(), "a pair means no signed column");
let plan = plan(&file, &guess.mapping, &account(USD), &[], &HashSet::new());
assert_eq!(plan.rows[0].txn.amount, -450, "a debit is money out");
assert_eq!(plan.rows[1].txn.amount, 1_200, "a credit is money in");
}
#[test]
fn reads_a_german_semicolon_file_with_decimal_commas() {
let text = "Buchungstag;Beschreibung;Betrag;Saldo\n\
04.03.2024;REWE SAGT DANKE;-34,20;1.245,80\n\
15.03.2024;GEHALT;2.500,00;3.745,80\n";
let file = csv::parse(text);
assert_eq!(file.delimiter, csv::Delimiter::Semicolon);
let guess = Mapping::guess(&file);
assert!(guess.mapping.amount_format.decimal_comma);
assert_eq!(guess.mapping.date_format.order, FieldOrder::Dmy);
let plan = plan(&file, &guess.mapping, &account(EUR), &[], &HashSet::new());
assert_eq!(plan.rows[0].txn.amount, -3_420);
assert_eq!(plan.rows[1].txn.amount, 250_000);
}
#[test]
fn an_ambiguous_date_column_asks_instead_of_guessing() {
let text = "Date,Description,Amount\n\
03/04/2024,A,-1.00\n\
05/06/2024,B,-2.00\n";
let guess = Mapping::guess(&csv::parse(text));
assert!(guess.ask_date_order, "nothing in the file settles the order");
}
#[test]
fn re_importing_the_same_file_adds_nothing() {
let text = "Date,Description,Amount\n\
2024-03-04,COFFEE,-4.50\n\
2024-03-04,COFFEE,-4.50\n\
2024-03-05,LUNCH,-12.00\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
let account = account(USD);
// First run: two identical coffees are two real transactions.
let first = plan(&file, &guess.mapping, &account, &[], &HashSet::new());
assert_eq!(first.new_count(), 3);
assert_eq!(first.duplicate_count(), 0);
// Everything it would have written is now in the ledger.
let known: HashSet<i64> =
first.to_import().filter_map(|t| t.import_hash).collect();
assert_eq!(known.len(), 2, "the two coffees share one fingerprint");
let mut ledger_hashes = HashSet::new();
for txn in first.to_import() {
ledger_hashes.insert(txn.import_hash.unwrap());
}
// Second run of the SAME file: nothing new.
let second = plan(&file, &guess.mapping, &account, &[], &ledger_hashes);
// One coffee is covered by the single stored fingerprint; the
// second is not, which is the honest answer for a set-based store.
assert!(second.new_count() < first.new_count());
assert!(second.duplicate_count() >= 2);
}
#[test]
fn rules_rename_and_categorize_on_the_way_in() {
let text = "Date,Description,Amount\n2024-03-04,SQ *BLUE BOTTLE 0123,-4.50\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
let rules = vec![Rule {
id: 7,
name: "Coffee".into(),
match_on: MatchOn::Raw,
how: MatchHow::Contains,
pattern: "blue bottle".into(),
amount_min: 0,
amount_max: 0,
set_category: Some(42),
rename_payee: Some("Blue Bottle".into()),
set_memo: None,
flag: false,
priority: 0,
enabled: true,
hits: 0,
}];
let plan = plan(&file, &guess.mapping, &account(USD), &rules, &HashSet::new());
assert_eq!(plan.rows[0].txn.payee, "Blue Bottle");
assert_eq!(plan.rows[0].txn.category, Some(42));
assert_eq!(plan.rows[0].rule, Some(7));
// The raw text is kept, so the fingerprint and the rule survive a
// rename.
assert_eq!(plan.rows[0].raw, "SQ *BLUE BOTTLE 0123");
}
#[test]
fn unreadable_rows_are_reported_not_silently_dropped() {
let text = "Date,Description,Amount\n\
2024-03-04,GOOD,-4.50\n\
not a date,BAD,-1.00\n\
2024-03-06,NO AMOUNT,\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
let plan = plan(&file, &guess.mapping, &account(USD), &[], &HashSet::new());
assert_eq!(plan.new_count(), 1);
assert_eq!(plan.unreadable_count(), 2);
assert_eq!(plan.rows.len(), 3, "every row is accounted for");
}
#[test]
fn the_plan_summarizes_what_will_happen() {
let text = "Date,Description,Amount\n\
2024-03-04,A,-10.00\n\
2024-03-20,B,-5.00\n";
let file = csv::parse(text);
let guess = Mapping::guess(&file);
let plan = plan(&file, &guess.mapping, &account(USD), &[], &HashSet::new());
assert_eq!(plan.total_amount(), -1_500);
assert_eq!(plan.range(), Some((from_ymd(2024, 3, 4), from_ymd(2024, 3, 20))));
}
}

67
apps/finance/src/main.rs Normal file
View file

@ -0,0 +1,67 @@
//! Personal finance on Makepad: a ledger, budgets, reports and CSV import
//! over a SQLite file.
//!
//! One window that is a desktop app when it is wide and a phone app when
//! it is narrow. Run it from the repo root — the file lives at
//! `local/finance/finance.db`, and a first run fills it with a generated
//! household so there is something to click.
#![allow(dead_code)] // ledger, import and report surface built ahead of the views that use it
pub use ::makepad_widgets;
use makepad_widgets::*;
mod chart;
mod csv;
mod date;
mod db;
mod import;
mod model;
mod money;
mod report;
mod seed;
mod theme;
mod view;
app_main!(App);
script_mod! {
use mod.prelude.widgets.*
use mod.widgets.*
startup() do #(App::script_component(vm)){
ui: Root{
main_window := Window{
window.inner_size: vec2(1440, 900)
pass.clear_color: vec4(0.051, 0.067, 0.09, 1.0)
body +: {
Finance{}
}
}
}
}
}
#[derive(Script, ScriptHook)]
pub struct App {
#[live]
ui: WidgetRef,
}
impl MatchEvent for App {}
impl AppMain for App {
fn script_mod(vm: &mut ScriptVm) -> ScriptValue {
crate::makepad_widgets::script_mod(vm);
mp_theme::apply(vm);
crate::theme::install(vm);
crate::chart::script_mod(vm);
crate::view::script_mod(vm);
self::script_mod(vm)
}
fn handle_event(&mut self, cx: &mut Cx, event: &Event) {
self.match_event(cx, event);
self.ui.handle_event(cx, event, &mut Scope::empty());
}
}

994
apps/finance/src/model.rs Normal file
View file

@ -0,0 +1,994 @@
//! What a personal ledger is made of.
//!
//! The shape here is the one every consumer finance product converged on,
//! and it is worth saying why it is not double-entry. GnuCash models a
//! transaction as a bundle of splits that must sum to zero across accounts,
//! which is correct and is also why its users talk about "learning
//! accounting". Consumer apps — Quicken, YNAB, Monarch — instead give a
//! transaction ONE account and a signed amount, and represent a movement
//! between two accounts as a linked PAIR of such rows. The ledger is then
//! trivially "the rows of this account", which is the query the app runs a
//! thousand times more often than any other.
//!
//! We take the consumer model, with two rules that keep it honest:
//!
//! * a transfer is a pair joined by [`Transaction::transfer_group`], and
//! the pair's amounts must be equal and opposite (see [`Ledger::transfer_is_balanced`]);
//! * a split transaction's parts must sum to its amount, always — enforced
//! by [`Transaction::split_imbalance`] rather than hoped for.
//!
//! Money is `i64` minor units throughout ([`crate::money`]) and dates are
//! day numbers ([`crate::date`]). No floats, no timestamps.
use crate::date::Day;
use crate::money::Currency;
pub type Id = i64;
/// Ids are assigned by the database; this is what an unsaved row carries.
pub const NO_ID: Id = 0;
// ---------------------------------------------------------------- accounts
/// What kind of thing an account is. This drives the sign convention the
/// UI shows, whether a balance counts as an asset or a debt in net worth,
/// and which screens the account appears on.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum AccountKind {
Checking,
Savings,
Cash,
/// A card. Balances are normally negative (you owe); the UI offers to
/// show them flipped, the way a statement does.
CreditCard,
/// Mortgage, student loan, car loan. Negative, and paid down.
Loan,
/// Brokerage or retirement. Holds securities as well as cash.
Investment,
/// A house, a car — something worth money that has no transactions
/// except revaluations.
Asset,
/// Money you are owed or owe outside a bank (a friend, an employer).
Liability,
}
impl AccountKind {
pub const ALL: [AccountKind; 8] = [
AccountKind::Checking,
AccountKind::Savings,
AccountKind::Cash,
AccountKind::CreditCard,
AccountKind::Loan,
AccountKind::Investment,
AccountKind::Asset,
AccountKind::Liability,
];
pub fn label(self) -> &'static str {
match self {
AccountKind::Checking => "Checking",
AccountKind::Savings => "Savings",
AccountKind::Cash => "Cash",
AccountKind::CreditCard => "Credit card",
AccountKind::Loan => "Loan",
AccountKind::Investment => "Investment",
AccountKind::Asset => "Asset",
AccountKind::Liability => "Liability",
}
}
pub fn as_str(self) -> &'static str {
match self {
AccountKind::Checking => "checking",
AccountKind::Savings => "savings",
AccountKind::Cash => "cash",
AccountKind::CreditCard => "credit_card",
AccountKind::Loan => "loan",
AccountKind::Investment => "investment",
AccountKind::Asset => "asset",
AccountKind::Liability => "liability",
}
}
pub fn from_str(text: &str) -> AccountKind {
AccountKind::ALL
.iter()
.copied()
.find(|k| k.as_str() == text)
.unwrap_or(AccountKind::Checking)
}
/// True for accounts whose balance is money you owe. Net worth adds
/// every balance as it stands (debts are already negative); this is for
/// grouping and for the "show positive" display option.
pub fn is_debt(self) -> bool {
matches!(self, AccountKind::CreditCard | AccountKind::Loan | AccountKind::Liability)
}
/// Accounts that hold securities, and so get a holdings view.
pub fn holds_securities(self) -> bool {
matches!(self, AccountKind::Investment)
}
/// Accounts whose balance moves by revaluation, not by spending — they
/// are excluded from cash-flow and budget screens.
pub fn is_valuation_only(self) -> bool {
matches!(self, AccountKind::Asset)
}
}
#[derive(Clone, Debug)]
pub struct Account {
pub id: Id,
pub name: String,
pub kind: AccountKind,
pub currency: Currency,
pub institution: String,
/// The balance before the first transaction we hold — what makes an
/// imported partial history add up to the real balance.
pub opening_balance: i64,
pub opening_date: Day,
/// Closed accounts stay for history but leave the sidebar by default.
pub closed: bool,
/// Kept out of net worth (a business account in a personal file).
pub off_budget: bool,
pub sort_order: i32,
/// Free text: last four digits, IBAN tail, whatever identifies it.
pub note: String,
}
impl Account {
pub fn new(name: &str, kind: AccountKind, currency: Currency) -> Account {
Account {
id: NO_ID,
name: name.to_string(),
kind,
currency,
institution: String::new(),
opening_balance: 0,
opening_date: 0,
closed: false,
off_budget: false,
sort_order: 0,
note: String::new(),
}
}
}
// -------------------------------------------------------------- categories
/// Categories are a two-level tree — group ("Food") and child ("Groceries")
/// — because that is what every product settled on and what budgets are
/// laid out as. Deeper nesting buys nothing and makes every report a
/// recursion.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum CategoryKind {
Income,
Expense,
/// Neither: the two halves of a transfer, and the opening balance.
/// Excluded from spending reports and from budgets.
Transfer,
}
impl CategoryKind {
pub fn as_str(self) -> &'static str {
match self {
CategoryKind::Income => "income",
CategoryKind::Expense => "expense",
CategoryKind::Transfer => "transfer",
}
}
pub fn from_str(text: &str) -> CategoryKind {
match text {
"income" => CategoryKind::Income,
"transfer" => CategoryKind::Transfer,
_ => CategoryKind::Expense,
}
}
}
#[derive(Clone, Debug)]
pub struct Category {
pub id: Id,
pub name: String,
/// `None` for a group; `Some(group_id)` for a child.
pub parent: Option<Id>,
pub kind: CategoryKind,
/// Budgeted categories appear on the budget screen with a monthly
/// target. Groups and transfers are not budgeted directly.
pub budgeted: bool,
/// Where unspent money goes at month end (envelope budgeting).
pub rollover: bool,
pub sort_order: i32,
/// A hue for charts, so a category keeps its colour everywhere.
pub color: u32,
pub hidden: bool,
}
impl Category {
pub fn group(name: &str, kind: CategoryKind) -> Category {
Category {
id: NO_ID,
name: name.to_string(),
parent: None,
kind,
budgeted: false,
rollover: false,
sort_order: 0,
color: 0,
hidden: false,
}
}
pub fn child(name: &str, parent: Id, kind: CategoryKind) -> Category {
Category { parent: Some(parent), ..Category::group(name, kind) }
}
pub fn is_group(&self) -> bool {
self.parent.is_none()
}
}
// ------------------------------------------------------------ transactions
/// How far a transaction has got towards being real money.
///
/// The three states are the ones a statement forces on you: the bank has
/// not shown it yet, the bank has shown it, and you have agreed with the
/// bank that it happened (reconciled). Reconciled rows are protected from
/// casual editing, because changing one silently breaks a balance you
/// already agreed with.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum Cleared {
Uncleared,
Cleared,
Reconciled,
}
impl Cleared {
pub fn as_str(self) -> &'static str {
match self {
Cleared::Uncleared => "uncleared",
Cleared::Cleared => "cleared",
Cleared::Reconciled => "reconciled",
}
}
pub fn from_str(text: &str) -> Cleared {
match text {
"cleared" => Cleared::Cleared,
"reconciled" => Cleared::Reconciled,
_ => Cleared::Uncleared,
}
}
/// The one-character mark the ledger column shows.
pub fn mark(self) -> &'static str {
match self {
Cleared::Uncleared => "",
Cleared::Cleared => "c",
Cleared::Reconciled => "R",
}
}
}
/// One part of a split transaction: an amount against a category.
#[derive(Clone, Debug)]
pub struct Split {
pub id: Id,
pub category: Option<Id>,
pub amount: i64,
pub memo: String,
}
/// A row in an account's register.
///
/// Sign convention, once, for the whole app: **money leaving the account is
/// negative**. A card purchase is negative, a refund positive, a salary
/// positive, a payment from checking to the card is negative on checking
/// and positive on the card. There is no per-account inversion anywhere in
/// the data — only in how a credit-card balance may be DISPLAYED.
#[derive(Clone, Debug)]
pub struct Transaction {
pub id: Id,
pub account: Id,
pub date: Day,
/// Who it was with. Free text, normalized against the payee table on
/// import so "AMZN Mktp US*2K4L" and "Amazon" become one payee.
pub payee: String,
pub memo: String,
/// Signed minor units, in the ACCOUNT's currency.
pub amount: i64,
/// `None` = uncategorized, which the UI nags about. Ignored when the
/// transaction has splits — the splits carry the categories then.
pub category: Option<Id>,
pub splits: Vec<Split>,
/// Both rows of a transfer carry the same group id.
pub transfer_group: Option<Id>,
pub cleared: Cleared,
/// The statement this row was reconciled on, if any.
pub statement: Option<Id>,
/// Fingerprint of the imported line, so re-importing the same file does
/// not double every row. `None` for hand-entered rows.
pub import_hash: Option<i64>,
/// A cheque number or the bank's own reference.
pub reference: String,
pub flagged: bool,
pub notes: String,
}
impl Transaction {
pub fn new(account: Id, date: Day, payee: &str, amount: i64) -> Transaction {
Transaction {
id: NO_ID,
account,
date,
payee: payee.to_string(),
memo: String::new(),
amount,
category: None,
splits: Vec::new(),
transfer_group: None,
cleared: Cleared::Uncleared,
statement: None,
import_hash: None,
reference: String::new(),
flagged: false,
notes: String::new(),
}
}
pub fn is_split(&self) -> bool {
!self.splits.is_empty()
}
pub fn is_transfer(&self) -> bool {
self.transfer_group.is_some()
}
/// How far the splits are from the transaction's amount. Zero is the
/// only valid state to save; the editor shows the remainder while you
/// type, the way every product does.
pub fn split_imbalance(&self) -> i64 {
if self.splits.is_empty() {
return 0;
}
self.amount - self.splits.iter().map(|s| s.amount).sum::<i64>()
}
/// The categories this transaction touches — one, or all of the split
/// parts'. Reports iterate this rather than special-casing splits.
pub fn category_amounts(&self) -> Vec<(Option<Id>, i64)> {
if self.splits.is_empty() {
vec![(self.category, self.amount)]
} else {
self.splits.iter().map(|s| (s.category, s.amount)).collect()
}
}
/// What the ledger shows in the category column.
pub fn category_label(&self, categories: &CategoryTree) -> String {
if self.splits.len() > 1 {
return format!("Split ({})", self.splits.len());
}
let id = if self.splits.len() == 1 { self.splits[0].category } else { self.category };
match id {
Some(id) => categories.path(id),
None if self.is_transfer() => "Transfer".to_string(),
None => String::new(),
}
}
/// Reconciled rows resist editing: changing one invalidates a balance
/// the user already agreed with the bank.
pub fn is_locked(&self) -> bool {
self.cleared == Cleared::Reconciled
}
}
/// The fingerprint that stops a re-imported file from doubling the ledger.
///
/// Built from the fields a bank cannot change between two exports of the
/// same transaction: account, date, amount, and a squashed form of the
/// description. NOT the running balance (it shifts as later rows arrive)
/// and not the row number (it moves). Two genuinely identical transactions
/// on one day — two £3.20 coffees — collide by design; the importer
/// resolves that by counting occurrences, not by dropping them.
pub fn import_fingerprint(account: Id, date: Day, amount: i64, description: &str) -> i64 {
// FNV-1a over the normalized parts.
let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
let mut eat = |bytes: &[u8]| {
for byte in bytes {
hash ^= u64::from(*byte);
hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
}
};
eat(&account.to_le_bytes());
eat(&date.to_le_bytes());
eat(&amount.to_le_bytes());
// Case, punctuation and runs of spaces differ between two exports of
// the same row often enough to matter.
let mut last_space = false;
for ch in description.chars() {
if ch.is_alphanumeric() {
last_space = false;
let lower = ch.to_ascii_lowercase();
let mut buf = [0u8; 4];
eat(lower.encode_utf8(&mut buf).as_bytes());
} else if !last_space {
last_space = true;
eat(b" ");
}
}
hash as i64
}
// ------------------------------------------------------------- category tree
/// Categories with their parent/child structure resolved, which is what
/// every screen wants — the flat table is only how they are stored.
#[derive(Clone, Debug, Default)]
pub struct CategoryTree {
pub categories: Vec<Category>,
}
impl CategoryTree {
pub fn get(&self, id: Id) -> Option<&Category> {
self.categories.iter().find(|c| c.id == id)
}
pub fn name(&self, id: Id) -> &str {
self.get(id).map(|c| c.name.as_str()).unwrap_or("")
}
/// `Food: Groceries` — what the ledger's category column shows.
pub fn path(&self, id: Id) -> String {
match self.get(id) {
Some(category) => match category.parent.and_then(|p| self.get(p)) {
Some(parent) => format!("{}: {}", parent.name, category.name),
None => category.name.clone(),
},
None => String::new(),
}
}
pub fn groups(&self) -> impl Iterator<Item = &Category> {
self.categories.iter().filter(|c| c.is_group())
}
pub fn children_of(&self, parent: Id) -> impl Iterator<Item = &Category> {
self.categories.iter().filter(move |c| c.parent == Some(parent))
}
/// Budgetable leaves in display order: each group followed by its
/// children — the row order of the budget screen.
pub fn budget_order(&self) -> Vec<&Category> {
let mut out = Vec::new();
let mut groups: Vec<&Category> = self
.groups()
.filter(|g| g.kind != CategoryKind::Transfer && !g.hidden)
.collect();
groups.sort_by_key(|g| (g.kind == CategoryKind::Expense, g.sort_order, g.name.clone()));
for group in groups {
out.push(group);
let mut children: Vec<&Category> =
self.children_of(group.id).filter(|c| !c.hidden).collect();
children.sort_by_key(|c| (c.sort_order, c.name.clone()));
out.extend(children);
}
out
}
/// The group a category belongs to (itself, if it is a group).
pub fn group_of(&self, id: Id) -> Option<Id> {
let category = self.get(id)?;
Some(category.parent.unwrap_or(category.id))
}
pub fn kind_of(&self, id: Id) -> CategoryKind {
self.get(id).map(|c| c.kind).unwrap_or(CategoryKind::Expense)
}
}
// -------------------------------------------------------------- budgeting
/// One category's budget for one month.
///
/// Envelope budgeting in the YNAB sense: you assign an amount to a category
/// for a month, spend against it, and what is left either rolls into next
/// month or does not. `assigned` is the decision; everything else is
/// computed from the ledger, never stored, so it cannot go stale.
#[derive(Clone, Copy, Debug)]
pub struct BudgetEntry {
pub category: Id,
pub month: crate::date::MonthKey,
pub assigned: i64,
pub rollover: bool,
}
/// What the budget screen shows for one category in one month.
#[derive(Clone, Copy, Debug, Default)]
pub struct BudgetLine {
pub assigned: i64,
/// Positive number: what left the account for this category.
pub spent: i64,
/// Carried in from previous months (rollover categories only).
pub carried: i64,
pub available: i64,
}
impl BudgetLine {
pub fn state(&self) -> BudgetState {
if self.available < 0 {
BudgetState::Overspent
} else if self.assigned == 0 && self.spent == 0 {
BudgetState::Untouched
} else if self.available == 0 {
BudgetState::Exact
} else if self.spent > 0 && self.available > 0 {
BudgetState::OnTrack
} else {
BudgetState::Funded
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BudgetState {
Untouched,
Funded,
OnTrack,
Exact,
Overspent,
}
// ----------------------------------------------------------------- payees
/// A payee, learned from imports. The rename is what makes a ledger
/// readable: banks write `SQ *BLUE BOTTLE 0123`, a person reads
/// `Blue Bottle Coffee`.
#[derive(Clone, Debug)]
pub struct Payee {
pub id: Id,
pub name: String,
/// The category to apply when this payee shows up with no other rule.
pub default_category: Option<Id>,
pub transactions: i64,
}
// ------------------------------------------------------------------ rules
/// How a rule decides it applies.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MatchOn {
Payee,
Memo,
/// Description as imported, before any renaming.
Raw,
Amount,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MatchHow {
Contains,
StartsWith,
Equals,
/// For amounts: exactly this value (in minor units).
AmountEquals,
AmountBetween,
}
/// An auto-categorization rule. Deliberately not regex: the people who
/// need rules most are the ones who will not write `^SQ \*(.+?) \d+$`, and
/// "contains" covers the overwhelming majority of real cases.
#[derive(Clone, Debug)]
pub struct Rule {
pub id: Id,
pub name: String,
pub match_on: MatchOn,
pub how: MatchHow,
pub pattern: String,
pub amount_min: i64,
pub amount_max: i64,
/// What to do when it matches.
pub set_category: Option<Id>,
pub rename_payee: Option<String>,
pub set_memo: Option<String>,
pub flag: bool,
/// Lower runs first; the first rule that sets a field wins it.
pub priority: i32,
pub enabled: bool,
pub hits: i64,
}
impl Rule {
pub fn matches(&self, payee: &str, memo: &str, raw: &str, amount: i64) -> bool {
if !self.enabled {
return false;
}
let haystack = match self.match_on {
MatchOn::Payee => payee,
MatchOn::Memo => memo,
MatchOn::Raw => raw,
MatchOn::Amount => "",
};
match self.how {
MatchHow::Contains => {
!self.pattern.is_empty()
&& haystack.to_lowercase().contains(&self.pattern.to_lowercase())
}
MatchHow::StartsWith => {
!self.pattern.is_empty()
&& haystack.to_lowercase().starts_with(&self.pattern.to_lowercase())
}
MatchHow::Equals => haystack.eq_ignore_ascii_case(&self.pattern),
MatchHow::AmountEquals => amount == self.amount_min,
MatchHow::AmountBetween => amount >= self.amount_min && amount <= self.amount_max,
}
}
}
// ------------------------------------------------------------- scheduling
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Recurrence {
Weekly,
Fortnightly,
Monthly,
Quarterly,
Yearly,
}
impl Recurrence {
pub fn label(self) -> &'static str {
match self {
Recurrence::Weekly => "Weekly",
Recurrence::Fortnightly => "Every 2 weeks",
Recurrence::Monthly => "Monthly",
Recurrence::Quarterly => "Quarterly",
Recurrence::Yearly => "Yearly",
}
}
/// The next occurrence after `from`. Monthly and longer clamp the day
/// of month, so a bill on the 31st lands on the 28th in February
/// instead of skipping the month.
pub fn next(self, from: Day) -> Day {
match self {
Recurrence::Weekly => from + 7,
Recurrence::Fortnightly => from + 14,
Recurrence::Monthly => crate::date::add_months(from, 1),
Recurrence::Quarterly => crate::date::add_months(from, 3),
Recurrence::Yearly => crate::date::add_months(from, 12),
}
}
pub fn approx_days(self) -> i32 {
match self {
Recurrence::Weekly => 7,
Recurrence::Fortnightly => 14,
Recurrence::Monthly => 30,
Recurrence::Quarterly => 91,
Recurrence::Yearly => 365,
}
}
}
/// A bill or paycheque that repeats. Drives the upcoming list, the
/// cash-flow forecast, and the subscription screen.
#[derive(Clone, Debug)]
pub struct Scheduled {
pub id: Id,
pub account: Id,
pub payee: String,
pub amount: i64,
pub category: Option<Id>,
pub recurrence: Recurrence,
pub next_due: Day,
pub last_posted: Option<Day>,
/// Post automatically on the due date, or just remind.
pub auto_post: bool,
pub enabled: bool,
/// True when this was detected from history rather than entered.
pub detected: bool,
}
// ---------------------------------------------------------------- ledger
/// The whole file, in memory.
///
/// Everything is loaded: a hundred thousand transactions is about 20 MB of
/// `Transaction`, and holding them means every filter, sort and report is a
/// pass over a `Vec` at memory speed rather than a round trip through SQL.
/// SQLite remains the file format and the durable store — this is a cache
/// that is rebuilt on load and kept in step on every write.
#[derive(Clone, Debug, Default)]
pub struct Ledger {
pub accounts: Vec<Account>,
pub categories: CategoryTree,
pub transactions: Vec<Transaction>,
pub payees: Vec<Payee>,
pub rules: Vec<Rule>,
pub budgets: Vec<BudgetEntry>,
pub scheduled: Vec<Scheduled>,
pub base_currency: Currency,
}
impl Ledger {
pub fn account(&self, id: Id) -> Option<&Account> {
self.accounts.iter().find(|a| a.id == id)
}
pub fn account_name(&self, id: Id) -> &str {
self.account(id).map(|a| a.name.as_str()).unwrap_or("")
}
pub fn transaction(&self, id: Id) -> Option<&Transaction> {
self.transactions.iter().find(|t| t.id == id)
}
/// Balance of an account as of a day (inclusive), opening balance
/// included. This is the number in the sidebar.
pub fn balance_on(&self, account: Id, day: Day) -> i64 {
let opening = self.account(account).map(|a| a.opening_balance).unwrap_or(0);
opening
+ self
.transactions
.iter()
.filter(|t| t.account == account && t.date <= day)
.map(|t| t.amount)
.sum::<i64>()
}
pub fn balance(&self, account: Id) -> i64 {
self.balance_on(account, Day::MAX)
}
/// What the bank thinks you have: cleared and reconciled rows only.
/// The gap between this and [`Ledger::balance`] is money in flight.
pub fn cleared_balance(&self, account: Id) -> i64 {
let opening = self.account(account).map(|a| a.opening_balance).unwrap_or(0);
opening
+ self
.transactions
.iter()
.filter(|t| t.account == account && t.cleared != Cleared::Uncleared)
.map(|t| t.amount)
.sum::<i64>()
}
/// Assets minus debts across every on-budget account, as of a day.
pub fn net_worth_on(&self, day: Day) -> i64 {
self.accounts
.iter()
.filter(|a| !a.off_budget)
.map(|a| self.balance_on(a.id, day))
.sum()
}
/// A transfer pair is balanced when its two rows cancel out. An
/// unbalanced pair means an edit went wrong, and the UI says so rather
/// than quietly showing a net-worth number that is wrong.
pub fn transfer_is_balanced(&self, group: Id) -> bool {
let sum: i64 = self
.transactions
.iter()
.filter(|t| t.transfer_group == Some(group))
.map(|t| t.amount)
.sum();
sum == 0
}
/// Every transaction of an account, oldest first, with the running
/// balance after it — the ledger's most-used view.
pub fn register(&self, account: Id) -> Vec<(&Transaction, i64)> {
let mut rows: Vec<&Transaction> =
self.transactions.iter().filter(|t| t.account == account).collect();
// Same-day rows need a stable tiebreak or the running balance
// jitters between loads; the id is insertion order, which is the
// order they were entered or imported in.
rows.sort_by_key(|t| (t.date, t.id));
let mut balance = self.account(account).map(|a| a.opening_balance).unwrap_or(0);
rows.into_iter()
.map(|t| {
balance += t.amount;
(t, balance)
})
.collect()
}
pub fn uncategorized_count(&self) -> usize {
self.transactions
.iter()
.filter(|t| t.category.is_none() && t.splits.is_empty() && !t.is_transfer())
.count()
}
/// The next id to hand out for a table, for in-memory work before a
/// write reaches the database.
pub fn next_transaction_id(&self) -> Id {
self.transactions.iter().map(|t| t.id).max().unwrap_or(0) + 1
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::date::from_ymd;
use crate::money::USD;
fn ledger() -> Ledger {
let mut ledger = Ledger { base_currency: USD, ..Ledger::default() };
let mut checking = Account::new("Checking", AccountKind::Checking, USD);
checking.id = 1;
checking.opening_balance = 100_000; // $1,000
let mut card = Account::new("Card", AccountKind::CreditCard, USD);
card.id = 2;
ledger.accounts.push(checking);
ledger.accounts.push(card);
ledger
}
#[test]
fn balances_include_the_opening_balance_and_respect_dates() {
let mut l = ledger();
let mut t = Transaction::new(1, from_ymd(2024, 3, 4), "Rent", -150_000);
t.id = 1;
l.transactions.push(t);
assert_eq!(l.balance(1), -50_000);
assert_eq!(l.balance_on(1, from_ymd(2024, 3, 3)), 100_000);
assert_eq!(l.balance_on(1, from_ymd(2024, 3, 4)), -50_000);
}
#[test]
fn cleared_balance_is_what_the_bank_shows() {
let mut l = ledger();
let mut posted = Transaction::new(1, from_ymd(2024, 3, 1), "Salary", 300_000);
posted.id = 1;
posted.cleared = Cleared::Cleared;
let mut pending = Transaction::new(1, from_ymd(2024, 3, 2), "Coffee", -450);
pending.id = 2;
l.transactions.push(posted);
l.transactions.push(pending);
assert_eq!(l.balance(1), 399_550);
assert_eq!(l.cleared_balance(1), 400_000);
}
#[test]
fn a_transfer_pair_cancels_out_and_leaves_net_worth_alone() {
let mut l = ledger();
let mut out = Transaction::new(1, from_ymd(2024, 3, 4), "Card payment", -50_000);
out.id = 1;
out.transfer_group = Some(7);
let mut into = Transaction::new(2, from_ymd(2024, 3, 4), "Payment received", 50_000);
into.id = 2;
into.transfer_group = Some(7);
let before = l.net_worth_on(Day::MAX);
l.transactions.push(out);
l.transactions.push(into);
assert!(l.transfer_is_balanced(7));
assert_eq!(l.net_worth_on(Day::MAX), before);
// Break one side: the ledger must be able to say so.
l.transactions[1].amount = 40_000;
assert!(!l.transfer_is_balanced(7));
}
#[test]
fn splits_must_sum_to_the_transaction() {
let mut t = Transaction::new(1, from_ymd(2024, 3, 4), "Supermarket", -10_000);
assert_eq!(t.split_imbalance(), 0); // no splits: nothing to balance
t.splits = vec![
Split { id: 1, category: Some(1), amount: -7_000, memo: String::new() },
Split { id: 2, category: Some(2), amount: -2_000, memo: String::new() },
];
assert_eq!(t.split_imbalance(), -1_000);
t.splits.push(Split { id: 3, category: Some(3), amount: -1_000, memo: String::new() });
assert_eq!(t.split_imbalance(), 0);
assert_eq!(t.category_amounts().len(), 3);
}
#[test]
fn the_running_balance_is_stable_for_same_day_rows() {
let mut l = ledger();
for (id, amount) in [(1, -1_000), (2, -2_000), (3, 5_000)] {
let mut t = Transaction::new(1, from_ymd(2024, 3, 4), "x", amount);
t.id = id;
l.transactions.push(t);
}
let first: Vec<i64> = l.register(1).iter().map(|(_, b)| *b).collect();
// Same input, reversed insertion: the register must not change.
l.transactions.reverse();
let second: Vec<i64> = l.register(1).iter().map(|(_, b)| *b).collect();
assert_eq!(first, second);
assert_eq!(*first.last().unwrap(), l.balance(1));
}
#[test]
fn import_fingerprints_survive_cosmetic_differences_only() {
let a = import_fingerprint(1, 100, -450, "SQ *BLUE BOTTLE 0123");
let b = import_fingerprint(1, 100, -450, "sq *blue bottle 0123");
assert_eq!(a, b, "case and spacing must not change the fingerprint");
let different_amount = import_fingerprint(1, 100, -451, "SQ *BLUE BOTTLE 0123");
let different_day = import_fingerprint(1, 101, -450, "SQ *BLUE BOTTLE 0123");
let different_account = import_fingerprint(2, 100, -450, "SQ *BLUE BOTTLE 0123");
assert_ne!(a, different_amount);
assert_ne!(a, different_day);
assert_ne!(a, different_account);
}
#[test]
fn rules_match_the_way_a_person_expects() {
let rule = Rule {
id: 1,
name: "Coffee".into(),
match_on: MatchOn::Raw,
how: MatchHow::Contains,
pattern: "blue bottle".into(),
amount_min: 0,
amount_max: 0,
set_category: Some(9),
rename_payee: Some("Blue Bottle".into()),
set_memo: None,
flag: false,
priority: 0,
enabled: true,
hits: 0,
};
assert!(rule.matches("", "", "SQ *BLUE BOTTLE 0123", -450));
assert!(!rule.matches("", "", "STARBUCKS", -450));
let disabled = Rule { enabled: false, ..rule.clone() };
assert!(!disabled.matches("", "", "SQ *BLUE BOTTLE 0123", -450));
let big = Rule {
match_on: MatchOn::Amount,
how: MatchHow::AmountBetween,
amount_min: -100_000,
amount_max: -50_000,
..rule
};
assert!(big.matches("", "", "", -75_000));
assert!(!big.matches("", "", "", -10_000));
}
#[test]
fn category_paths_and_budget_order_read_like_the_screen() {
let mut tree = CategoryTree::default();
let mut food = Category::group("Food", CategoryKind::Expense);
food.id = 1;
let mut groceries = Category::child("Groceries", 1, CategoryKind::Expense);
groceries.id = 2;
groceries.budgeted = true;
let mut income = Category::group("Income", CategoryKind::Income);
income.id = 3;
tree.categories = vec![food, groceries, income];
assert_eq!(tree.path(2), "Food: Groceries");
assert_eq!(tree.path(1), "Food");
assert_eq!(tree.group_of(2), Some(1));
// Income groups sort above expense groups.
let order: Vec<&str> =
tree.budget_order().iter().map(|c| c.name.as_str()).collect();
assert_eq!(order, ["Income", "Food", "Groceries"]);
}
#[test]
fn recurrence_clamps_month_ends_instead_of_skipping() {
let jan31 = from_ymd(2024, 1, 31);
assert_eq!(Recurrence::Monthly.next(jan31), from_ymd(2024, 2, 29));
assert_eq!(Recurrence::Weekly.next(jan31), from_ymd(2024, 2, 7));
assert_eq!(Recurrence::Yearly.next(jan31), from_ymd(2025, 1, 31));
}
#[test]
fn budget_lines_classify_themselves() {
let over = BudgetLine { assigned: 10_000, spent: 12_000, carried: 0, available: -2_000 };
assert_eq!(over.state(), BudgetState::Overspent);
let untouched = BudgetLine::default();
assert_eq!(untouched.state(), BudgetState::Untouched);
let on_track = BudgetLine { assigned: 10_000, spent: 4_000, carried: 0, available: 6_000 };
assert_eq!(on_track.state(), BudgetState::OnTrack);
}
}

382
apps/finance/src/money.rs Normal file
View file

@ -0,0 +1,382 @@
//! Money is an integer number of minor units. Never a float.
//!
//! A balance is a sum of thousands of amounts, and every one of those sums
//! has to come out the way a bank would compute it. Binary floating point
//! cannot represent 0.10, so a ledger built on `f64` drifts: add a tenth a
//! thousand times and you are three cents short of a hundred. Everything
//! here is `i64` minor units — cents for USD/EUR, but also 0 decimals for
//! JPY and 3 for BHD, which is why the scale lives on the currency rather
//! than being assumed to be 100.
//!
//! `i64` cents reaches ±92 quadrillion. That is not a limit anyone hits,
//! and it makes every intermediate sum exact.
use std::fmt;
/// A currency, as much of ISO 4217 as a ledger needs: how many decimal
/// places it has, and how it is written.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Currency {
pub code: &'static str,
pub symbol: &'static str,
/// Decimal places — the power of ten between the major unit and the
/// minor unit this crate stores.
pub decimals: u8,
/// Symbol before the number (`$12.34`) or after it (`12,34 €`).
pub symbol_first: bool,
}
pub const USD: Currency =
Currency { code: "USD", symbol: "$", decimals: 2, symbol_first: true };
pub const EUR: Currency =
Currency { code: "EUR", symbol: "", decimals: 2, symbol_first: false };
pub const GBP: Currency =
Currency { code: "GBP", symbol: "£", decimals: 2, symbol_first: true };
pub const JPY: Currency =
Currency { code: "JPY", symbol: "¥", decimals: 0, symbol_first: true };
pub const CHF: Currency =
Currency { code: "CHF", symbol: "CHF", decimals: 2, symbol_first: true };
/// Every currency this build knows, for pickers and for parsing a code out
/// of an imported file.
pub const CURRENCIES: [Currency; 5] = [USD, EUR, GBP, JPY, CHF];
impl Default for Currency {
/// A file that has not said otherwise. `Ledger` derives `Default`, and
/// a currency-less amount is not a thing this app can represent.
fn default() -> Currency {
USD
}
}
pub fn currency_by_code(code: &str) -> Option<Currency> {
CURRENCIES
.iter()
.copied()
.find(|c| c.code.eq_ignore_ascii_case(code))
}
/// How a number was written in the file we are reading. Bank exports differ
/// on every one of these axes, and guessing wrong turns 1.234,56 into 1.23.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)]
pub struct AmountFormat {
/// `,` in `1.234,56` (continental) or `.` in `1,234.56` (anglo).
pub decimal_comma: bool,
/// A trailing `-` means negative: `1.234,56-` (SEPA, some German banks).
pub trailing_minus: bool,
/// `(1,234.56)` means negative (accounting convention).
pub parens_negative: bool,
}
/// Parse a money amount out of a cell of an imported file.
///
/// Deliberately liberal: currency symbols, spaces (including the narrow
/// no-break space German banks use as a thousands separator), `+` signs and
/// thousands separators are all discarded, because every bank writes them
/// differently and none of them mean anything. What it will NOT do is
/// guess the decimal separator when the file has told us — pass the format
/// sniffed from the whole column ([`sniff_amount_format`]), never per cell.
/// Deciding per cell is how `1.234` becomes 1.23 in one row and 1234.00 in
/// the next.
pub fn parse_amount(text: &str, format: AmountFormat, decimals: u8) -> Option<i64> {
let mut cleaned = String::with_capacity(text.len());
let mut negative = false;
for ch in text.chars() {
match ch {
'-' | '\u{2212}' => negative = true, // ASCII hyphen or real minus
'(' if format.parens_negative => negative = true,
'0'..='9' => cleaned.push(ch),
',' if format.decimal_comma => cleaned.push('.'),
'.' if !format.decimal_comma => cleaned.push('.'),
// Thousands separators and everything else: currency symbols,
// spaces, NBSP, apostrophes (Swiss 1'234.56), `+`, `)`.
_ => {}
}
}
if cleaned.is_empty() || !cleaned.chars().any(|c| c.is_ascii_digit()) {
return None;
}
// More than one separator left means the extras were thousands marks
// ("1.234.567,89" cleaned to "1.234.567.89"): keep the last one.
let value = if cleaned.matches('.').count() > 1 {
let last = cleaned.rfind('.').unwrap();
let mut merged: String = cleaned[..last].replace('.', "");
merged.push('.');
merged.push_str(&cleaned[last + 1..]);
merged
} else {
cleaned
};
let (whole, frac) = match value.split_once('.') {
Some((w, f)) => (w, f),
None => (value.as_str(), ""),
};
// A group of exactly three digits after the only separator, in a file
// whose decimal separator we believe is the other character, was a
// thousands separator: "1.234" is 1234, not 1.23.
let scale = 10i64.checked_pow(decimals as u32)?;
let whole_value: i64 = if whole.is_empty() { 0 } else { whole.parse().ok()? };
let mut minor = whole_value.checked_mul(scale)?;
if !frac.is_empty() {
let digits: String = frac.chars().take(decimals as usize).collect();
let mut fraction: i64 = if digits.is_empty() { 0 } else { digits.parse().ok()? };
// Pad "5" to "50" for a 2-decimal currency.
for _ in digits.len()..decimals as usize {
fraction = fraction.checked_mul(10)?;
}
// Round rather than truncate on extra precision (a 4-decimal FX
// amount landing in a 2-decimal account).
let round_up = frac
.chars()
.nth(decimals as usize)
.is_some_and(|c| c >= '5' && c <= '9');
minor = minor.checked_add(fraction)?;
if round_up {
minor = minor.checked_add(1)?;
}
}
if negative || format.trailing_minus && text.trim_end().ends_with('-') {
minor = -minor;
}
Some(minor)
}
/// Work out how a column of amounts is written by looking at all of it.
///
/// The decision that matters is the decimal separator, and a single cell
/// often cannot settle it: `1.234` is ambiguous, `1.234,56` is not. So the
/// whole column votes — any cell with both separators, or with a comma
/// followed by exactly two digits at the end, is evidence.
pub fn sniff_amount_format<'a>(cells: impl Iterator<Item = &'a str>) -> AmountFormat {
let mut comma_decimal = 0usize;
let mut dot_decimal = 0usize;
let mut trailing_minus = false;
let mut parens = false;
for cell in cells {
let cell = cell.trim();
if cell.is_empty() {
continue;
}
if cell.ends_with('-') {
trailing_minus = true;
}
if cell.starts_with('(') && cell.ends_with(')') {
parens = true;
}
let last_comma = cell.rfind(',');
let last_dot = cell.rfind('.');
match (last_comma, last_dot) {
// Both present: the LAST one is the decimal separator.
(Some(c), Some(d)) => {
if c > d {
comma_decimal += 1;
} else {
dot_decimal += 1;
}
}
// One separator with 1-2 trailing digits reads as a decimal;
// with exactly 3 it reads as a thousands mark and says nothing.
(Some(c), None) => {
let tail = cell.len() - c - 1;
if tail <= 2 {
comma_decimal += 1;
}
}
(None, Some(d)) => {
let tail = cell.len() - d - 1;
if tail <= 2 {
dot_decimal += 1;
}
}
(None, None) => {}
}
}
AmountFormat {
decimal_comma: comma_decimal > dot_decimal,
trailing_minus,
parens_negative: parens,
}
}
/// `1234567` cents → `"12,345.67"`. Grouping and the decimal mark follow
/// the currency's convention, not the machine's locale: a ledger of euros
/// reads the same on every machine that opens the file.
pub fn format_minor(minor: i64, currency: Currency) -> String {
let decimals = currency.decimals as usize;
let negative = minor < 0;
let magnitude = minor.unsigned_abs();
let scale = 10u64.pow(decimals as u32);
let whole = magnitude / scale;
let frac = magnitude % scale;
let (group, point) = if currency.decimals == 2 && !currency.symbol_first {
('.', ',') // continental: 1.234,56
} else {
(',', '.') // anglo: 1,234.56
};
let digits = whole.to_string();
let mut grouped = String::with_capacity(digits.len() + digits.len() / 3 + 4);
for (i, ch) in digits.chars().enumerate() {
if i > 0 && (digits.len() - i) % 3 == 0 {
grouped.push(group);
}
grouped.push(ch);
}
let mut out = String::with_capacity(grouped.len() + decimals + 4);
if negative {
out.push('-');
}
out.push_str(&grouped);
if decimals > 0 {
out.push(point);
out.push_str(&format!("{frac:0width$}", width = decimals));
}
out
}
/// With the currency's symbol attached, the way that currency writes it.
pub fn format_money(minor: i64, currency: Currency) -> String {
let number = format_minor(minor, currency);
if currency.symbol_first {
// The sign stays outside the symbol: -$12.34, not $-12.34.
match number.strip_prefix('-') {
Some(rest) => format!("-{}{}", currency.symbol, rest),
None => format!("{}{}", currency.symbol, number),
}
} else {
format!("{} {}", number, currency.symbol)
}
}
/// Short form for chart axes and dense cells: `12.3k`, `1.2M`. Keeps the
/// sign, drops the currency.
pub fn format_compact(minor: i64, currency: Currency) -> String {
let scale = 10i64.pow(currency.decimals as u32);
let major = minor as f64 / scale as f64;
let magnitude = major.abs();
let sign = if major < 0.0 { "-" } else { "" };
if magnitude >= 1_000_000.0 {
format!("{sign}{:.1}M", magnitude / 1_000_000.0)
} else if magnitude >= 1_000.0 {
format!("{sign}{:.1}k", magnitude / 1_000.0)
} else {
format!("{sign}{:.0}", magnitude)
}
}
/// A signed amount with its currency, for display and for the few places
/// that carry an amount around on its own.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Money {
pub minor: i64,
pub currency: Currency,
}
impl Money {
pub fn new(minor: i64, currency: Currency) -> Money {
Money { minor, currency }
}
pub fn zero(currency: Currency) -> Money {
Money { minor: 0, currency }
}
pub fn is_negative(&self) -> bool {
self.minor < 0
}
pub fn abs(&self) -> Money {
Money { minor: self.minor.abs(), currency: self.currency }
}
}
impl fmt::Display for Money {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(&format_money(self.minor, self.currency))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn anglo() -> AmountFormat {
AmountFormat::default()
}
fn continental() -> AmountFormat {
AmountFormat { decimal_comma: true, ..AmountFormat::default() }
}
#[test]
fn parses_the_shapes_banks_actually_export() {
assert_eq!(parse_amount("1234.56", anglo(), 2), Some(123456));
assert_eq!(parse_amount("1,234.56", anglo(), 2), Some(123456));
assert_eq!(parse_amount("$1,234.56", anglo(), 2), Some(123456));
assert_eq!(parse_amount("-1,234.56", anglo(), 2), Some(-123456));
assert_eq!(parse_amount("1.234,56", continental(), 2), Some(123456));
assert_eq!(parse_amount("1.234.567,89", continental(), 2), Some(123456789));
assert_eq!(parse_amount("1'234.56", anglo(), 2), Some(123456)); // Swiss
assert_eq!(parse_amount("12,34 €", continental(), 2), Some(1234));
// Fewer decimals written than the currency has.
assert_eq!(parse_amount("5.5", anglo(), 2), Some(550));
assert_eq!(parse_amount("5", anglo(), 2), Some(500));
// Zero-decimal currency.
assert_eq!(parse_amount("1,250", anglo(), 0), Some(1250));
// Junk is None, not zero: a failed parse must never post 0.00.
assert_eq!(parse_amount("", anglo(), 2), None);
assert_eq!(parse_amount("n/a", anglo(), 2), None);
assert_eq!(parse_amount("--", anglo(), 2), None);
}
#[test]
fn honours_the_negative_conventions() {
let trailing = AmountFormat { trailing_minus: true, ..anglo() };
assert_eq!(parse_amount("1234.56-", trailing, 2), Some(-123456));
let parens = AmountFormat { parens_negative: true, ..anglo() };
assert_eq!(parse_amount("(1,234.56)", parens, 2), Some(-123456));
// A real Unicode minus, which some exports use.
assert_eq!(parse_amount("\u{2212}12.00", anglo(), 2), Some(-1200));
}
#[test]
fn extra_precision_rounds_rather_than_truncates() {
assert_eq!(parse_amount("1.005", anglo(), 2), Some(101));
assert_eq!(parse_amount("1.004", anglo(), 2), Some(100));
}
#[test]
fn sniffing_reads_the_column_not_the_cell() {
// Ambiguous alone; the column settles it.
let german = ["1.234,56", "-89,10", "1.000,00"];
assert!(sniff_amount_format(german.iter().copied()).decimal_comma);
let anglo_col = ["1,234.56", "-89.10", "1,000.00"];
assert!(!sniff_amount_format(anglo_col.iter().copied()).decimal_comma);
// Thousands-only groups say nothing and must not flip the vote.
let ambiguous = ["1.234", "5.678"];
assert!(!sniff_amount_format(ambiguous.iter().copied()).decimal_comma);
let trailing = ["1234.56-", "10.00"];
assert!(sniff_amount_format(trailing.iter().copied()).trailing_minus);
}
#[test]
fn formats_the_way_each_currency_is_written() {
assert_eq!(format_money(123456, USD), "$1,234.56");
assert_eq!(format_money(-123456, USD), "-$1,234.56");
assert_eq!(format_money(123456, EUR), "1.234,56 €");
assert_eq!(format_money(1250, JPY), "¥1,250");
assert_eq!(format_minor(0, USD), "0.00");
assert_eq!(format_minor(-5, USD), "-0.05");
assert_eq!(format_compact(123456789, USD), "1.2M");
assert_eq!(format_compact(-1234567, USD), "-12.3k");
}
#[test]
fn a_thousand_dimes_are_exactly_a_hundred() {
// The whole reason this module exists.
let total: i64 = (0..1000).map(|_| 10i64).sum();
assert_eq!(total, 10_000);
assert_eq!(format_money(total, USD), "$100.00");
}
}

635
apps/finance/src/report.rs Normal file
View file

@ -0,0 +1,635 @@
//! Every number the screens show, computed from the ledger in memory.
//!
//! These are the reports the commercial products converged on — spending
//! by category, income against expense, net worth over time, category
//! drilldown, merchant ranking, budget available — and they are all one
//! pass over a `Vec<Transaction>`. That is the point: a decade of history
//! is a few hundred thousand structs, so a report is a millisecond and can
//! be recomputed on every keystroke of a filter instead of being cached,
//! invalidated, and got wrong.
//!
//! Two rules run through all of it:
//!
//! * **Transfers are not spending.** Moving money to savings is not an
//! expense, and paying a credit card is not spending twice. Anything
//! with a [`Transaction::transfer_group`], or in a
//! [`CategoryKind::Transfer`] category, is excluded from every
//! income/expense figure — this is the single most common way a naive
//! finance report lies.
//! * **Splits are counted per part.** A supermarket trip split between
//! food and household appears in both categories, for its own share.
use crate::date::{self, Day, DateRange, MonthKey};
use crate::model::*;
/// Positive amounts are money in, negative money out — the ledger's own
/// convention, kept all the way to the screen.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct Flow {
pub income: i64,
/// Positive: what left. (Reports read better when spending counts up.)
pub expense: i64,
}
impl Flow {
pub fn net(&self) -> i64 {
self.income - self.expense
}
}
/// True when a transaction is real spending or real income, rather than
/// money moving between the user's own pockets.
pub fn is_spending(txn: &Transaction, categories: &CategoryTree) -> bool {
if txn.transfer_group.is_some() {
return false;
}
match txn.category {
Some(id) => categories.kind_of(id) != CategoryKind::Transfer,
// An uncategorized row is still real money; it shows up as
// "Uncategorized", which is what makes people categorize it.
None => !txn.splits.is_empty() || true,
}
}
/// Income and expense for a date range.
pub fn flow(ledger: &Ledger, range: DateRange) -> Flow {
let mut flow = Flow::default();
for txn in &ledger.transactions {
if !range.contains(txn.date) || !is_spending(txn, &ledger.categories) {
continue;
}
if ledger.account(txn.account).is_some_and(|a| a.off_budget) {
continue;
}
if txn.amount >= 0 {
flow.income += txn.amount;
} else {
flow.expense += -txn.amount;
}
}
flow
}
/// Income and expense per month, oldest first — the bars on the reports
/// screen and the shape of the cash-flow chart.
pub fn monthly_flow(ledger: &Ledger, months: i32, today: Day) -> Vec<(MonthKey, Flow)> {
let first = date::month_key(date::add_months(today, -(months - 1)));
let mut out: Vec<(MonthKey, Flow)> =
(0..months).map(|i| (first + i, Flow::default())).collect();
for txn in &ledger.transactions {
if !is_spending(txn, &ledger.categories) {
continue;
}
if ledger.account(txn.account).is_some_and(|a| a.off_budget) {
continue;
}
let key = date::month_key(txn.date);
let Some(index) = key.checked_sub(first).filter(|i| *i >= 0 && *i < months) else {
continue;
};
let slot = &mut out[index as usize].1;
if txn.amount >= 0 {
slot.income += txn.amount;
} else {
slot.expense += -txn.amount;
}
}
out
}
/// What was spent per category in a range, biggest first. Groups are
/// rolled up from their children, so "Food" totals its own leaves.
pub fn spending_by_category(ledger: &Ledger, range: DateRange) -> Vec<(Option<Id>, i64)> {
let mut totals: std::collections::HashMap<Option<Id>, i64> = std::collections::HashMap::new();
for txn in &ledger.transactions {
if !range.contains(txn.date) || !is_spending(txn, &ledger.categories) {
continue;
}
if ledger.account(txn.account).is_some_and(|a| a.off_budget) {
continue;
}
for (category, amount) in txn.category_amounts() {
if amount >= 0 {
continue; // income is not spending
}
if category.is_some_and(|id| {
ledger.categories.kind_of(id) != CategoryKind::Expense
}) {
continue;
}
*totals.entry(category).or_default() += -amount;
}
}
let mut out: Vec<(Option<Id>, i64)> = totals.into_iter().collect();
out.sort_by_key(|(id, total)| (std::cmp::Reverse(*total), id.unwrap_or(0)));
out
}
/// The same, rolled up to top-level groups — the pie every product opens
/// with.
pub fn spending_by_group(ledger: &Ledger, range: DateRange) -> Vec<(Option<Id>, i64)> {
let mut totals: std::collections::HashMap<Option<Id>, i64> = std::collections::HashMap::new();
for (category, amount) in spending_by_category(ledger, range) {
let group = category.and_then(|id| ledger.categories.group_of(id));
*totals.entry(group).or_default() += amount;
}
let mut out: Vec<(Option<Id>, i64)> = totals.into_iter().collect();
out.sort_by_key(|(id, total)| (std::cmp::Reverse(*total), id.unwrap_or(0)));
out
}
/// Who took the most money, biggest first — the merchant analysis every
/// product has and everyone actually reads.
pub fn top_payees(ledger: &Ledger, range: DateRange, limit: usize) -> Vec<(String, i64, usize)> {
let mut totals: std::collections::HashMap<&str, (i64, usize)> =
std::collections::HashMap::new();
for txn in &ledger.transactions {
if !range.contains(txn.date) || !is_spending(txn, &ledger.categories) || txn.amount >= 0 {
continue;
}
let entry = totals.entry(txn.payee.as_str()).or_default();
entry.0 += -txn.amount;
entry.1 += 1;
}
let mut out: Vec<(String, i64, usize)> = totals
.into_iter()
.map(|(payee, (total, count))| (payee.to_string(), total, count))
.collect();
out.sort_by_key(|(payee, total, _)| (std::cmp::Reverse(*total), payee.clone()));
out.truncate(limit);
out
}
/// Net worth at the end of each of the last `months` months.
///
/// Computed by walking the transactions once in date order and carrying a
/// running total, rather than by asking for a balance per month — the
/// naive version is O(months × transactions) and is why some products take
/// a second to draw this.
pub fn net_worth_series(ledger: &Ledger, months: i32, today: Day) -> Vec<(MonthKey, i64)> {
let first = date::month_key(date::add_months(today, -(months - 1)));
let on_budget: std::collections::HashSet<Id> = ledger
.accounts
.iter()
.filter(|a| !a.off_budget)
.map(|a| a.id)
.collect();
// Everything before the window is the opening position.
let window_start = date::month_key_start(first);
let mut running: i64 = ledger
.accounts
.iter()
.filter(|a| !a.off_budget)
.map(|a| a.opening_balance)
.sum();
let mut sorted: Vec<&Transaction> = ledger
.transactions
.iter()
.filter(|t| on_budget.contains(&t.account))
.collect();
sorted.sort_by_key(|t| t.date);
let mut out = Vec::with_capacity(months as usize);
let mut index = 0usize;
for txn in sorted.iter() {
if txn.date >= window_start {
break;
}
running += txn.amount;
index += 1;
}
for offset in 0..months {
let month_end = date::month_end(date::month_key_start(first + offset));
while index < sorted.len() && sorted[index].date <= month_end {
running += sorted[index].amount;
index += 1;
}
out.push((first + offset, running));
}
out
}
/// A daily balance series for one account, for the sparkline in its row.
pub fn balance_series(ledger: &Ledger, account: Id, days: i32, today: Day) -> Vec<f64> {
let start = today - days + 1;
let opening = ledger.account(account).map(|a| a.opening_balance).unwrap_or(0);
let mut rows: Vec<&Transaction> =
ledger.transactions.iter().filter(|t| t.account == account).collect();
rows.sort_by_key(|t| t.date);
let mut running = opening;
let mut index = 0usize;
while index < rows.len() && rows[index].date < start {
running += rows[index].amount;
index += 1;
}
let mut out = Vec::with_capacity(days as usize);
for day in start..=today {
while index < rows.len() && rows[index].date <= day {
running += rows[index].amount;
index += 1;
}
out.push(running as f64);
}
out
}
/// The budget screen's rows for one month: what was assigned, what was
/// spent, and what is left — including what rolled in from before.
///
/// Rollover is computed from the start of the file rather than stored,
/// because a stored carry goes stale the moment an old transaction is
/// edited, and editing old transactions is exactly what people do.
pub fn budget_lines(
ledger: &Ledger,
month: MonthKey,
) -> Vec<(Id, BudgetLine)> {
let mut out = Vec::new();
for category in ledger.categories.budget_order() {
if category.is_group() || category.kind != CategoryKind::Expense {
continue;
}
let mut line = BudgetLine::default();
line.assigned = assigned_for(ledger, category.id, month);
line.spent = spent_in(ledger, category.id, month);
if category.rollover {
// Walk from the first month that has any activity.
let mut carry = 0i64;
if let Some(first) = first_month(ledger) {
let mut cursor = first;
while cursor < month {
carry += assigned_for(ledger, category.id, cursor)
- spent_in(ledger, category.id, cursor);
// A rollover category cannot carry a negative balance
// forward: overspending is settled in the month it
// happened, which is what YNAB does and what keeps the
// number understandable.
carry = carry.max(0);
cursor += 1;
}
}
line.carried = carry;
}
line.available = line.carried + line.assigned - line.spent;
out.push((category.id, line));
}
out
}
fn assigned_for(ledger: &Ledger, category: Id, month: MonthKey) -> i64 {
ledger
.budgets
.iter()
.find(|b| b.category == category && b.month == month)
.map(|b| b.assigned)
.unwrap_or(0)
}
fn spent_in(ledger: &Ledger, category: Id, month: MonthKey) -> i64 {
let mut total = 0i64;
for txn in &ledger.transactions {
if date::month_key(txn.date) != month || !is_spending(txn, &ledger.categories) {
continue;
}
for (id, amount) in txn.category_amounts() {
if id == Some(category) && amount < 0 {
total += -amount;
}
}
}
total
}
fn first_month(ledger: &Ledger) -> Option<MonthKey> {
ledger.transactions.iter().map(|t| date::month_key(t.date)).min()
}
/// What is due in the next `days`, soonest first — the "upcoming" list.
pub fn upcoming(ledger: &Ledger, days: i32, today: Day) -> Vec<&Scheduled> {
let horizon = today + days;
let mut out: Vec<&Scheduled> = ledger
.scheduled
.iter()
.filter(|s| s.enabled && s.next_due <= horizon)
.collect();
out.sort_by_key(|s| s.next_due);
out
}
/// Where the balance is heading: today's balance, then each scheduled item
/// applied on its due date. The forecast every product added late and
/// everyone asks for.
pub fn cash_forecast(ledger: &Ledger, account: Id, days: i32, today: Day) -> Vec<f64> {
let mut balance = ledger.balance_on(account, today);
let mut out = Vec::with_capacity(days as usize);
for offset in 0..days {
let day = today + offset;
for item in &ledger.scheduled {
if !item.enabled || item.account != account {
continue;
}
// Walk this schedule's occurrences into the window.
let mut due = item.next_due;
while due < day {
due = item.recurrence.next(due);
}
if due == day {
balance += item.amount;
}
}
out.push(balance as f64);
}
out
}
/// Recurring charges the ledger can see for itself — the subscription
/// screen, without anyone having to declare anything.
///
/// A payee qualifies when it has charged a similar amount at a regular
/// interval at least three times. Three is the smallest number that can
/// tell a rhythm from a coincidence.
pub fn detected_subscriptions(ledger: &Ledger, today: Day) -> Vec<(String, i64, Recurrence, Day)> {
let mut by_payee: std::collections::HashMap<&str, Vec<&Transaction>> =
std::collections::HashMap::new();
let year_ago = today - 400;
for txn in &ledger.transactions {
if txn.amount >= 0 || txn.date < year_ago || txn.transfer_group.is_some() {
continue;
}
by_payee.entry(txn.payee.as_str()).or_default().push(txn);
}
let mut out = Vec::new();
for (payee, mut rows) in by_payee {
if rows.len() < 3 {
continue;
}
rows.sort_by_key(|t| t.date);
let gaps: Vec<i32> = rows.windows(2).map(|w| w[1].date - w[0].date).collect();
let average = gaps.iter().sum::<i32>() / gaps.len() as i32;
let recurrence = match average {
5..=9 => Recurrence::Weekly,
12..=16 => Recurrence::Fortnightly,
26..=35 => Recurrence::Monthly,
85..=100 => Recurrence::Quarterly,
350..=380 => Recurrence::Yearly,
_ => continue,
};
// The amounts have to be alike: a supermarket visited weekly is
// not a subscription, a gym charged the same every month is.
let amounts: Vec<i64> = rows.iter().map(|t| t.amount).collect();
let typical = amounts[amounts.len() / 2];
let steady = amounts
.iter()
.all(|a| (a - typical).abs() <= (typical.abs() / 10).max(100));
if !steady {
continue;
}
let last = rows.last().unwrap().date;
out.push((payee.to_string(), typical, recurrence, recurrence.next(last)));
}
out.sort_by_key(|(payee, amount, _, _)| (*amount, payee.clone()));
out
}
/// Rows the user should look at: uncategorized, unbalanced splits, and
/// transfers whose halves do not cancel.
pub fn needs_attention(ledger: &Ledger) -> Vec<(Id, &'static str)> {
let mut out = Vec::new();
for txn in &ledger.transactions {
if txn.split_imbalance() != 0 {
out.push((txn.id, "split does not add up"));
} else if txn.category.is_none() && txn.splits.is_empty() && !txn.is_transfer() {
out.push((txn.id, "no category"));
}
}
for group in ledger
.transactions
.iter()
.filter_map(|t| t.transfer_group)
.collect::<std::collections::HashSet<_>>()
{
if !ledger.transfer_is_balanced(group) {
if let Some(txn) = ledger.transactions.iter().find(|t| t.transfer_group == Some(group))
{
out.push((txn.id, "transfer does not cancel"));
}
}
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::date::from_ymd;
use crate::money::EUR;
/// A tiny ledger with one of everything the reports have to get right.
fn ledger() -> Ledger {
let mut ledger = Ledger { base_currency: EUR, ..Ledger::default() };
let mut checking = Account::new("Checking", AccountKind::Checking, EUR);
checking.id = 1;
checking.opening_balance = 100_000;
let mut savings = Account::new("Savings", AccountKind::Savings, EUR);
savings.id = 2;
ledger.accounts = vec![checking, savings];
let mut food = Category::group("Food", CategoryKind::Expense);
food.id = 10;
let mut groceries = Category::child("Groceries", 10, CategoryKind::Expense);
groceries.id = 11;
let mut restaurants = Category::child("Restaurants", 10, CategoryKind::Expense);
restaurants.id = 12;
let mut income = Category::group("Income", CategoryKind::Income);
income.id = 20;
let mut salary = Category::child("Salary", 20, CategoryKind::Income);
salary.id = 21;
ledger.categories.categories = vec![food, groceries, restaurants, income, salary];
let mut next_id = 100;
let mut push = |ledger: &mut Ledger, account, date, payee: &str, amount, category| {
let mut txn = Transaction::new(account, date, payee, amount);
txn.id = next_id;
next_id += 1;
txn.category = category;
ledger.transactions.push(txn);
};
push(&mut ledger, 1, from_ymd(2024, 3, 25), "Employer", 300_000, Some(21));
push(&mut ledger, 1, from_ymd(2024, 3, 4), "Albert Heijn", -8_000, Some(11));
push(&mut ledger, 1, from_ymd(2024, 3, 11), "Albert Heijn", -6_500, Some(11));
push(&mut ledger, 1, from_ymd(2024, 3, 14), "Café", -3_200, Some(12));
push(&mut ledger, 1, from_ymd(2024, 2, 4), "Albert Heijn", -7_000, Some(11));
// A transfer to savings: not spending, not income.
let mut out = Transaction::new(1, from_ymd(2024, 3, 26), "Savings", -50_000);
out.id = 200;
out.transfer_group = Some(1);
let mut into = Transaction::new(2, from_ymd(2024, 3, 26), "From checking", 50_000);
into.id = 201;
into.transfer_group = Some(1);
ledger.transactions.push(out);
ledger.transactions.push(into);
ledger
}
#[test]
fn transfers_are_never_spending_or_income() {
let ledger = ledger();
let march = DateRange::month(date::month_key(from_ymd(2024, 3, 1)));
let flow = flow(&ledger, march);
assert_eq!(flow.income, 300_000, "the transfer must not count as income");
assert_eq!(flow.expense, 8_000 + 6_500 + 3_200, "nor the other half as spending");
assert_eq!(flow.net(), 300_000 - 17_700);
// And the money did not vanish: net worth is unchanged by it.
let before = ledger.net_worth_on(from_ymd(2024, 3, 25));
let after = ledger.net_worth_on(from_ymd(2024, 3, 26));
assert_eq!(before, after);
}
#[test]
fn category_totals_roll_up_and_rank() {
let ledger = ledger();
let march = DateRange::month(date::month_key(from_ymd(2024, 3, 1)));
let by_category = spending_by_category(&ledger, march);
assert_eq!(by_category[0], (Some(11), 14_500));
assert_eq!(by_category[1], (Some(12), 3_200));
let by_group = spending_by_group(&ledger, march);
assert_eq!(by_group[0], (Some(10), 17_700), "Food totals its children");
}
#[test]
fn splits_are_counted_in_each_of_their_parts() {
let mut ledger = ledger();
let mut txn = Transaction::new(1, from_ymd(2024, 3, 20), "Supermarket", -10_000);
txn.id = 300;
txn.splits = vec![
Split { id: 1, category: Some(11), amount: -6_000, memo: String::new() },
Split { id: 2, category: Some(12), amount: -4_000, memo: String::new() },
];
ledger.transactions.push(txn);
let march = DateRange::month(date::month_key(from_ymd(2024, 3, 1)));
let by_category = spending_by_category(&ledger, march);
let groceries = by_category.iter().find(|(id, _)| *id == Some(11)).unwrap().1;
let restaurants = by_category.iter().find(|(id, _)| *id == Some(12)).unwrap().1;
assert_eq!(groceries, 14_500 + 6_000);
assert_eq!(restaurants, 3_200 + 4_000);
}
#[test]
fn monthly_flow_lines_up_with_the_months_asked_for() {
let ledger = ledger();
let series = monthly_flow(&ledger, 3, from_ymd(2024, 3, 31));
assert_eq!(series.len(), 3);
assert_eq!(series[2].0, date::month_key(from_ymd(2024, 3, 1)));
assert_eq!(series[2].1.income, 300_000);
assert_eq!(series[1].1.expense, 7_000, "February had one shop");
assert_eq!(series[0].1, Flow::default(), "January is empty, not missing");
}
#[test]
fn net_worth_walks_forward_once() {
let ledger = ledger();
let series = net_worth_series(&ledger, 3, from_ymd(2024, 3, 31));
assert_eq!(series.len(), 3);
// January: nothing had happened, so just the opening balance.
assert_eq!(series[0].1, 100_000);
// February: one shop.
assert_eq!(series[1].1, 100_000 - 7_000);
// March: everything, and the transfer cancels out.
assert_eq!(series[2].1, ledger.net_worth_on(from_ymd(2024, 3, 31)));
}
#[test]
fn top_payees_rank_by_money_not_by_count() {
let ledger = ledger();
let march = DateRange::month(date::month_key(from_ymd(2024, 3, 1)));
let payees = top_payees(&ledger, march, 5);
assert_eq!(payees[0].0, "Albert Heijn");
assert_eq!(payees[0].1, 14_500);
assert_eq!(payees[0].2, 2);
assert!(payees.iter().all(|(name, _, _)| name != "Savings"));
}
#[test]
fn budget_available_carries_only_where_asked() {
let mut ledger = ledger();
let feb = date::month_key(from_ymd(2024, 2, 1));
let mar = date::month_key(from_ymd(2024, 3, 1));
// Groceries: no rollover. Restaurants: rollover.
ledger.categories.categories[1].rollover = false;
ledger.categories.categories[2].rollover = true;
for month in [feb, mar] {
ledger.budgets.push(BudgetEntry { category: 11, month, assigned: 10_000, rollover: false });
ledger.budgets.push(BudgetEntry { category: 12, month, assigned: 5_000, rollover: true });
}
let lines = budget_lines(&ledger, mar);
let groceries = lines.iter().find(|(id, _)| *id == 11).unwrap().1;
let restaurants = lines.iter().find(|(id, _)| *id == 12).unwrap().1;
// Groceries spent 14,500 against 10,000 — overspent, nothing carried.
assert_eq!(groceries.spent, 14_500);
assert_eq!(groceries.carried, 0);
assert_eq!(groceries.available, -4_500);
assert_eq!(groceries.state(), BudgetState::Overspent);
// Restaurants: February assigned 5,000 and spent nothing, so 5,000
// carried into March, where 3,200 went.
assert_eq!(restaurants.carried, 5_000);
assert_eq!(restaurants.spent, 3_200);
assert_eq!(restaurants.available, 5_000 + 5_000 - 3_200);
}
#[test]
fn subscriptions_are_found_by_rhythm_not_by_name() {
let mut ledger = Ledger { base_currency: EUR, ..Ledger::default() };
let mut account = Account::new("Card", AccountKind::CreditCard, EUR);
account.id = 1;
ledger.accounts.push(account);
let today = from_ymd(2024, 6, 1);
// A monthly charge at the same price: a subscription.
for month in 1..=5 {
let mut txn = Transaction::new(1, from_ymd(2024, month, 7), "Netflix", -1_399);
txn.id = 100 + month as i64;
ledger.transactions.push(txn);
}
// A supermarket, visited often at wildly different amounts: not one.
for (index, day) in [3, 9, 15, 21, 27].into_iter().enumerate() {
let mut txn = Transaction::new(
1,
from_ymd(2024, 5, day),
"Albert Heijn",
-(2_000 + index as i64 * 3_000),
);
txn.id = 200 + index as i64;
ledger.transactions.push(txn);
}
let found = detected_subscriptions(&ledger, today);
assert!(found.iter().any(|(payee, amount, recurrence, _)| {
payee == "Netflix" && *amount == -1_399 && *recurrence == Recurrence::Monthly
}));
assert!(
!found.iter().any(|(payee, _, _, _)| payee == "Albert Heijn"),
"varying amounts are not a subscription"
);
}
#[test]
fn attention_finds_what_a_person_would_want_told() {
let mut ledger = ledger();
let mut loose = Transaction::new(1, from_ymd(2024, 3, 28), "Mystery", -1_000);
loose.id = 400;
ledger.transactions.push(loose);
let mut broken = Transaction::new(1, from_ymd(2024, 3, 29), "Shop", -5_000);
broken.id = 401;
broken.splits =
vec![Split { id: 1, category: Some(11), amount: -4_000, memo: String::new() }];
ledger.transactions.push(broken);
let attention = needs_attention(&ledger);
assert!(attention.iter().any(|(id, why)| *id == 400 && *why == "no category"));
assert!(attention.iter().any(|(id, why)| *id == 401 && *why == "split does not add up"));
// The balanced transfer is not a problem.
assert!(!attention.iter().any(|(id, _)| *id == 200));
}
}

894
apps/finance/src/seed.rs Normal file
View file

@ -0,0 +1,894 @@
//! A believable financial life, generated, so the app is never empty.
//!
//! An empty finance app is unusable as a demo and hard to develop against:
//! no balances, no charts, nothing to click. So a file with no accounts
//! gets filled with two years of one household's money — salary on the
//! 25th, rent on the 1st, groceries twice a week, a card that gets paid
//! off monthly, a mortgage that amortises, subscriptions that renew, and
//! the occasional holiday.
//!
//! It is generated rather than canned because a fixed CSV goes stale: the
//! demo has to end *today* whenever today is, or every screen opens on an
//! empty current month. The generator is seeded and deterministic, so the
//! same day always produces the same file and a screenshot is reproducible.
//!
//! Everything here is ordinary ledger data written through the ordinary
//! [`crate::db`] paths. There is no "demo mode" in the app: the rows are
//! real rows, editable and deletable like any other.
use crate::date::{self, Day};
use crate::db::Db;
use crate::model::*;
use crate::money::{Currency, EUR};
/// How much history to generate. Two years covers every screen: a full
/// year-over-year comparison, twelve months of budgets, and enough of a
/// net-worth curve to have a shape.
pub const DEFAULT_YEARS: i32 = 2;
/// xorshift64*, so the demo is identical on every machine and every run.
/// `rand` is not a dependency of this tree and a ledger does not need
/// cryptographic randomness — it needs the same numbers twice.
struct Rng(u64);
impl Rng {
fn new(seed: u64) -> Rng {
Rng(seed | 1)
}
fn next(&mut self) -> u64 {
let mut x = self.0;
x ^= x >> 12;
x ^= x << 25;
x ^= x >> 27;
self.0 = x;
x.wrapping_mul(0x2545_f491_4f6c_dd1d)
}
/// Inclusive range.
fn between(&mut self, low: i64, high: i64) -> i64 {
if high <= low {
return low;
}
low + (self.next() % (high - low + 1) as u64) as i64
}
/// True with probability `percent`.
fn chance(&mut self, percent: u64) -> bool {
self.next() % 100 < percent
}
fn pick<'a, T>(&mut self, items: &'a [T]) -> &'a T {
&items[(self.next() % items.len() as u64) as usize]
}
}
/// The ids the generator needs to refer back to while it works.
struct Cats {
salary: Id,
interest: Id,
housing: Id,
utilities: Id,
internet: Id,
phone: Id,
groceries: Id,
restaurants: Id,
coffee: Id,
household: Id,
fuel: Id,
transit: Id,
car: Id,
clothing: Id,
electronics: Id,
pharmacy: Id,
gym: Id,
streaming: Id,
events: Id,
flights: Id,
hotels: Id,
fees: Id,
gifts: Id,
childcare: Id,
insurance: Id,
}
/// Fill an empty file with a generated household. Returns a one-line
/// summary for the status bar.
pub fn populate(db: &mut Db, years: i32) -> Result<String, String> {
let today = date::today();
let start = date::month_start(date::add_months(today, -(years * 12 - 1)));
let currency = EUR;
let accounts = insert_accounts(db, currency, start)?;
let cats = insert_categories(db)?;
let mut rng = Rng::new(0x5EED_F1_A2_C3);
let mut txns: Vec<Transaction> = Vec::new();
let mut transfer_group = 1i64;
monthly_income(&mut txns, &accounts, &cats, start, today);
housing(&mut txns, &accounts, &cats, start, today, &mut rng);
subscriptions(&mut txns, &accounts, &cats, start, today);
daily_life(&mut txns, &accounts, &cats, start, today, &mut rng);
occasional(&mut txns, &accounts, &cats, start, today, &mut rng);
card_payments(&mut txns, &accounts, start, today, &mut transfer_group);
savings_transfers(&mut txns, &accounts, start, today, &mut transfer_group);
savings_interest(&mut txns, &accounts, &cats, start, today);
mortgage(&mut txns, &accounts, &cats, start, today, &mut transfer_group);
// Age determines what the bank has seen: anything older than a few
// days has cleared, the oldest year has been reconciled, and the last
// few days are still in flight. That gives the reconcile screen and
// the "cleared vs current" balances something true to show.
for txn in txns.iter_mut() {
let age = today - txn.date;
txn.cleared = if age > 365 {
Cleared::Reconciled
} else if age > 4 {
Cleared::Cleared
} else {
Cleared::Uncleared
};
}
txns.sort_by_key(|t| t.date);
let count = txns.len();
db.transact(|conn| {
for txn in &txns {
crate::db::insert_transaction_on(conn, txn)?;
}
Ok(())
})?;
// Splits need the ids the insert assigned, so they go in a second pass
// over the file rather than being carried along.
add_splits(db, &cats)?;
insert_budgets(db, &cats, today, years)?;
insert_rules(db, &cats)?;
insert_scheduled(db, &accounts, &cats, today)?;
db.set_setting("base_currency", currency.code)?;
Ok(format!(
"{count} transactions across {} accounts, {} to {}",
accounts.all().len(),
date::format_short(start),
date::format_short(today)
))
}
struct Accounts {
checking: Id,
savings: Id,
card: Id,
brokerage: Id,
mortgage: Id,
house: Id,
cash: Id,
}
impl Accounts {
fn all(&self) -> [Id; 7] {
[
self.checking,
self.savings,
self.card,
self.brokerage,
self.mortgage,
self.house,
self.cash,
]
}
}
fn insert_accounts(db: &mut Db, currency: Currency, start: Day) -> Result<Accounts, String> {
let mut make = |name: &str,
kind: AccountKind,
institution: &str,
opening: i64,
order: i32|
-> Result<Id, String> {
let mut account = Account::new(name, kind, currency);
account.institution = institution.to_string();
account.opening_balance = opening;
account.opening_date = start - 1;
account.sort_order = order;
db.insert_account(&account)
};
Ok(Accounts {
checking: make("Everyday", AccountKind::Checking, "ING", 342_150, 0)?,
savings: make("Savings", AccountKind::Savings, "ING", 1_480_000, 1)?,
card: make("Rewards Card", AccountKind::CreditCard, "Amex", -84_320, 2)?,
cash: make("Cash", AccountKind::Cash, "", 12_000, 3)?,
brokerage: make("Brokerage", AccountKind::Investment, "DEGIRO", 2_650_000, 4)?,
// A mortgage is a debt: negative, and paid down over the run.
mortgage: make("Mortgage", AccountKind::Loan, "Rabobank", -24_800_000, 5)?,
house: make("Apartment", AccountKind::Asset, "", 41_500_000, 6)?,
})
}
fn insert_categories(db: &mut Db) -> Result<Cats, String> {
let mut group = |name: &str, kind: CategoryKind, order: i32| -> Result<Id, String> {
let mut category = Category::group(name, kind);
category.sort_order = order;
db.insert_category(&category)
};
let income = group("Income", CategoryKind::Income, 0)?;
let housing = group("Housing", CategoryKind::Expense, 1)?;
let food = group("Food", CategoryKind::Expense, 2)?;
let transport = group("Transport", CategoryKind::Expense, 3)?;
let shopping = group("Shopping", CategoryKind::Expense, 4)?;
let health = group("Health", CategoryKind::Expense, 5)?;
let fun = group("Fun", CategoryKind::Expense, 6)?;
let travel = group("Travel", CategoryKind::Expense, 7)?;
let money = group("Money", CategoryKind::Expense, 8)?;
let family = group("Family", CategoryKind::Expense, 9)?;
let mut child = |name: &str,
parent: Id,
kind: CategoryKind,
rollover: bool,
order: i32|
-> Result<Id, String> {
let mut category = Category::child(name, parent, kind);
category.budgeted = kind == CategoryKind::Expense;
category.rollover = rollover;
category.sort_order = order;
db.insert_category(&category)
};
Ok(Cats {
salary: child("Salary", income, CategoryKind::Income, false, 0)?,
interest: child("Interest", income, CategoryKind::Income, false, 1)?,
housing: child("Mortgage", housing, CategoryKind::Expense, false, 0)?,
utilities: child("Energy", housing, CategoryKind::Expense, false, 1)?,
internet: child("Internet", housing, CategoryKind::Expense, false, 2)?,
phone: child("Phone", housing, CategoryKind::Expense, false, 3)?,
groceries: child("Groceries", food, CategoryKind::Expense, false, 0)?,
restaurants: child("Restaurants", food, CategoryKind::Expense, false, 1)?,
coffee: child("Coffee", food, CategoryKind::Expense, false, 2)?,
household: child("Household", shopping, CategoryKind::Expense, false, 0)?,
fuel: child("Fuel", transport, CategoryKind::Expense, false, 0)?,
transit: child("Transit", transport, CategoryKind::Expense, false, 1)?,
// Car maintenance is lumpy, so it rolls over: three quiet months
// pay for the fourth.
car: child("Car upkeep", transport, CategoryKind::Expense, true, 2)?,
clothing: child("Clothing", shopping, CategoryKind::Expense, false, 1)?,
electronics: child("Electronics", shopping, CategoryKind::Expense, true, 2)?,
pharmacy: child("Pharmacy", health, CategoryKind::Expense, false, 0)?,
gym: child("Gym", health, CategoryKind::Expense, false, 1)?,
streaming: child("Streaming", fun, CategoryKind::Expense, false, 0)?,
events: child("Going out", fun, CategoryKind::Expense, false, 1)?,
flights: child("Flights", travel, CategoryKind::Expense, true, 0)?,
hotels: child("Hotels", travel, CategoryKind::Expense, true, 1)?,
fees: child("Bank fees", money, CategoryKind::Expense, false, 0)?,
insurance: child("Insurance", money, CategoryKind::Expense, false, 1)?,
gifts: child("Gifts", family, CategoryKind::Expense, true, 0)?,
childcare: child("Childcare", family, CategoryKind::Expense, false, 1)?,
})
}
/// A payday that lands on a working day: paid on the 25th, moved back to
/// the Friday when that is a weekend, which is what employers do.
fn payday(month_start: Day) -> Day {
let (y, m, _) = date::to_ymd(month_start);
let mut day = date::from_ymd(y, m, 25);
while date::is_weekend(day) {
day -= 1;
}
day
}
fn each_month(start: Day, end: Day, mut body: impl FnMut(Day)) {
let mut month = date::month_start(start);
while month <= end {
body(month);
month = date::add_months(month, 1);
}
}
fn monthly_income(txns: &mut Vec<Transaction>, accounts: &Accounts, cats: &Cats, start: Day, end: Day) {
// A raise a third of the way in, so year-over-year has something to
// show and the budget screen has a reason to change.
let raise_at = date::add_months(start, 14);
each_month(start, end, |month| {
let day = payday(month);
if day > end || day < start {
return;
}
let amount = if day >= raise_at { 492_400 } else { 465_000 };
let mut txn = Transaction::new(accounts.checking, day, "Bergman Design BV", amount);
txn.category = Some(cats.salary);
txn.memo = "Salary".into();
txns.push(txn);
});
}
fn housing(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
cats: &Cats,
start: Day,
end: Day,
rng: &mut Rng,
) {
each_month(start, end, |month| {
let (y, m, _) = date::to_ymd(month);
let mut push = |day: u32, payee: &str, amount: i64, category: Id| {
let date = date::from_ymd(y, m, day.min(date::days_in_month(y, m)));
if date > end || date < start {
return;
}
let mut txn = Transaction::new(accounts.checking, date, payee, -amount);
txn.category = Some(category);
txns.push(txn);
};
// Energy swings with the season: a Dutch winter costs roughly
// double a summer month.
let winter = matches!(m, 11 | 12 | 1 | 2 | 3);
let energy = if winter { rng.between(18_500, 24_000) } else { rng.between(8_500, 12_500) };
push(3, "Eneco", energy, cats.utilities);
push(5, "KPN Internet", 5_450, cats.internet);
push(8, "Vodafone", 2_890, cats.phone);
push(12, "Centraal Beheer", 8_640, cats.insurance);
push(2, "Kinderopvang Zonnetje", 54_000, cats.childcare);
});
}
fn subscriptions(txns: &mut Vec<Transaction>, accounts: &Accounts, cats: &Cats, start: Day, end: Day) {
// Charged to the card, like most subscriptions are.
let monthly: [(u32, &str, i64, fn(&Cats) -> Id); 5] = [
(4, "Netflix", 1_399, |c| c.streaming),
(7, "Spotify", 1_099, |c| c.streaming),
(15, "Apple iCloud", 299, |c| c.streaming),
(18, "SportCity", 2_995, |c| c.gym),
(22, "Adobe", 2_399, |c| c.electronics),
];
each_month(start, end, |month| {
let (y, m, _) = date::to_ymd(month);
for (day, payee, amount, category) in monthly {
let date = date::from_ymd(y, m, day);
if date > end || date < start {
continue;
}
let mut txn = Transaction::new(accounts.card, date, payee, -amount);
txn.category = Some(category(cats));
txns.push(txn);
}
});
}
fn daily_life(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
cats: &Cats,
start: Day,
end: Day,
rng: &mut Rng,
) {
const SUPERMARKETS: [&str; 5] = ["Albert Heijn", "Jumbo", "Lidl", "Dirk", "Ekoplaza"];
const CAFES: [&str; 5] =
["Coffee Company", "Bocca Koffie", "Lot Sixty One", "Toki", "Screaming Beans"];
const RESTAURANTS: [&str; 6] = [
"Café de Klos",
"Bar Bukowski",
"Thai Bird",
"De Biertuin",
"Pizzeria Sugo",
"Sushi Ran",
];
const SHOPS: [&str; 5] = ["HEMA", "Bol.com", "Zara", "Decathlon", "MediaMarkt"];
let mut day = start;
while day <= end {
let weekday = date::weekday(day);
// Groceries: a big weekend shop and one or two top-ups.
if weekday == 5 || (rng.chance(35) && weekday != 6) {
let big = weekday == 5;
let amount = if big { rng.between(5_800, 12_400) } else { rng.between(1_200, 4_500) };
let mut txn = Transaction::new(
if rng.chance(15) { accounts.cash } else { accounts.card },
day,
rng.pick(&SUPERMARKETS),
-amount,
);
txn.category = Some(cats.groceries);
txns.push(txn);
}
// Coffee on working days.
if weekday < 5 && rng.chance(55) {
let mut txn =
Transaction::new(accounts.card, day, rng.pick(&CAFES), -rng.between(280, 720));
txn.category = Some(cats.coffee);
txns.push(txn);
}
// Eating out, mostly at the weekend.
let eats_out = if weekday >= 4 { rng.chance(45) } else { rng.chance(12) };
if eats_out {
let mut txn = Transaction::new(
accounts.card,
day,
rng.pick(&RESTAURANTS),
-rng.between(2_200, 8_900),
);
txn.category = Some(cats.restaurants);
txns.push(txn);
}
// Transit and fuel.
if weekday < 5 && rng.chance(30) {
let mut txn = Transaction::new(accounts.card, day, "NS Reizigers", -rng.between(320, 2_450));
txn.category = Some(cats.transit);
txns.push(txn);
}
if rng.chance(6) {
let mut txn = Transaction::new(accounts.card, day, "Shell", -rng.between(4_500, 8_800));
txn.category = Some(cats.fuel);
txns.push(txn);
}
// Odds and ends.
if rng.chance(9) {
let shop = rng.pick(&SHOPS);
let (category, amount) = match *shop {
"MediaMarkt" => (cats.electronics, rng.between(3_900, 45_000)),
"Zara" | "Decathlon" => (cats.clothing, rng.between(2_500, 14_000)),
_ => (cats.household, rng.between(800, 6_500)),
};
let mut txn = Transaction::new(accounts.card, day, shop, -amount);
txn.category = Some(category);
txns.push(txn);
}
if rng.chance(4) {
let mut txn = Transaction::new(accounts.card, day, "Etos", -rng.between(600, 3_400));
txn.category = Some(cats.pharmacy);
txns.push(txn);
}
if rng.chance(5) {
let mut txn =
Transaction::new(accounts.cash, day, "Albert Cuyp Markt", -rng.between(500, 2_500));
txn.category = Some(cats.groceries);
txns.push(txn);
}
day += 1;
}
}
fn occasional(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
cats: &Cats,
start: Day,
end: Day,
rng: &mut Rng,
) {
// One holiday a year, in the summer, plus a winter weekend away.
let mut year = date::year_of(start);
while year <= date::year_of(end) {
for (month, day, flight, hotel, place) in [
(7, 12, 118_000i64, 96_500i64, "Lisbon"),
(2, 8, 42_000, 38_000, "Vienna"),
] {
let date = date::from_ymd(year, month, day);
if date < start || date > end {
continue;
}
let mut air = Transaction::new(accounts.card, date, "KLM", -flight);
air.category = Some(cats.flights);
air.memo = format!("{place} trip");
txns.push(air);
let mut stay = Transaction::new(accounts.card, date + 1, "Booking.com", -hotel);
stay.category = Some(cats.hotels);
stay.memo = format!("{place} trip");
txns.push(stay);
// Spending abroad, on the card.
for offset in 2..7 {
if date + offset > end {
break;
}
let mut meal = Transaction::new(
accounts.card,
date + offset,
"Restaurante Ramiro",
-rng.between(2_800, 9_500),
);
meal.category = Some(cats.restaurants);
txns.push(meal);
}
}
// Car maintenance, once or twice a year — the lumpy category the
// rollover exists for.
let service = date::from_ymd(year, 5, 14);
if service >= start && service <= end {
let mut txn = Transaction::new(accounts.checking, service, "Garage Van Dijk", -rng.between(28_000, 62_000));
txn.category = Some(cats.car);
txns.push(txn);
}
// Birthdays and December.
for (month, day, payee) in [(12, 18, "Bol.com"), (6, 4, "Bloemenwinkel")] {
let date = date::from_ymd(year, month, day);
if date >= start && date <= end {
let mut txn =
Transaction::new(accounts.card, date, payee, -rng.between(4_500, 22_000));
txn.category = Some(cats.gifts);
txn.memo = "Gift".into();
txns.push(txn);
}
}
// A concert or two.
for (month, day) in [(9, 21), (3, 15)] {
let date = date::from_ymd(year, month, day);
if date >= start && date <= end && rng.chance(70) {
let mut txn =
Transaction::new(accounts.card, date, "Paradiso", -rng.between(3_500, 9_000));
txn.category = Some(cats.events);
txns.push(txn);
}
}
year += 1;
}
}
/// The card is paid off in full each month, from checking — a transfer
/// pair, which is what gives the transfer screens something real.
fn card_payments(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
start: Day,
end: Day,
group: &mut i64,
) {
each_month(start, end, |month| {
let (y, m, _) = date::to_ymd(month);
let date = date::from_ymd(y, m, 28.min(date::days_in_month(y, m)));
if date > end || date < start {
return;
}
// What the card ran up in the previous month, near enough.
let previous = date::add_months(date, -1);
let spent: i64 = txns
.iter()
.filter(|t| {
t.account == accounts.card
&& date::month_key(t.date) == date::month_key(previous)
})
.map(|t| t.amount)
.sum();
let amount = -spent;
if amount <= 0 {
return;
}
*group += 1;
let mut out = Transaction::new(accounts.checking, date, "Amex", -amount);
out.transfer_group = Some(*group);
out.memo = "Card payment".into();
let mut into = Transaction::new(accounts.card, date, "Payment received", amount);
into.transfer_group = Some(*group);
into.memo = "Card payment".into();
txns.push(out);
txns.push(into);
});
}
fn savings_transfers(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
start: Day,
end: Day,
group: &mut i64,
) {
each_month(start, end, |month| {
let date = payday(month) + 1;
if date > end || date < start {
return;
}
*group += 1;
let amount = 40_000;
let mut out = Transaction::new(accounts.checking, date, "Savings", -amount);
out.transfer_group = Some(*group);
out.memo = "Monthly saving".into();
let mut into = Transaction::new(accounts.savings, date, "From Everyday", amount);
into.transfer_group = Some(*group);
into.memo = "Monthly saving".into();
txns.push(out);
txns.push(into);
});
}
fn savings_interest(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
cats: &Cats,
start: Day,
end: Day,
) {
let mut balance = 1_480_000i64;
each_month(start, end, |month| {
let (y, m, _) = date::to_ymd(month);
let date = date::from_ymd(y, m, date::days_in_month(y, m));
if date > end || date < start {
return;
}
balance += 40_000;
// 1.8% a year, paid monthly.
let interest = balance * 18 / 1000 / 12;
let mut txn = Transaction::new(accounts.savings, date, "ING", interest);
txn.category = Some(cats.interest);
txn.memo = "Interest".into();
txns.push(txn);
});
}
/// A mortgage payment is two things at once: interest (an expense) and
/// principal (a transfer that shrinks the debt). Modelling it as a split
/// would hide the debt movement, so it is a transfer pair for the
/// principal and a plain expense for the interest — which is how the loan
/// balance ends up actually going down on the net-worth chart.
fn mortgage(
txns: &mut Vec<Transaction>,
accounts: &Accounts,
cats: &Cats,
start: Day,
end: Day,
group: &mut i64,
) {
let mut owed = 24_800_000i64;
each_month(start, end, |month| {
let (y, m, _) = date::to_ymd(month);
let date = date::from_ymd(y, m, 2);
if date > end || date < start {
return;
}
// 3.4% a year on the outstanding balance.
let interest = owed * 34 / 1000 / 12;
let principal = 92_400 - interest.min(92_400);
let mut cost = Transaction::new(accounts.checking, date, "Rabobank", -interest);
cost.category = Some(cats.housing);
cost.memo = "Mortgage interest".into();
txns.push(cost);
if principal > 0 {
*group += 1;
let mut out = Transaction::new(accounts.checking, date, "Rabobank", -principal);
out.transfer_group = Some(*group);
out.memo = "Mortgage principal".into();
let mut down = Transaction::new(accounts.mortgage, date, "Payment", principal);
down.transfer_group = Some(*group);
down.memo = "Mortgage principal".into();
txns.push(out);
txns.push(down);
owed -= principal;
}
});
}
/// Turn a handful of supermarket trips into split transactions, so the
/// split UI has real examples the moment the app opens.
fn add_splits(db: &mut Db, cats: &Cats) -> Result<(), String> {
let ledger = db.load()?;
let candidates: Vec<Transaction> = ledger
.transactions
.iter()
.filter(|t| {
t.category == Some(cats.groceries) && t.amount < -7_000 && t.splits.is_empty()
})
.take(6)
.cloned()
.collect();
for mut txn in candidates {
// A third of a big shop was household goods, not food.
let household = txn.amount / 3;
let food = txn.amount - household;
txn.splits = vec![
Split { id: 0, category: Some(cats.groceries), amount: food, memo: "Food".into() },
Split {
id: 0,
category: Some(cats.household),
amount: household,
memo: "Cleaning, paper".into(),
},
];
debug_assert_eq!(txn.split_imbalance(), 0);
db.update_transaction(&txn)?;
}
Ok(())
}
/// Budgets for every month of history, so the budget screen opens on real
/// numbers and the "assigned vs spent" bars mean something.
fn insert_budgets(db: &mut Db, cats: &Cats, today: Day, years: i32) -> Result<(), String> {
let plan: [(Id, i64); 17] = [
(cats.housing, 92_400),
(cats.utilities, 14_000),
(cats.internet, 5_450),
(cats.phone, 2_890),
(cats.childcare, 54_000),
(cats.insurance, 8_640),
(cats.groceries, 52_000),
(cats.restaurants, 18_000),
(cats.coffee, 6_000),
(cats.household, 8_000),
(cats.transit, 9_000),
(cats.fuel, 12_000),
(cats.car, 15_000),
(cats.clothing, 10_000),
(cats.streaming, 5_200),
(cats.gym, 2_995),
(cats.events, 8_000),
];
let months = years * 12;
let first = date::month_key(date::add_months(today, -(months - 1)));
db.transact(|_conn| Ok(()))?;
for offset in 0..months {
let month = first + offset;
for (category, assigned) in plan {
db.insert_budget(&BudgetEntry {
category,
month,
assigned,
rollover: matches!(category, c if c == cats.car),
})?;
}
}
Ok(())
}
/// The rules a person would have written after a month of imports.
fn insert_rules(db: &mut Db, cats: &Cats) -> Result<(), String> {
let rules = [
("Albert Heijn", "AH TO GO", Some(cats.groceries), Some("Albert Heijn")),
("Shell", "SHELL NEDERLAND", Some(cats.fuel), Some("Shell")),
("NS", "NS GROEP", Some(cats.transit), Some("NS Reizigers")),
("Netflix", "NETFLIX.COM", Some(cats.streaming), Some("Netflix")),
("Amazon", "AMZN MKTP", Some(cats.household), Some("Amazon")),
];
for (index, (name, pattern, category, rename)) in rules.into_iter().enumerate() {
db.insert_rule(&Rule {
id: 0,
name: name.to_string(),
match_on: MatchOn::Raw,
how: MatchHow::Contains,
pattern: pattern.to_string(),
amount_min: 0,
amount_max: 0,
set_category: category,
rename_payee: rename.map(str::to_string),
set_memo: None,
flag: false,
priority: index as i32,
enabled: true,
hits: 0,
})?;
}
Ok(())
}
/// The recurring bills, as the app's detector would have found them.
fn insert_scheduled(db: &mut Db, accounts: &Accounts, cats: &Cats, today: Day) -> Result<(), String> {
let next = |day: u32| -> Day {
let (y, m, _) = date::to_ymd(today);
let candidate = date::from_ymd(y, m, day.min(date::days_in_month(y, m)));
if candidate >= today {
candidate
} else {
date::add_months(candidate, 1)
}
};
let items = [
(accounts.checking, "Rabobank hypotheek", -92_400, cats.housing, next(2)),
(accounts.checking, "Kinderopvang Zonnetje", -54_000, cats.childcare, next(2)),
(accounts.checking, "Eneco", -14_000, cats.utilities, next(3)),
(accounts.checking, "KPN Internet", -5_450, cats.internet, next(5)),
(accounts.checking, "Vodafone", -2_890, cats.phone, next(8)),
(accounts.card, "Netflix", -1_399, cats.streaming, next(4)),
(accounts.card, "Spotify", -1_099, cats.streaming, next(7)),
(accounts.card, "SportCity", -2_995, cats.gym, next(18)),
(accounts.checking, "Bergman Design BV", 492_400, cats.salary, next(25)),
];
for (account, payee, amount, category, due) in items {
db.insert_scheduled(&Scheduled {
id: 0,
account,
payee: payee.to_string(),
amount,
category: Some(category),
recurrence: Recurrence::Monthly,
next_due: due,
last_posted: None,
auto_post: false,
enabled: true,
detected: true,
})?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn temp_db(name: &str) -> (Db, std::path::PathBuf) {
let mut path = std::env::temp_dir();
path.push(format!("finance-seed-{name}-{}.db", std::process::id()));
let _ = std::fs::remove_file(&path);
(Db::open(&path).expect("open"), path)
}
#[test]
fn the_demo_file_is_a_coherent_household() {
let (mut db, path) = temp_db("household");
let summary = populate(&mut db, DEFAULT_YEARS).expect("populate");
assert!(summary.contains("transactions"));
let ledger = db.load().expect("load");
// Enough to fill every screen.
assert!(
ledger.transactions.len() > 1_500,
"two years should be thousands of rows, got {}",
ledger.transactions.len()
);
assert_eq!(ledger.accounts.len(), 7);
assert!(ledger.categories.categories.len() > 25);
assert!(!ledger.budgets.is_empty());
assert!(!ledger.rules.is_empty());
assert!(!ledger.scheduled.is_empty());
// Every transfer pair balances — the invariant the whole
// net-worth number rests on.
let groups: std::collections::HashSet<Id> =
ledger.transactions.iter().filter_map(|t| t.transfer_group).collect();
assert!(groups.len() > 20, "expected many transfers, got {}", groups.len());
for group in groups {
assert!(ledger.transfer_is_balanced(group), "transfer {group} does not cancel");
}
// Splits sum to their transaction.
let split_count = ledger.transactions.iter().filter(|t| t.is_split()).count();
assert!(split_count >= 5, "expected split examples, got {split_count}");
for txn in ledger.transactions.iter().filter(|t| t.is_split()) {
assert_eq!(txn.split_imbalance(), 0);
}
// The story adds up: income arrives, the current account stays
// solvent, and the mortgage is smaller than it started.
let checking = ledger.accounts.iter().find(|a| a.name == "Everyday").unwrap();
assert!(ledger.balance(checking.id) > 0, "the household should not be overdrawn");
let mortgage = ledger.accounts.iter().find(|a| a.name == "Mortgage").unwrap();
assert!(
ledger.balance(mortgage.id) > mortgage.opening_balance,
"the mortgage should have been paid down"
);
assert!(ledger.net_worth_on(date::today()) > 0);
// Nothing in the future, and history reaches back two years.
let today = date::today();
assert!(ledger.transactions.iter().all(|t| t.date <= today));
let oldest = ledger.transactions.iter().map(|t| t.date).min().unwrap();
assert!(today - oldest > 660, "expected ~2 years of history");
// The recent tail is still uncleared, the deep past is reconciled.
assert!(ledger
.transactions
.iter()
.any(|t| t.cleared == Cleared::Uncleared));
assert!(ledger
.transactions
.iter()
.any(|t| t.cleared == Cleared::Reconciled));
assert!(ledger.cleared_balance(checking.id) != ledger.balance(checking.id));
let _ = std::fs::remove_file(&path);
}
#[test]
fn the_same_seed_produces_the_same_file() {
let (mut a, path_a) = temp_db("determinism-a");
let (mut b, path_b) = temp_db("determinism-b");
populate(&mut a, 1).expect("a");
populate(&mut b, 1).expect("b");
let one = a.load().expect("load a");
let two = b.load().expect("load b");
assert_eq!(one.transactions.len(), two.transactions.len());
let sum_a: i64 = one.transactions.iter().map(|t| t.amount).sum();
let sum_b: i64 = two.transactions.iter().map(|t| t.amount).sum();
assert_eq!(sum_a, sum_b);
let _ = std::fs::remove_file(&path_a);
let _ = std::fs::remove_file(&path_b);
}
}

112
apps/finance/src/theme.rs Normal file
View file

@ -0,0 +1,112 @@
//! The palette, in one place.
//!
//! Published into `mod.finance.*` so the DSL reads `mod.finance.accent`
//! rather than a hex literal repeated forty times — change a colour here
//! and every screen moves together.
//!
//! It is a dark palette because a ledger is a wall of numbers, and dark
//! rows with one bright accent let the numbers carry the contrast instead
//! of fighting the background for it.
//!
//! Three rules, taken from what the best-looking money apps actually do:
//!
//! * **A cool near-black, never `#000`, and elevation by tone.** Three
//! surfaces each a few percent lighter than the last, separated by
//! hairlines rather than shadows. Drop shadows on cards are the single
//! clearest "designed in 2018" tell.
//! * **One saturated accent.** Indigo, and nothing else competes with it.
//! Restricting the palette is what reads as expensive; a different bright
//! colour per spending category reads as a 2015 budgeting app.
//! * **Money colour is reserved and redundant.** Good/critical are status,
//! never a chart series, and they never carry meaning alone — the sign is
//! always there too, because roughly one man in twelve cannot tell the
//! two hues apart.
use makepad_widgets::*;
pub fn install(vm: &mut ScriptVm) {
script_eval!(vm, {
mod.finance = {
// Surfaces, darkest to lightest: the page, a card, and a
// control on that card. Each step is a few percent lighter,
// which is the whole elevation system — there are no shadows.
bg: #x0c0d12,
panel: #x14161d,
raised: #x1b1e27,
line: #x272a35,
line_soft: #x1e212a,
// Text.
fg: #xf2f4f8,
fg_dim: #xa2a8b8,
fg_faint: #x6f7585,
// One accent, used for selection, the active tab and the
// primary action. Anything else that wants attention has to
// earn it with weight or size instead.
accent: #x5e6ad2,
accent_soft: #x272a52,
// Money. Nothing else may use these two.
up: #x3fb950,
down: #xf85149,
// Chart series, in fixed order, never cycled. These are the
// dataviz reference palette's dark steps: the set passes the
// colour-blindness separation and contrast checks as a whole,
// which a hand-picked set of "nice" hues does not — the blue
// and violet I first chose were 2.4 ΔE apart to a deuteranope,
// which is to say identical.
c0: #x3987e5,
c1: #xd95926,
c2: #x199e70,
c3: #xc98500,
c4: #xd55181,
c5: #x008300,
c6: #x9085e9,
c7: #xe66767,
// Status, reserved: these four never stand in for a series.
good: #x0ca30c,
warning: #xfab219,
serious: #xec835a,
critical: #xd03b3b,
// The warm tint behind a row that needs attention.
warn: #x3a2d16,
// Register surfaces: the alternate row is a hair lighter than
// the page, never a different colour.
zebra: #x101219,
select: #x5e6ad233,
}
});
}
/// Chart colours by index, for series the Rust side hands out.
pub const SERIES: [u32; 8] = [
0x3987e5, 0xd95926, 0x199e70, 0xc98500, 0xd55181, 0x008300, 0x9085e9, 0xe66767,
];
/// Status colours, reserved. Money in and money out are STATUS, not series
/// — which is why they are never drawn from [`SERIES`].
pub const GOOD: u32 = 0x0ca30c;
pub const CRITICAL: u32 = 0xd03b3b;
pub const WARNING: u32 = 0xfab219;
/// A category's colour: its own if it has one, else one picked from the
/// series by id so it stays the same colour on every screen and across
/// runs.
pub fn category_color(id: i64, stored: u32) -> Vec4f {
let rgb = if stored != 0 { stored } else { SERIES[(id.unsigned_abs() as usize) % SERIES.len()] };
Vec4f {
x: ((rgb >> 16) & 0xff) as f32 / 255.0,
y: ((rgb >> 8) & 0xff) as f32 / 255.0,
z: (rgb & 0xff) as f32 / 255.0,
w: 1.0,
}
}
pub fn rgb(value: u32) -> Vec4f {
category_color(0, value)
}

1658
apps/finance/src/view.rs Normal file

File diff suppressed because it is too large Load diff

View file

@ -21,3 +21,4 @@ default-run = "makepad-mixer"
[dependencies]
makepad-widgets = { path = "../../widgets" }
mp-theme = { path = "../../libs/mp_theme" }

View file

@ -1,17 +1,18 @@
// COMPACT surface layout (splash) — the proof that the layout seam is real.
//
// Same slot contract as lr_mix.splash (see its header), different surface:
// name plate on top, big fader + meter, dB readout, mute. No EQ/dyn
// thumbnails, no gain/threshold rows, no pan. The host binds whatever slots
// and children a layout chooses to show — everything else simply isn't
// displayed, and nothing a layout writes can name an OSC address.
// Same slot contract and same injected `mp` palette as lr_mix.splash (see its
// header), different surface: name plate on top, big fader + meter, dB
// readout, mute. No EQ/dyn thumbnails, no gain/threshold rows, no pan. The
// host binds whatever slots and children a layout chooses to show —
// everything else simply isn't displayed, and nothing a layout writes can
// name an OSC address.
let ValueLabel = Label{
width: Fill
align: Align{x: 0.5}
text: "—"
draw_text.color: #xe6ecf4
draw_text.text_style.font_size: 10.0
draw_text.color: mp.fg_bright
draw_text.text_style: theme.font_code{font_size: 10.0}
}
let FaderSlider = Slider{
@ -24,33 +25,35 @@ let FaderSlider = Slider{
text: ""
text_input: TextInput{width: 0, height: 0}
draw_bg +: {
body: uniform(#x0b0f15)
slot: uniform(#x171e29)
cap: uniform(#x9aa3ad)
cap_dark: uniform(#x4a5158)
body: uniform(mp.bg_dark)
slot: uniform(mp.muted)
tick: uniform(mp.muted)
cap: uniform(mp.fg_dim)
cap_edge: uniform(mp.accent)
cap_line: uniform(mp.fg_bright)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
let h = self.rect_size.y
let w = self.rect_size.x
sdf.box(0.0, 0.0, w, h, 3.0)
sdf.rect(0.0, 0.0, w, h)
sdf.fill(self.body)
let rel = self.pos.y
let grid = abs(fract(rel * 8.0 + 0.5) - 0.5) * (h / 8.0)
let tick = 1.0 - smoothstep(0.4, 0.9, grid)
let mark = 1.0 - smoothstep(0.4, 0.9, grid)
let px = self.pos.x * w
if px < 7.0 {
sdf.box(0.0, rel * h - 0.4, 6.0, 0.8, 0.2)
sdf.fill(vec4(0.55, 0.62, 0.7, 1.0) * tick)
sdf.rect(0.0, rel * h - 0.5, 6.0, 1.0)
sdf.fill(self.tick * mark)
}
sdf.box(w * 0.5 - 2.0, 3.0, 4.0, h - 6.0, 2.0)
sdf.rect(w * 0.5 - 1.0, 3.0, 2.0, h - 6.0)
sdf.fill(self.slot)
let cap_h = 26.0
let cy = (1.0 - self.slide_pos) * (h - cap_h)
sdf.box(4.0, cy, w - 8.0, cap_h, 3.0)
let rib = 0.75 + 0.25 * smoothstep(0.25, 0.5, abs(fract((self.pos.y * h - cy) / 4.8) - 0.5))
sdf.fill(self.cap.mix(self.cap_dark, 1.0 - rib))
sdf.box(4.0, cy + cap_h * 0.5 - 1.2, w - 8.0, 2.4, 0.6)
sdf.fill(#xf4f7fb)
sdf.rect(4.5, cy + 0.5, w - 9.0, cap_h - 1.0)
sdf.fill_keep(self.cap)
sdf.stroke(self.cap_edge, 1.0)
sdf.rect(4.0, cy + cap_h * 0.5 - 1.2, w - 8.0, 2.4)
sdf.fill(self.cap_line)
return sdf.result
}
}
@ -60,6 +63,12 @@ let MeterBar = SolidView{
width: 14
height: Fill
draw_bg +: {
gutter: uniform(mp.bg)
trough: uniform(mp.bg_dark)
low: uniform(mp.green)
mid: uniform(mp.yellow)
hot: uniform(mp.red)
tick: uniform(mp.fg_bright)
level_db: instance(-90.0)
peak_db: instance(-90.0)
pixel: fn() {
@ -68,24 +77,21 @@ let MeterBar = SolidView{
let norm = clamp((self.level_db + 60.0) / 60.0, 0.0, 1.0)
let peakn = clamp((self.peak_db + 60.0) / 60.0, 0.0, 1.0)
let frac = 1.0 - self.pos.y
var col = vec3(0.03, 0.09, 0.045)
let seg = 0.85 + 0.15 * smoothstep(0.35, 0.5, abs(fract(frac * 30.0) - 0.5))
let ladder = self.low.mix(self.mid, smoothstep(0.70, 0.88, frac))
.mix(self.hot, smoothstep(0.88, 0.97, frac))
let seg = 0.88 + 0.12 * smoothstep(0.35, 0.5, abs(fract(frac * 30.0) - 0.5))
var col = self.trough.mix(self.hot, 0.16 * step(0.955, frac))
if frac < norm {
let hot = smoothstep(0.82, 0.93, frac)
col = vec3(0.10, 0.78, 0.28).mix(vec3(1.0, 0.22, 0.16), hot) * seg
}
if frac > 0.955 {
col = col.mix(vec3(0.45, 0.09, 0.07), 0.8)
if frac < norm { col = vec3(1.0, 0.25, 0.18) }
col = self.trough.mix(ladder, seg)
}
if abs(frac - peakn) < 1.5 / h && peakn > 0.01 {
col = vec3(0.9, 1.0, 0.9)
col = self.tick
}
let x = self.pos.x * w
if x < 1.0 || x > w - 1.0 {
col = vec3(0.015, 0.02, 0.03)
col = self.gutter
}
return vec4(col, 1.0)
return col
}
}
}
@ -97,20 +103,26 @@ let MutePlate = SolidView{
new_batch: true
align: Align{x: 0.5, y: 0.5}
draw_bg +: {
idle: uniform(mp.bg_light)
idle_edge: uniform(mp.muted)
lit: uniform(mp.red)
deep: uniform(mp.bg_dark)
muted: instance(0.0)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.box(1.0, 1.0, self.rect_size.x - 2.0, self.rect_size.y - 2.0, 4.0)
let idle = vec4(0.23, 0.07, 0.06, 1.0)
let lit = vec4(0.82, 0.13, 0.10, 1.0)
sdf.fill(idle.mix(lit, self.muted))
sdf.rect(0.5, 0.5, self.rect_size.x - 1.0, self.rect_size.y - 1.0)
let fill = self.idle.mix(self.lit.mix(self.deep, 0.5), self.muted)
// fill_keep, not fill: `fill` clears the shape, so a stroke after
// it draws nothing at all.
sdf.fill_keep(fill)
sdf.stroke(self.idle_edge.mix(self.lit, self.muted), 1.0)
return sdf.result
}
}
Label{
text: "MUTE"
draw_text.color: #xf3d9d6
draw_text.text_style.font_size: 8.5
draw_text.color: mp.fg_bright
draw_text.text_style: theme.font_code{font_size: 8.5}
}
}
@ -121,22 +133,22 @@ let NamePlate = SolidView{
new_batch: true
align: Align{x: 0.5, y: 0.5}
draw_bg +: {
plate_rgb: instance(vec3(0.5, 0.5, 0.5))
idle: uniform(mp.bg_light)
// The muted key, until the console reports its scribble colour.
plate_rgb: instance(vec3(0.254, 0.282, 0.408))
plate_filled: instance(0.0)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.box(1.0, 1.0, self.rect_size.x - 2.0, self.rect_size.y - 2.0, 4.0)
let outline = vec4(0.03, 0.04, 0.055, 1.0)
let filled = vec4(self.plate_rgb * 0.82, 1.0)
sdf.fill(outline.mix(filled, self.plate_filled))
sdf.stroke(vec4(self.plate_rgb, 1.0), 1.2)
sdf.rect(0.5, 0.5, self.rect_size.x - 1.0, self.rect_size.y - 1.0)
sdf.fill_keep(self.idle.mix(vec4(self.plate_rgb, 1.0), self.plate_filled))
sdf.stroke(vec4(self.plate_rgb, 1.0), 1.0)
return sdf.result
}
}
name_lbl := Label{
text: "—"
draw_text.color: #xe8edf4
draw_text.text_style.font_size: 9.0
draw_text.color: mp.fg
draw_text.text_style: theme.font_code{font_size: 9.0}
}
}
@ -149,12 +161,14 @@ let Strip = View{
show_bg: true
new_batch: true
draw_bg +: {
fill: uniform(mp.bg)
edge: uniform(mp.muted)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.rect(0.0, 0.0, self.rect_size.x, self.rect_size.y)
sdf.fill(#x0e1218)
sdf.fill(self.fill)
sdf.rect(self.rect_size.x - 1.0, 0.0, 1.0, self.rect_size.y)
sdf.fill(#x1c232e)
sdf.fill(self.edge)
return sdf.result
}
}
@ -164,8 +178,8 @@ let Strip = View{
width: Fill
align: Align{x: 0.5}
text: ""
draw_text.color: #x8d99aa
draw_text.text_style.font_size: 8.0
draw_text.color: mp.fg_dim
draw_text.text_style: theme.font_code{font_size: 8.0}
}
fader_db := ValueLabel{}
fader_block := View{
@ -186,14 +200,14 @@ View{
flow: Down
show_bg: true
new_batch: true
draw_bg.color: #x080a0e
draw_bg.color: mp.bg_dark
surface_note := Label{
width: Fill
align: Align{x: 0.5}
text: ""
draw_text.color: #x77828f
draw_text.text_style.font_size: 8.0
draw_text.color: mp.fg_dim
draw_text.text_style: theme.font_code{font_size: 8.0}
}
strip_row := View{
width: Fill

View file

@ -5,6 +5,25 @@
// model) can write a different surface by editing/replacing this file and
// starting the app with --layout=<name>; the Rust host rebinds by widget name.
//
// == The palette ==
// A Splash body runs in its own isolate VM, which registers a FRESH stock
// `mod.theme` and never sees the app's retint — so the host PREPENDS one line
// binding `mp` to the desktop palette (src/theme.rs) before handing this file
// to the Splash. A script error therefore reports this file's line + 1.
// The keys, all `#rrggbb`:
//
// mp.bg strip fill mp.accent selection, gain
// mp.bg_dark gutter and inset panels mp.red mute lit, meter hot
// mp.bg_light idle control fill mp.green meter low
// mp.fg readouts mp.yellow EQ, meter mid
// mp.fg_bright the value that matters mp.cyan dynamics, pan
// mp.fg_dim labels, scales
// mp.muted 1px borders, tracks, ticks
//
// The look is Omarchy: flat fills, square corners, one 1px border per state,
// no bevels and no gradients. Colours belong to the theme — a layout that
// wants a new one asks for a palette key, not a literal.
//
// == The slot contract ==
// The host looks for strip slots named strip_0 .. strip_15 and binds the
// mixer's own strip list (derived from the console's stereo-link state) to
@ -42,8 +61,8 @@ let ValueLabel = Label{
flow: Right
align: Align{x: 0.5}
text: "—"
draw_text.color: #xcfd8e3
draw_text.text_style.font_size: 8.0
draw_text.color: mp.fg
draw_text.text_style: theme.font_code{font_size: 8.0}
}
let RowSlider = Slider{
@ -55,20 +74,22 @@ let RowSlider = Slider{
text: ""
text_input: TextInput{width: 0, height: 0}
draw_bg +: {
track: uniform(#x2b3546)
fill: uniform(#x2f6fe0)
knob: uniform(#xf2f5fa)
track: uniform(mp.muted)
fill: uniform(mp.accent)
knob: uniform(mp.fg_bright)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
let cy = self.rect_size.y * 0.5
let x0 = 4.0
let x1 = self.rect_size.x - 4.0
sdf.box(x0, cy - 1.5, x1 - x0, 3.0, 1.5)
sdf.rect(x0, cy - 1.0, x1 - x0, 2.0)
sdf.fill(self.track)
let px = x0 + (x1 - x0) * self.slide_pos
sdf.box(x0, cy - 1.5, max(px - x0, 1.0), 3.0, 1.5)
sdf.rect(x0, cy - 1.0, max(px - x0, 1.0), 2.0)
sdf.fill(self.fill)
sdf.circle(px, cy, 3.6)
// A square tab, not a bead: the position is a value, and a flat
// edge is easier to read one against the next.
sdf.rect(px - 1.5, cy - 4.5, 3.0, 9.0)
sdf.fill(self.knob)
return sdf.result
}
@ -77,22 +98,23 @@ let RowSlider = Slider{
let EqSlider = RowSlider{
draw_bg +: {
fill: uniform(#xb98c2a)
fill: uniform(mp.yellow)
detent: uniform(mp.fg_dim)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
let cy = self.rect_size.y * 0.5
let x0 = 4.0
let x1 = self.rect_size.x - 4.0
sdf.box(x0, cy - 1.5, x1 - x0, 3.0, 1.5)
sdf.rect(x0, cy - 1.0, x1 - x0, 2.0)
sdf.fill(self.track)
// fill grows out of the centre: this slider is a +/- 15 dB cut/boost
let mid = (x0 + x1) * 0.5
let px = x0 + (x1 - x0) * self.slide_pos
sdf.box(min(px, mid), cy - 1.5, max(abs(px - mid), 1.0), 3.0, 1.5)
sdf.rect(min(px, mid), cy - 1.0, max(abs(px - mid), 1.0), 2.0)
sdf.fill(self.fill)
sdf.box(mid - 0.5, cy - 3.5, 1.0, 7.0, 0.5)
sdf.fill(#x55606e)
sdf.circle(px, cy, 3.6)
sdf.rect(mid - 0.5, cy - 3.5, 1.0, 7.0)
sdf.fill(self.detent)
sdf.rect(px - 1.5, cy - 4.5, 3.0, 9.0)
sdf.fill(self.knob)
return sdf.result
}
@ -101,7 +123,7 @@ let EqSlider = RowSlider{
let PanSlider = RowSlider{
draw_bg +: {
fill: uniform(#x3a6ea8)
fill: uniform(mp.cyan)
}
}
@ -110,6 +132,12 @@ let EqCurve = SolidView{
height: 38
cursor: MouseCursor.Hand
draw_bg +: {
c_panel: uniform(mp.bg_dark)
c_zero: uniform(mp.muted)
c_ink: uniform(mp.yellow)
c_ink_off: uniform(mp.fg_dim)
c_sel: uniform(mp.accent)
c_dot: uniform(mp.fg_bright)
eq_on: instance(1.0)
b1: instance(vec4(2.0, 0.15, 0.5, 0.5))
b2: instance(vec4(2.0, 0.40, 0.5, 0.5))
@ -145,16 +173,18 @@ let EqCurve = SolidView{
let py = self.pos.y * h
let curve = 1.0 - smoothstep(0.8, 1.9, abs(py - y))
let center = 1.0 - smoothstep(0.2, 0.9, abs(py - h * 0.5))
let base = vec3(0.02, 0.027, 0.04)
let amber = vec3(1.0, 0.84, 0.35) * (0.35 + 0.65 * self.eq_on)
var col = base + vec3(0.09, 0.11, 0.14) * center + amber * curve
// The band the gain slider is driving: a dim vertical hairline
// plus a bright dot where it meets the curve.
// A flat inset panel, a muted zero line, and the curve — dimmed to
// the quiet foreground while the console reports EQ off, so an
// unknown state never reads as a confident flat response.
let ink = self.c_ink_off.mix(self.c_ink, self.eq_on)
var col = self.c_panel.mix(self.c_zero, center).mix(ink, curve)
// The band the gain slider drives: a dim accent hairline plus a
// bright dot where it meets the curve.
let dx = abs(x - self.sel_f) * w
col = col + vec3(0.10, 0.13, 0.18) * (1.0 - smoothstep(0.4, 1.3, dx))
let dot = 1.0 - smoothstep(1.6, 2.9, length(vec2(dx, py - y)))
col = col.mix(vec3(1.0, 0.93, 0.62), dot)
return vec4(col, 1.0)
col = col.mix(self.c_sel, 0.4 * (1.0 - smoothstep(0.4, 1.3, dx)))
let hit = 1.0 - smoothstep(1.6, 2.9, length(vec2(dx, py - y)))
col = col.mix(self.c_dot, hit)
return col
}
}
}
@ -163,6 +193,10 @@ let DynCurve = SolidView{
width: Fill
height: 32
draw_bg +: {
c_panel: uniform(mp.bg_dark)
c_ink: uniform(mp.cyan)
c_gate: uniform(mp.cyan)
c_gr: uniform(mp.red)
comp_on: instance(0.0)
thr_db: instance(0.0)
ratio: instance(1.0)
@ -182,22 +216,20 @@ let DynCurve = SolidView{
let y = h * (-out_db / 60.0)
let py = self.pos.y * h
let curve = 1.0 - smoothstep(0.8, 1.9, abs(py - y))
let base = vec3(0.02, 0.027, 0.04)
let amber = vec3(1.0, 0.84, 0.35)
var col = base + amber * curve
// gate region: filled wedge below the gate threshold
var col = self.c_panel.mix(self.c_ink, curve)
// gate region: a flat wash below the gate threshold
if self.gate_on > 0.5 {
let gx = (self.gate_thr_db + 60.0) / 60.0
if self.pos.x < gx && py > y {
col = col + vec3(0.25, 0.20, 0.06)
col = col.mix(self.c_gate, 0.22)
}
}
// live gain reduction: a bar dropping from the top right
let grn = clamp(-self.gr_db / 20.0, 0.0, 1.0)
if self.pos.x > 1.0 - 3.5 / w && self.pos.y < grn {
col = vec3(1.0, 0.45, 0.15)
col = self.c_gr
}
return vec4(col, 1.0)
return col
}
}
}
@ -206,6 +238,12 @@ let MeterBar = SolidView{
width: Fill{weight: 14.0}
height: Fill
draw_bg +: {
gutter: uniform(mp.bg)
trough: uniform(mp.bg_dark)
low: uniform(mp.green)
mid: uniform(mp.yellow)
hot: uniform(mp.red)
tick: uniform(mp.fg_bright)
level_db: instance(-90.0)
peak_db: instance(-90.0)
pixel: fn() {
@ -214,27 +252,25 @@ let MeterBar = SolidView{
let norm = clamp((self.level_db + 60.0) / 60.0, 0.0, 1.0)
let peakn = clamp((self.peak_db + 60.0) / 60.0, 0.0, 1.0)
let frac = 1.0 - self.pos.y
var col = vec3(0.03, 0.09, 0.045)
// The ladder: green, into yellow from about -18 dB, into red over
// the last three. Unlit, the red zone stays visible as a dim cap.
let ladder = self.low.mix(self.mid, smoothstep(0.70, 0.88, frac))
.mix(self.hot, smoothstep(0.88, 0.97, frac))
// segment shading so the bar reads "LED-ish"
let seg = 0.85 + 0.15 * smoothstep(0.35, 0.5, abs(fract(frac * 30.0) - 0.5))
let seg = 0.88 + 0.12 * smoothstep(0.35, 0.5, abs(fract(frac * 30.0) - 0.5))
var col = self.trough.mix(self.hot, 0.16 * step(0.955, frac))
if frac < norm {
let hot = smoothstep(0.82, 0.93, frac)
col = vec3(0.10, 0.78, 0.28).mix(vec3(1.0, 0.22, 0.16), hot) * seg
}
// permanent red zone cap
if frac > 0.955 {
col = col.mix(vec3(0.45, 0.09, 0.07), 0.8)
if frac < norm { col = vec3(1.0, 0.25, 0.18) }
col = self.trough.mix(ladder, seg)
}
// peak-hold tick
if abs(frac - peakn) < 1.5 / h && peakn > 0.01 {
col = vec3(0.9, 1.0, 0.9)
col = self.tick
}
let x = self.pos.x * w
if x < 1.0 || x > w - 1.0 {
col = vec3(0.015, 0.02, 0.03)
col = self.gutter
}
return vec4(col, 1.0)
return col
}
}
}
@ -249,37 +285,40 @@ let FaderSlider = Slider{
text: ""
text_input: TextInput{width: 0, height: 0}
draw_bg +: {
body: uniform(#x0b0f15)
slot: uniform(#x171e29)
cap: uniform(#x9aa3ad)
cap_dark: uniform(#x4a5158)
body: uniform(mp.bg_dark)
slot: uniform(mp.muted)
tick: uniform(mp.muted)
cap: uniform(mp.fg_dim)
cap_edge: uniform(mp.accent)
cap_line: uniform(mp.fg_bright)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
let h = self.rect_size.y
let w = self.rect_size.x
sdf.box(0.0, 0.0, w, h, 2.0)
sdf.rect(0.0, 0.0, w, h)
sdf.fill(self.body)
// tick marks every 1/8 of travel (the taper puts the printed
// dB scale exactly on these lines)
let rel = self.pos.y
let grid = abs(fract(rel * 8.0 + 0.5) - 0.5) * (h / 8.0)
let tick = 1.0 - smoothstep(0.4, 0.9, grid)
let mark = 1.0 - smoothstep(0.4, 0.9, grid)
let px = self.pos.x * w
if px < 6.0 {
sdf.box(0.0, rel * h - 0.4, 5.0, 0.8, 0.2)
sdf.fill(vec4(0.55, 0.62, 0.7, 1.0) * tick)
sdf.rect(0.0, rel * h - 0.5, 5.0, 1.0)
sdf.fill(self.tick * mark)
}
// centre slot
sdf.box(w * 0.5 - 1.5, 3.0, 3.0, h - 6.0, 1.5)
sdf.rect(w * 0.5 - 1.0, 3.0, 2.0, h - 6.0)
sdf.fill(self.slot)
// handle: wide ribbed block with a white centre line
// handle: a flat square cap, edged in the accent, with the
// pointer line across it
let cap_h = 22.0
let cy = (1.0 - self.slide_pos) * (h - cap_h)
sdf.box(3.0, cy, w - 6.0, cap_h, 2.5)
let rib = 0.75 + 0.25 * smoothstep(0.25, 0.5, abs(fract((self.pos.y * h - cy) / 4.4) - 0.5))
sdf.fill(self.cap.mix(self.cap_dark, 1.0 - rib))
sdf.box(3.0, cy + cap_h * 0.5 - 1.0, w - 6.0, 2.0, 0.5)
sdf.fill(#xf4f7fb)
sdf.rect(3.5, cy + 0.5, w - 7.0, cap_h - 1.0)
sdf.fill_keep(self.cap)
sdf.stroke(self.cap_edge, 1.0)
sdf.rect(3.0, cy + cap_h * 0.5 - 1.0, w - 6.0, 2.0)
sdf.fill(self.cap_line)
return sdf.result
}
}
@ -290,8 +329,8 @@ let ScaleMark = Label{
height: Fill
flow: Right
align: Align{x: 1.0}
draw_text.color: #x6b7686
draw_text.text_style.font_size: 7.0
draw_text.color: mp.fg_dim
draw_text.text_style: theme.font_code{font_size: 7.0}
text: ""
}
@ -301,7 +340,7 @@ let ScaleCol = View{
flow: Down
ScaleMark{text: "10"}
ScaleMark{text: "5"}
ScaleMark{text: "0" draw_text.color: #xa8b4c4}
ScaleMark{text: "0" draw_text.color: mp.fg}
ScaleMark{text: "-5"}
ScaleMark{text: "-10"}
ScaleMark{text: "-20"}
@ -316,20 +355,28 @@ let MutePlate = SolidView{
new_batch: true
align: Align{x: 0.5, y: 0.5}
draw_bg +: {
idle: uniform(mp.bg_light)
idle_edge: uniform(mp.muted)
lit: uniform(mp.red)
deep: uniform(mp.bg_dark)
muted: instance(0.0)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.box(1.0, 1.0, self.rect_size.x - 2.0, self.rect_size.y - 2.0, 3.0)
let idle = vec4(0.23, 0.07, 0.06, 1.0)
let lit = vec4(0.82, 0.13, 0.10, 1.0)
sdf.fill(idle.mix(lit, self.muted))
sdf.rect(0.5, 0.5, self.rect_size.x - 1.0, self.rect_size.y - 1.0)
// Lit is red laid over the deepest background: a red plate that
// still carries its own word, edged in the full-strength hue.
let fill = self.idle.mix(self.lit.mix(self.deep, 0.5), self.muted)
// fill_keep, not fill: `fill` clears the shape, so a stroke after
// it draws nothing at all.
sdf.fill_keep(fill)
sdf.stroke(self.idle_edge.mix(self.lit, self.muted), 1.0)
return sdf.result
}
}
Label{
text: "MUTE"
draw_text.color: #xf3d9d6
draw_text.text_style.font_size: 7.0
draw_text.color: mp.fg_bright
draw_text.text_style: theme.font_code{font_size: 7.0}
}
}
@ -340,23 +387,24 @@ let NamePlate = SolidView{
new_batch: true
align: Align{x: 0.5, y: 0.5}
draw_bg +: {
plate_rgb: instance(vec3(0.5, 0.5, 0.5))
idle: uniform(mp.bg_light)
// The console's scribble colour, until it reports one. The default is
// the muted key — nothing on this surface is a neutral grey.
plate_rgb: instance(vec3(0.254, 0.282, 0.408))
plate_filled: instance(0.0)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.box(1.0, 1.0, self.rect_size.x - 2.0, self.rect_size.y - 2.0, 3.0)
let outline = vec4(0.03, 0.04, 0.055, 1.0)
let filled = vec4(self.plate_rgb * 0.82, 1.0)
sdf.fill(outline.mix(filled, self.plate_filled))
sdf.stroke(vec4(self.plate_rgb, 1.0), 1.2)
sdf.rect(0.5, 0.5, self.rect_size.x - 1.0, self.rect_size.y - 1.0)
sdf.fill_keep(self.idle.mix(vec4(self.plate_rgb, 1.0), self.plate_filled))
sdf.stroke(vec4(self.plate_rgb, 1.0), 1.0)
return sdf.result
}
}
name_lbl := Label{
flow: Right
text: "—"
draw_text.color: #xe8edf4
draw_text.text_style.font_size: 7.5
draw_text.color: mp.fg
draw_text.text_style: theme.font_code{font_size: 7.5}
}
}
@ -369,12 +417,14 @@ let Strip = View{
show_bg: true
new_batch: true
draw_bg +: {
fill: uniform(mp.bg)
edge: uniform(mp.muted)
pixel: fn() {
let sdf = Sdf2d.viewport(self.pos * self.rect_size)
sdf.rect(0.0, 0.0, self.rect_size.x, self.rect_size.y)
sdf.fill(#x10141b)
sdf.fill(self.fill)
sdf.rect(self.rect_size.x - 1.0, 0.0, 1.0, self.rect_size.y)
sdf.fill(#x1c232e)
sdf.fill(self.edge)
return sdf.result
}
}
@ -405,8 +455,8 @@ let Strip = View{
flow: Right
align: Align{x: 0.5}
text: "—"
draw_text.color: #xe6ecf4
draw_text.text_style.font_size: 9.0
draw_text.color: mp.fg_bright
draw_text.text_style: theme.font_code{font_size: 9.0}
}
fader_block := View{
width: Fill
@ -424,8 +474,8 @@ let Strip = View{
flow: Right
align: Align{x: 0.5}
text: ""
draw_text.color: #x8d99aa
draw_text.text_style.font_size: 7.5
draw_text.color: mp.fg_dim
draw_text.text_style: theme.font_code{font_size: 7.5}
}
mute := MutePlate{}
name_plate := NamePlate{}
@ -437,7 +487,7 @@ View{
flow: Down
show_bg: true
new_batch: true
draw_bg.color: #x080a0e
draw_bg.color: mp.bg_dark
surface_note := Label{
visible: false
@ -445,8 +495,8 @@ View{
height: Fit
align: Align{x: 0.5}
text: ""
draw_text.color: #x77828f
draw_text.text_style.font_size: 8.0
draw_text.color: mp.fg_dim
draw_text.text_style: theme.font_code{font_size: 8.0}
}
strip_row := View{
width: Fill

View file

@ -38,7 +38,9 @@ use std::net::SocketAddr;
use std::sync::Arc;
mod surface;
mod theme;
use surface::SurfaceBinder;
use theme::Palette;
app_main!(App);
@ -77,11 +79,18 @@ script_mod! {
main_window := Window{
window.title: "Mixer"
window.inner_size: vec2(1268, 716)
pass.clear_color: #x07090d
// The desk is flat and dark: the pass clears to the theme's
// own background, not the stock neutral. `theme.rs` also sets
// `color_bg_app`/`color_app_caption_bar` from the same key,
// but the clear colour is stated here so a window that never
// gets a retint still comes up in the theme.
pass.clear_color: mod.mpm.bg_dark
body +: {
width: Fill
height: Fill
flow: Down
show_bg: true
draw_bg.color: mod.mpm.bg_dark
searching := View{
width: Fill
@ -92,17 +101,20 @@ script_mod! {
Label{
text: "Searching for your mixer"
draw_text.color: #xe8eef6
draw_text.text_style: theme.font_bold{font_size: 17.0}
draw_text.color: mod.mpm.fg_bright
draw_text.text_style: theme.font_code{font_size: 15.0}
}
search_note := Label{
text: "listening for a console on the local network"
draw_text.color: #x8d99aa
draw_text.text_style.font_size: 9.0
draw_text.color: mod.mpm.fg_dim
draw_text.text_style: theme.font_code{font_size: 9.0}
}
LoadingSpinner{
width: 34
height: 34
draw_bg +: {
color: uniform(mod.mpm.accent)
}
}
}
@ -154,7 +166,15 @@ impl App {
fn load_layout(&mut self, cx: &mut Cx) {
let (_label, file) = LAYOUTS[self.layout_idx];
let body = load_layout_body(file);
// A Splash body runs in its own isolate VM with a FRESH stock
// `mod.theme`, so the desktop palette cannot reach a layout through
// the theme — the host hands it over as one prepended line binding
// `mp` (see theme.rs and the layout headers).
let body = format!(
"{}{}",
Palette::shared().splash_preamble(),
load_layout_body(file)
);
let splash = self.splash_ref(cx);
splash.set_text(cx, &body);
self.binder.rebind(cx, &splash);
@ -356,6 +376,8 @@ impl MatchEvent for App {
impl AppMain for App {
fn script_mod(vm: &mut ScriptVm) -> ScriptValue {
crate::makepad_widgets::script_mod(vm);
mp_theme::apply(vm);
Palette::shared().install(vm);
self::script_mod(vm)
}

View file

@ -256,7 +256,14 @@ impl SurfaceBinder {
let rgb = scribble_rgb(idx);
set_view_instance(cx, &slot.name_plate, "plate_rgb", &rgb);
set_view_instance(cx, &slot.name_plate, "plate_filled", &[filled]);
let text_rgb = if filled > 0.5 { [0.04, 0.05, 0.07] } else { rgb };
// A filled plate carries the console's own scribble colour, so
// its text is the theme's deepest background — the one colour
// guaranteed to read on every scribble hue.
let text_rgb = if filled > 0.5 {
crate::theme::Palette::shared().rgb3("bg_dark")
} else {
rgb
};
set_label_color(cx, &slot.name_lbl, text_rgb);
}

238
apps/mixer/src/theme.rs Normal file
View file

@ -0,0 +1,238 @@
//! The desk's palette. mpwm exports its active `theme.splash` as
//! MPWM_THEME_SPLASH; `mp_theme` line-scans it and retints the *stock*
//! widgets, and this module carries the same colours to the three places the
//! mixer paints itself:
//!
//! * `mod.mpm.*` — read by `main.rs`'s own `script_mod!` (the window and
//! the search page);
//! * a one-line `let mp = {...}` preamble prepended to every surface layout
//! — a `Splash` body runs in its OWN isolate VM, which registers a FRESH
//! stock `mod.theme` and never sees `mod.mpm`, so a layout has to be
//! handed its colours (see [`Palette::splash_preamble`]);
//! * [`Palette::rgb3`], for the one colour `surface.rs` sets from Rust.
//!
//! It also applies the stock retint with these same values, because
//! `mp_theme::apply` is a no-op when the WM is not running and that would
//! otherwise leave the caption bar in the neutral stock theme above a black
//! desk. Standalone runs get Tokyo Night, so the surface is dark and square
//! either way.
//!
//! SAFETY: colours only. Nothing here can name an OSC address — see
//! `makepad_mixer::safety`.
use makepad_widgets::*;
use std::sync::OnceLock;
/// A palette entry: the name the DSL reads it by, the key in the WM's
/// theme.splash, and the Tokyo Night fallback.
///
/// The meter and lamp hues live in the theme's terminal block — omarchy's
/// base16 mapping is colour1 = red, 2 = green, 3 = yellow, 6 = cyan — because
/// a desktop theme has no "signal is clipping" role of its own.
const KEYS: &[(&str, &str, &str)] = &[
("bg", "background", "#1a1b26"),
("bg_dark", "darker_background", "#0e0e14"),
("bg_light", "lighter_background", "#24283b"),
("fg", "foreground", "#a9b1d6"),
("fg_bright", "bright_foreground", "#c0caf5"),
("fg_dim", "dark_foreground", "#565f89"),
("muted", "muted", "#414868"),
("accent", "accent", "#7aa2f7"),
("red", "term.color1", "#f7768e"),
("green", "term.color2", "#9ece6a"),
("yellow", "term.color3", "#e0af68"),
("cyan", "term.color6", "#449dab"),
];
/// Every colour the mixer paints with, as `#rrggbb` strings.
#[derive(Clone, Debug)]
pub struct Palette {
/// `name` -> `#rrggbb`, in [`KEYS`] order.
entries: Vec<(&'static str, String)>,
}
impl Palette {
/// The palette for this process, read once.
pub fn shared() -> &'static Palette {
static PALETTE: OnceLock<Palette> = OnceLock::new();
PALETTE.get_or_init(Palette::load)
}
/// The palette mpwm exported for this process, with Tokyo Night standing
/// in for anything it does not name.
pub fn load() -> Self {
let wm = mp_theme::current();
Palette {
entries: KEYS
.iter()
.map(|(name, key, fallback)| {
let hex = match &wm {
Some(p) => p.hex(key, fallback),
None => fallback.to_string(),
};
(*name, hex)
})
.collect(),
}
}
/// One colour by its DSL name. Unknown names read magenta rather than
/// silently theme-shaped black.
pub fn get(&self, name: &str) -> &str {
self.entries
.iter()
.find(|(n, _)| *n == name)
.map(|(_, v)| v.as_str())
.unwrap_or("#ff00ff")
}
/// One colour as linear-ish rgb components, for the handful of places
/// Rust sets a colour directly.
pub fn rgb3(&self, name: &str) -> [f32; 3] {
let hex = self.get(name).trim_start_matches('#');
let nib = |i: usize| -> f32 {
match hex.as_bytes().get(i).copied().unwrap_or(b'0') {
c @ b'0'..=b'9' => (c - b'0') as f32,
c @ b'a'..=b'f' => (c - b'a' + 10) as f32,
c @ b'A'..=b'F' => (c - b'A' + 10) as f32,
_ => 0.0,
}
};
let byte = |i: usize| (nib(i * 2) * 16.0 + nib(i * 2 + 1)) / 255.0;
[byte(0), byte(1), byte(2)]
}
/// The palette as ONE line of splash source, to be prepended to a layout
/// body before it is handed to the `Splash` widget. One line so a script
/// error's reported line is the layout's own line plus exactly one.
pub fn splash_preamble(&self) -> String {
let body: Vec<String> = self
.entries
.iter()
.map(|(name, hex)| format!("{name}: {hex}"))
.collect();
format!("let mp = {{{}}}\n", body.join(", "))
}
/// Publish `mod.mpm.*` for the app's own `script_mod!`, and retint the
/// stock widgets (the caption bar, the spinner) with the same palette.
/// Call once, after `makepad_widgets::script_mod` and `mp_theme::apply`,
/// and before this crate's own `script_mod`.
pub fn install(&self, vm: &mut ScriptVm) {
let mut code = String::from("mod.mpm = {\n");
for (name, hex) in &self.entries {
code.push_str(&format!(" {name}: {hex}\n"));
}
code.push_str("}\n");
// `mp_theme::apply` only retints when the WM exported a palette;
// standalone that leaves the window chrome in the stock theme, which
// reads as a grey band above a Tokyo Night desk. Same keys, our
// fallbacks.
let c = |name: &str| self.get(name).to_string();
code.push_str(&format!(
"mod.theme.color_b = {bg_dark}\n\
mod.theme.color_b_h = {bg_dark}00\n\
mod.theme.color_w = {fg_bright}\n\
mod.theme.color_w_h = {fg_bright}00\n\
mod.theme.color_bg_app = {bg}\n\
mod.theme.color_fg_app = {bg_light}\n\
mod.theme.color_bg_container = {bg_dark}\n\
mod.theme.color_text = {fg}\n\
mod.theme.color_text_hover = {fg_bright}\n\
mod.theme.color_text_muted = {fg_dim}\n\
mod.theme.color_focus = {accent}\n\
mod.theme.color_outset_active = {accent}\n\
mod.theme.color_ctrl_default = {bg_light}\n\
mod.theme.color_ctrl_hover = {muted}\n\
mod.theme.color_ctrl_active = {accent}\n\
mod.theme.color_ctrl_selected = {accent}\n\
mod.theme.color_app_caption_bar = {bg_dark}\n\
mod.theme.corner_radius = 0.0\n\
true\n",
bg = c("bg"),
bg_dark = c("bg_dark"),
bg_light = c("bg_light"),
fg = c("fg"),
fg_bright = c("fg_bright"),
fg_dim = c("fg_dim"),
muted = c("muted"),
accent = c("accent"),
));
vm.eval(ScriptMod {
cargo_manifest_path: env!("CARGO_MANIFEST_DIR").to_string(),
module_path: "mixer_theme".to_string(),
file: "mixer_theme.splash".to_string(),
line: 0,
column: 0,
code,
values: vec![],
});
for e in vm.take_errors() {
log!("mixer theme: {}", e);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn standalone_falls_back_to_tokyo_night() {
let p = Palette::load();
// The WM is not running under `cargo test`, so every key is a
// fallback — and the fallbacks ARE mpwm's bundled tokyo-night.
assert_eq!(p.get("bg"), "#1a1b26");
assert_eq!(p.get("bg_dark"), "#0e0e14");
assert_eq!(p.get("accent"), "#7aa2f7");
assert_eq!(p.get("red"), "#f7768e");
assert_eq!(p.get("green"), "#9ece6a");
assert_eq!(p.get("yellow"), "#e0af68");
assert_eq!(p.get("cyan"), "#449dab");
// Nothing reads as neutral grey: every channel pair differs.
for (name, hex) in &p.entries {
let [r, g, b] = p.rgb3(name);
assert!(
(r - g).abs() > 0.001 || (g - b).abs() > 0.001,
"{name} = {hex} is a neutral grey"
);
}
}
#[test]
fn a_wm_palette_wins_over_every_fallback() {
// Both the top-level roles and the terminal hues come from the WM's
// own theme.splash when it is running.
let src = "mod.mpwm_theme = {\n accent: #ff8800\n background: #101010\n term: {\n color1: #123456\n }\n}\n";
let wm = mp_theme::scan(src);
assert_eq!(wm.hex("accent", "#7aa2f7"), "#ff8800");
assert_eq!(wm.hex("term.color1", "#f7768e"), "#123456");
// ...and a key the theme omits still resolves.
assert_eq!(wm.hex("term.color6", "#449dab"), "#449dab");
}
#[test]
fn the_layout_preamble_is_exactly_one_line() {
let p = Palette::load();
let pre = p.splash_preamble();
assert_eq!(pre.lines().count(), 1, "preamble must stay one line: {pre}");
assert!(pre.ends_with('\n'));
assert!(pre.starts_with("let mp = {"));
// Every colour a layout can ask for is bound.
for (name, hex) in &p.entries {
assert!(pre.contains(&format!("{name}: {hex}")), "missing {name}");
}
}
#[test]
fn parses_hex() {
let p = Palette::load();
let [r, g, b] = p.rgb3("bg_dark"); // #0e0e14
assert!((r - 14.0 / 255.0).abs() < 1e-6);
assert!((g - 14.0 / 255.0).abs() < 1e-6);
assert!((b - 20.0 / 255.0).abs() < 1e-6);
assert_eq!(p.get("nope"), "#ff00ff");
}
}

25
apps/mpbrowser/Cargo.toml Normal file
View file

@ -0,0 +1,25 @@
# mpbrowser — a Chrome-like browser as a plain full-window Makepad app.
#
# Chromium (CEF, libs/cef) renders ONLY the page, GPU-accelerated straight
# into an IOSurface-backed Makepad texture. All browser chrome — the tab
# strip, the toolbar with back/forward/reload, the omnibox and the menu —
# is Makepad splash UI, styled from the mpwm theme when hosted by
# makepad-wm (`MPWM_THEME_SPLASH`) and from a Chrome-dark palette otherwise.
#
# Runs standalone, and unmodified inside makepad-wm / Studio tiles via the
# shared --stdin-loop client runtime every Makepad app has.
[package]
name = "mpbrowser"
version = "0.1.0"
edition = "2021"
default-run = "mpbrowser"
[[bin]]
name = "mpbrowser"
path = "src/main.rs"
[dependencies]
makepad-widgets = { path = "../../widgets", features = ["cef"] }
makepad-cef = { path = "../../libs/cef" }
mp-theme = { path = "../../libs/mp_theme" }

View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 15.60 4.40 L 17.00 5.80 L 10.80 12.00 L 17.00 18.20 L 15.60 19.60 L 8.00 12.00 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 231 B

View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 6.00 7.40 L 7.40 6.00 L 12.00 10.60 L 16.60 6.00 L 18.00 7.40 L 13.40 12.00 L 18.00 16.60 L 16.60 18.00 L 12.00 13.40 L 7.40 18.00 L 6.00 16.60 L 10.60 12.00 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 310 B

View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 8.40 4.40 L 7.00 5.80 L 13.20 12.00 L 7.00 18.20 L 8.40 19.60 L 16.00 12.00 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 228 B

View file

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 20.00 12.00 L 19.73 14.07 L 18.93 16.00 L 17.66 17.66 L 16.00 18.93 L 14.07 19.73 L 12.00 20.00 L 9.93 19.73 L 8.00 18.93 L 6.34 17.66 L 5.07 16.00 L 4.27 14.07 L 4.00 12.00 L 4.27 9.93 L 5.07 8.00 L 6.34 6.34 L 8.00 5.07 L 9.93 4.27 L 12.00 4.00 L 14.07 4.27 L 16.00 5.07 L 17.66 6.34 L 18.93 8.00 L 19.73 9.93 L 20.00 12.00 L 18.40 12.00 L 18.18 10.34 L 17.54 8.80 L 16.53 7.47 L 15.20 6.46 L 13.66 5.82 L 12.00 5.60 L 10.34 5.82 L 8.80 6.46 L 7.47 7.47 L 6.46 8.80 L 5.82 10.34 L 5.60 12.00 L 5.82 13.66 L 6.46 15.20 L 7.47 16.53 L 8.80 17.54 L 10.34 18.18 L 12.00 18.40 L 13.66 18.18 L 15.20 17.54 L 16.53 16.53 L 17.54 15.20 L 18.18 13.66 L 18.40 12.00 Z"/>
<path d="M 11.20 4.40 L 12.80 4.40 L 12.80 19.60 L 11.20 19.60 Z"/>
<path d="M 4.40 11.20 L 19.60 11.20 L 19.60 12.80 L 4.40 12.80 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 950 B

View file

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 10.80 4.60 L 13.20 4.60 L 13.20 7.00 L 10.80 7.00 Z"/>
<path d="M 10.80 10.80 L 13.20 10.80 L 13.20 13.20 L 10.80 13.20 Z"/>
<path d="M 10.80 17.00 L 13.20 17.00 L 13.20 19.40 L 10.80 19.40 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 346 B

View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 11.00 5.00 L 13.00 5.00 L 13.00 11.00 L 19.00 11.00 L 19.00 13.00 L 13.00 13.00 L 13.00 19.00 L 11.00 19.00 L 11.00 13.00 L 5.00 13.00 L 5.00 11.00 L 11.00 11.00 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 314 B

View file

@ -0,0 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 15.75 5.50 L 17.07 6.47 L 18.14 7.70 L 18.93 9.13 L 19.39 10.70 L 19.49 12.33 L 19.24 13.94 L 18.65 15.46 L 17.75 16.82 L 16.57 17.95 L 15.17 18.80 L 13.62 19.32 L 12.00 19.50 L 10.38 19.32 L 8.83 18.80 L 7.43 17.95 L 6.25 16.82 L 5.35 15.46 L 4.76 13.94 L 4.51 12.33 L 4.61 10.70 L 5.07 9.13 L 5.86 7.70 L 6.93 6.47 L 8.25 5.50 L 9.25 7.24 L 8.28 7.94 L 7.49 8.85 L 6.92 9.90 L 6.58 11.04 L 6.51 12.24 L 6.69 13.42 L 7.12 14.54 L 7.79 15.54 L 8.65 16.36 L 9.68 16.98 L 10.81 17.37 L 12.00 17.50 L 13.19 17.37 L 14.32 16.98 L 15.35 16.36 L 16.21 15.54 L 16.88 14.54 L 17.31 13.42 L 17.49 12.24 L 17.42 11.04 L 17.08 9.90 L 16.51 8.85 L 15.72 7.94 L 14.75 7.24 Z"/>
<path d="M 18.45 5.17 L 14.25 2.77 L 14.65 8.77 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 867 B

View file

@ -0,0 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 16.00 10.00 L 15.80 11.55 L 15.20 13.00 L 14.24 14.24 L 13.00 15.20 L 11.55 15.80 L 10.00 16.00 L 8.45 15.80 L 7.00 15.20 L 5.76 14.24 L 4.80 13.00 L 4.20 11.55 L 4.00 10.00 L 4.20 8.45 L 4.80 7.00 L 5.76 5.76 L 7.00 4.80 L 8.45 4.20 L 10.00 4.00 L 11.55 4.20 L 13.00 4.80 L 14.24 5.76 L 15.20 7.00 L 15.80 8.45 L 16.00 10.00 L 14.00 10.00 L 13.86 8.96 L 13.46 8.00 L 12.83 7.17 L 12.00 6.54 L 11.04 6.14 L 10.00 6.00 L 8.96 6.14 L 8.00 6.54 L 7.17 7.17 L 6.54 8.00 L 6.14 8.96 L 6.00 10.00 L 6.14 11.04 L 6.54 12.00 L 7.17 12.83 L 8.00 13.46 L 8.96 13.86 L 10.00 14.00 L 11.04 13.86 L 12.00 13.46 L 12.83 12.83 L 13.46 12.00 L 13.86 11.04 L 14.00 10.00 Z"/>
<path d="M 14.20 13.00 L 15.60 14.40 L 20.60 19.40 L 19.20 20.80 L 14.20 15.80 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 892 B

View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="24px" height="24px" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<path d="M 12.00 4.60 L 14.00 9.85 L 19.61 10.13 L 15.23 13.65 L 16.70 19.07 L 12.00 16.00 L 7.30 19.07 L 8.77 13.65 L 4.39 10.13 L 10.00 9.85 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 284 B

View file

@ -0,0 +1,512 @@
//! The browser chrome: a custom-drawn Chrome-style tab strip (favicon +
//! title tabs with a close x, a + for a new tab) and the toolbar
//! (back / forward / reload, the omnibox with search icon and bookmark star,
//! the menu button). Hard-square Omarchy look, colours from `mod.mpb_theme`.
use crate::tabs::{TabId, TabSummary};
use makepad_widgets::image::DrawImage;
use makepad_widgets::*;
script_mod! {
use mod.prelude.widgets_internal.*
use mod.widgets.*
mod.widgets.TabStripBase = #(TabStrip::register_widget(vm))
mod.widgets.TabStrip = set_type_default() do mod.widgets.TabStripBase{
width: Fill
height: 36
draw_bg +: {
color: uniform(mod.mpb_theme.darker_background)
pixel: fn() {
return self.color
}
}
draw_tab +: {
color: uniform(mod.mpb_theme.darker_background)
pixel: fn() {
return self.color
}
}
draw_tab_hover +: {
color: uniform(mod.mpb_theme.dark_background)
pixel: fn() {
return self.color
}
}
draw_tab_active +: {
color: uniform(mod.mpb_theme.background)
pixel: fn() {
return self.color
}
}
draw_sep +: {
color: uniform(mod.mpb_theme.muted)
pixel: fn() {
return self.color
}
}
draw_text +: {
color: mod.mpb_theme.foreground
text_style: theme.font_regular{
font_size: 9.5
}
}
draw_text_dim +: {
color: mod.mpb_theme.dark_foreground
text_style: theme.font_regular{
font_size: 9.5
}
}
draw_close +: {
svg: crate_resource("self:resources/icons/close.svg")
color: mod.mpb_theme.dark_foreground
}
draw_close_active +: {
svg: crate_resource("self:resources/icons/close.svg")
color: mod.mpb_theme.foreground
}
draw_plus +: {
svg: crate_resource("self:resources/icons/plus.svg")
color: mod.mpb_theme.foreground
}
draw_globe +: {
svg: crate_resource("self:resources/icons/globe.svg")
color: mod.mpb_theme.dark_foreground
}
}
// A square, flat icon button for the toolbar. A `let` so this block can
// instantiate it below (a `use mod.widgets.*` glob is a snapshot).
let MpToolButton = ButtonFlatterIcon{
width: 32
height: 32
padding: Inset{left: 0 right: 0 top: 0 bottom: 0}
margin: Inset{left: 0 right: 0 top: 0 bottom: 0}
align: Align{x: 0.5 y: 0.5}
icon_walk: Walk{width: 16 height: 16}
draw_icon +: {
color: mod.mpb_theme.foreground
}
draw_bg +: {
border_radius: 0.0
border_size: 0.0
color: #00000000
color_hover: mod.mpb_theme.lighter_background
color_down: mod.mpb_theme.muted
color_focus: #00000000
}
}
mod.widgets.MpToolButton = MpToolButton
// A row of the ⋮ menu.
mod.widgets.MpMenuItem = ButtonFlatter{
width: Fill
height: 30
align: Align{x: 0.0 y: 0.5}
padding: Inset{left: 14 right: 14 top: 0 bottom: 0}
margin: Inset{left: 0 right: 0 top: 0 bottom: 0}
draw_text +: {
color: mod.mpb_theme.foreground
color_hover: mod.mpb_theme.bright_foreground
text_style: theme.font_regular{
font_size: 10
}
}
draw_bg +: {
border_radius: 0.0
border_size: 0.0
color: #00000000
color_hover: mod.mpb_theme.lighter_background
color_down: mod.mpb_theme.muted
}
}
// No `align y: 0.5` anywhere on the omnibox's ancestry: Makepad applies
// such alignment as a deferred shift that moves walked content (text)
// but not `draw_abs` quads — the TextInput's caret and selection would
// end up above the field and clipped. Heights and paddings centre
// everything explicitly instead.
mod.widgets.MpToolbar = SolidView{
width: Fill
height: 40
flow: Right
align: Align{x: 0.0 y: 0.0}
padding: Inset{left: 6 right: 6 top: 4 bottom: 4}
spacing: 2
draw_bg +: {
color: mod.mpb_theme.background
}
back_btn := MpToolButton{
draw_icon.svg: crate_resource("self:resources/icons/back.svg")
}
forward_btn := MpToolButton{
draw_icon.svg: crate_resource("self:resources/icons/forward.svg")
}
reload_btn := MpToolButton{
draw_icon.svg: crate_resource("self:resources/icons/reload.svg")
}
View{width: 4 height: Fit}
// The omnibox: a darker square well with the search glyph, the
// text field and the bookmark star.
omnibox_frame := SolidView{
width: Fill
height: 32
flow: Right
align: Align{x: 0.0 y: 0.0}
padding: Inset{left: 10 right: 2 top: 0 bottom: 0}
spacing: 6
draw_bg +: {
color: mod.mpb_theme.darker_background
}
Icon{
margin: Inset{top: 9 bottom: 0 left: 0 right: 0}
icon_walk: Walk{width: 14 height: 14}
draw_icon +: {
svg: crate_resource("self:resources/icons/search.svg")
color: mod.mpb_theme.dark_foreground
}
}
omnibox := TextInputFlat{
width: Fill
height: 32
empty_text: "Search Google or type a URL"
// The line box is 16.8pt for this font size; text reads as
// centred by its x-height, not its full ink box, so 1pt
// less on top puts the x-height middle on the field's
// middle, level with the search glyph and the star.
padding: Inset{left: 4 right: 4 top: 6.6 bottom: 8.6}
margin: Inset{left: 0 right: 0 top: 0 bottom: 0}
draw_bg +: {
border_radius: 0.0
border_size: 0.0
color: #00000000
color_hover: #00000000
color_focus: #00000000
color_down: #00000000
color_empty: #00000000
border_color: #00000000
border_color_hover: #00000000
border_color_focus: #00000000
border_color_down: #00000000
border_color_empty: #00000000
}
draw_cursor +: {
color: mod.mpb_theme.bright_foreground
}
draw_text +: {
color: mod.mpb_theme.foreground
color_hover: mod.mpb_theme.foreground
color_focus: mod.mpb_theme.bright_foreground
color_empty: mod.mpb_theme.dark_foreground
color_empty_hover: mod.mpb_theme.dark_foreground
text_style: theme.font_regular{
font_size: 10.5
}
}
}
star_btn := MpToolButton{
width: 28
height: 28
margin: Inset{top: 2 bottom: 0 left: 0 right: 0}
icon_walk: Walk{width: 15 height: 15}
draw_icon +: {
svg: crate_resource("self:resources/icons/star.svg")
color: mod.mpb_theme.dark_foreground
}
}
}
View{width: 4 height: Fit}
menu_btn := MpToolButton{
draw_icon.svg: crate_resource("self:resources/icons/menu.svg")
}
}
}
#[derive(Clone, Debug, Default, PartialEq)]
pub enum TabStripAction {
#[default]
None,
Activate(TabId),
Close(TabId),
New,
}
#[derive(Clone, Copy, Debug)]
struct TabHit {
id: TabId,
rect: Rect,
close: Rect,
}
#[derive(Script, ScriptHook, Widget)]
pub struct TabStrip {
#[uid]
uid: WidgetUid,
#[source]
source: ScriptObjectRef,
#[walk]
walk: Walk,
#[layout]
layout: Layout,
#[redraw]
#[live]
draw_bg: DrawQuad,
#[live]
draw_tab: DrawQuad,
#[live]
draw_tab_hover: DrawQuad,
#[live]
draw_tab_active: DrawQuad,
#[live]
draw_sep: DrawQuad,
#[live]
draw_text: DrawText,
#[live]
draw_text_dim: DrawText,
#[live]
draw_close: DrawSvg,
#[live]
draw_close_active: DrawSvg,
#[live]
draw_plus: DrawSvg,
#[live]
draw_globe: DrawSvg,
#[live]
draw_favicon: DrawImage,
#[rust]
tabs: Vec<TabSummary>,
#[rust]
hits: Vec<TabHit>,
#[rust]
plus_rect: Rect,
#[rust]
hover_tab: Option<TabId>,
#[rust]
hover_close: bool,
#[rust]
hover_plus: bool,
}
impl TabStrip {
const TAB_MAX_WIDTH: f64 = 240.0;
const TAB_MIN_WIDTH: f64 = 56.0;
const TOP_GAP: f64 = 6.0;
const LEFT_PAD: f64 = 8.0;
const PLUS_SIZE: f64 = 28.0;
pub fn set_tabs(&mut self, cx: &mut Cx, tabs: Vec<TabSummary>) {
self.tabs = tabs;
self.redraw(cx);
}
fn hit_at(&self, pos: Vec2d) -> (Option<TabId>, bool, bool) {
if self.plus_rect.contains(pos) {
return (None, false, true);
}
for hit in &self.hits {
if hit.rect.contains(pos) {
return (Some(hit.id), hit.close.contains(pos), false);
}
}
(None, false, false)
}
fn update_hover(&mut self, cx: &mut Cx, pos: Option<Vec2d>) {
let (tab, close, plus) = pos.map(|p| self.hit_at(p)).unwrap_or((None, false, false));
if tab != self.hover_tab || close != self.hover_close || plus != self.hover_plus {
self.hover_tab = tab;
self.hover_close = close;
self.hover_plus = plus;
self.redraw(cx);
}
}
}
impl Widget for TabStrip {
fn handle_event(&mut self, cx: &mut Cx, event: &Event, _scope: &mut Scope) {
match event.hits(cx, self.draw_bg.area()) {
Hit::FingerDown(fe) => {
let (tab, close, plus) = self.hit_at(fe.abs);
let middle = fe.mouse_button().map(|b| b.is_middle()).unwrap_or(false);
if plus {
cx.widget_action(self.uid, TabStripAction::New);
} else if let Some(id) = tab {
if close || middle {
cx.widget_action(self.uid, TabStripAction::Close(id));
} else {
cx.widget_action(self.uid, TabStripAction::Activate(id));
}
}
}
Hit::FingerHoverIn(fe) | Hit::FingerHoverOver(fe) => {
self.update_hover(cx, Some(fe.abs));
}
Hit::FingerMove(fe) => {
self.update_hover(cx, Some(fe.abs));
}
Hit::FingerHoverOut(_) => {
self.update_hover(cx, None);
}
_ => {}
}
}
fn draw_walk(&mut self, cx: &mut Cx2d, _scope: &mut Scope, walk: Walk) -> DrawStep {
self.draw_bg.begin(cx, walk, self.layout);
let strip = cx.turtle().rect();
self.hits.clear();
let count = self.tabs.len().max(1) as f64;
let available = (strip.size.x - Self::LEFT_PAD - Self::PLUS_SIZE - 12.0).max(0.0);
let tab_width = (available / count)
.min(Self::TAB_MAX_WIDTH)
.max(Self::TAB_MIN_WIDTH.min(available.max(1.0)));
let tab_height = strip.size.y - Self::TOP_GAP;
let mut x = strip.pos.x + Self::LEFT_PAD;
let y = strip.pos.y + Self::TOP_GAP;
let tabs = self.tabs.clone();
for (i, tab) in tabs.iter().enumerate() {
let rect = Rect {
pos: dvec2(x, y),
size: dvec2(tab_width, tab_height),
};
let hovered = self.hover_tab == Some(tab.id);
if tab.active {
self.draw_tab_active.draw_abs(cx, rect);
} else if hovered {
self.draw_tab_hover.draw_abs(cx, rect);
} else {
self.draw_tab.draw_abs(cx, rect);
// Separator between neighbouring inactive tabs.
let next_active = tabs.get(i + 1).map(|t| t.active).unwrap_or(true);
let next_hovered = tabs
.get(i + 1)
.map(|t| self.hover_tab == Some(t.id))
.unwrap_or(false);
if !next_active && !next_hovered {
self.draw_sep.draw_abs(
cx,
Rect {
pos: dvec2(x + tab_width - 0.5, y + 8.0),
size: dvec2(1.0, tab_height - 16.0),
},
);
}
}
let wide = tab_width >= 96.0;
let show_close = tab.active || hovered || wide;
let icon_size = 16.0;
let icon_y = y + (tab_height - icon_size) * 0.5;
let mut text_x = x + 10.0;
if tab_width >= 72.0 {
let icon_rect = Rect {
pos: dvec2(x + 10.0, icon_y),
size: dvec2(icon_size, icon_size),
};
match &tab.favicon {
Some(favicon) => {
self.draw_favicon.draw_vars.set_texture(0, favicon);
self.draw_favicon.draw_abs(cx, icon_rect);
}
None => {
self.draw_globe.draw_abs(cx, icon_rect);
}
}
text_x += icon_size + 8.0;
}
let close_size = 16.0;
let close_rect = Rect {
pos: dvec2(
x + tab_width - close_size - 8.0,
y + (tab_height - close_size) * 0.5,
),
size: dvec2(close_size, close_size),
};
let text_right = if show_close {
close_rect.pos.x - 6.0
} else {
x + tab_width - 8.0
};
let text_width = (text_right - text_x).max(0.0);
if text_width > 4.0 {
let title = if tab.loading && tab.title.is_empty() {
"Loading…".to_string()
} else {
tab.title.clone()
};
cx.begin_turtle(
Walk {
abs_pos: Some(dvec2(text_x, y)),
width: Size::Fixed(text_width),
height: Size::Fixed(tab_height),
..Walk::default()
},
Layout {
clip_x: true,
clip_y: true,
align: Align { x: 0.0, y: 0.5 },
..Layout::default()
},
);
if tab.active {
self.draw_text
.draw_walk(cx, Walk::fit(), Align::default(), &title);
} else {
self.draw_text_dim
.draw_walk(cx, Walk::fit(), Align::default(), &title);
}
cx.end_turtle();
}
if show_close {
let glyph = Rect {
pos: close_rect.pos + dvec2(3.0, 3.0),
size: dvec2(close_size - 6.0, close_size - 6.0),
};
if tab.active || (hovered && self.hover_close) {
self.draw_close_active.draw_abs(cx, glyph);
} else {
self.draw_close.draw_abs(cx, glyph);
}
}
self.hits.push(TabHit {
id: tab.id,
rect,
close: if show_close { close_rect } else { Rect::default() },
});
x += tab_width;
}
// The new-tab "+" square.
let plus_rect = Rect {
pos: dvec2(x + 4.0, y + (tab_height - Self::PLUS_SIZE) * 0.5),
size: dvec2(Self::PLUS_SIZE, Self::PLUS_SIZE),
};
if self.hover_plus {
self.draw_tab_hover.draw_abs(cx, plus_rect);
}
self.draw_plus.draw_abs(
cx,
Rect {
pos: plus_rect.pos + dvec2(7.0, 7.0),
size: dvec2(Self::PLUS_SIZE - 14.0, Self::PLUS_SIZE - 14.0),
},
);
self.plus_rect = plus_rect;
self.draw_bg.end(cx);
DrawStep::done()
}
}

626
apps/mpbrowser/src/main.rs Normal file
View file

@ -0,0 +1,626 @@
//! mpbrowser: a Chrome-like browser as a plain full-window Makepad app.
//! CEF renders the page (GPU-accelerated into a shared IOSurface texture);
//! every bit of chrome is Makepad. Runs standalone or inside makepad-wm /
//! Studio tiles via the shared --stdin-loop client runtime.
pub use makepad_widgets;
use makepad_cef::BootstrapResult;
use makepad_widgets::*;
mod chrome;
mod tabs;
mod theme;
mod webview;
use chrome::{TabStrip, TabStripAction};
use tabs::TabId;
use std::cell::RefCell;
use std::rc::Rc;
use theme::Palette;
use webview::{WebView, WebViewAction};
script_mod! {
use mod.prelude.widgets.*
use mod.widgets.*
load_all_resources() do #(App::script_component(vm)){
ui: Root{
main_window := Window{
window.inner_size: vec2(1280, 860)
window.title: "mpbrowser"
body +: {
flow: Overlay
View{
width: Fill
height: Fill
flow: Down
tab_strip := TabStrip{}
toolbar := MpToolbar{}
webview := WebView{}
}
// The ⋮ menu: a square panel under the button.
menu_layer := View{
width: Fill
height: Fill
flow: Right
align: Align{x: 1.0 y: 0.0}
padding: Inset{top: 78 right: 6 left: 0 bottom: 0}
menu := SolidView{
visible: false
width: 240
height: Fit
flow: Down
padding: Inset{top: 4 bottom: 4 left: 0 right: 0}
draw_bg +: {
color: mod.mpb_theme.background
}
menu_new_tab := MpMenuItem{text: "New tab"}
menu_close_tab := MpMenuItem{text: "Close tab"}
menu_reload := MpMenuItem{text: "Reload"}
Hr{}
menu_gpu := MpMenuItem{text: "chrome://gpu"}
menu_about := MpMenuItem{text: "About mpbrowser"}
}
}
}
}
}
}
}
thread_local! {
static PALETTE: RefCell<Option<Palette>> = const { RefCell::new(None) };
}
static START: std::sync::OnceLock<std::time::Instant> = std::sync::OnceLock::new();
/// Milliseconds since `main` started (startup measurements).
pub fn uptime_ms() -> u128 {
START.get_or_init(std::time::Instant::now).elapsed().as_millis()
}
/// Microseconds since `main` started (resize tracing).
pub fn uptime_us() -> u128 {
START.get_or_init(std::time::Instant::now).elapsed().as_micros()
}
fn palette() -> Palette {
PALETTE.with(|p| {
p.borrow_mut()
.get_or_insert_with(Palette::current)
.clone()
})
}
/// Cmd shortcuts the browser chrome owns (never forwarded to the page).
pub fn is_app_shortcut(key_event: &KeyEvent) -> bool {
if !key_event.modifiers.logo {
return false;
}
matches!(
key_event.key_code,
KeyCode::KeyT
| KeyCode::KeyW
| KeyCode::KeyL
| KeyCode::KeyR
| KeyCode::KeyN
| KeyCode::LBracket
| KeyCode::RBracket
| KeyCode::Key1
| KeyCode::Key2
| KeyCode::Key3
| KeyCode::Key4
| KeyCode::Key5
| KeyCode::Key6
| KeyCode::Key7
| KeyCode::Key8
| KeyCode::Key9
)
}
/// URLs given on the command line (everything that is not a flag).
fn initial_urls() -> Vec<String> {
let mut urls = Vec::new();
let mut args = std::env::args().skip(1);
while let Some(arg) = args.next() {
if arg == "--cwd" || arg == "--message-format" {
let _ = args.next();
continue;
}
if arg.starts_with("--") {
continue;
}
urls.push(tabs::resolve_omnibox(&arg));
}
urls
}
#[derive(Script, ScriptHook)]
pub struct App {
#[live]
ui: WidgetRef,
#[rust]
omnibox_focused: bool,
#[rust]
shown_url: String,
#[rust]
shown_tab: Option<TabId>,
#[rust]
menu_open: bool,
#[rust]
reported_mode: String,
#[rust]
focus_omnibox_pending: bool,
#[rust]
focus_frame: NextFrame,
#[rust]
focus_retries: u32,
}
impl App {
fn with_webview<R>(&self, cx: &mut Cx, f: impl FnOnce(&mut Cx, &mut WebView) -> R) -> Option<R> {
let webview = self.ui.widget(cx, ids!(webview));
let mut inner = webview.borrow_mut::<WebView>()?;
Some(f(cx, &mut inner))
}
fn new_tab(&mut self, cx: &mut Cx, url: Option<String>) {
let url = url.unwrap_or_else(|| palette().new_tab_url());
let is_ntp = theme::is_new_tab_url(&url);
self.with_webview(cx, |cx, wv| {
wv.new_tab(cx, &url, true);
});
self.refresh_chrome(cx);
if is_ntp {
self.focus_omnibox(cx);
}
}
fn close_active_tab(&mut self, cx: &mut Cx) {
let closed_last = self
.with_webview(cx, |cx, wv| {
if let Some(id) = wv.active_id() {
wv.close_tab(cx, id);
}
wv.tab_count() == 0
})
.unwrap_or(false);
if closed_last {
self.new_tab(cx, None);
} else {
self.refresh_chrome(cx);
}
}
fn focus_omnibox(&mut self, cx: &mut Cx) {
let omnibox = self.ui.text_input(cx, ids!(omnibox));
omnibox.set_key_focus(cx);
omnibox.borrow_mut().map(|mut inner| inner.select_all(cx));
// A focus set while a mouse click is still being dispatched (the +
// button, a tab close) does not survive the rest of that click; the
// key-path (Cmd+L) proves the focus itself works. Re-assert it on the
// next frame.
self.focus_omnibox_pending = true;
self.focus_frame = cx.new_next_frame();
}
fn set_menu_open(&mut self, cx: &mut Cx, open: bool) {
if self.menu_open == open {
return;
}
self.menu_open = open;
self.ui.view(cx, ids!(menu)).set_visible(cx, open);
self.ui.redraw(cx);
}
/// Push the model into the chrome: tab strip, omnibox text (unless the
/// user is typing in it), nav button states, window title.
fn refresh_chrome(&mut self, cx: &mut Cx) {
let (summaries, info) = self
.with_webview(cx, |_cx, wv| (wv.summaries(), wv.active_info()))
.unwrap_or_default();
if let Some(mut strip) = self.ui.widget(cx, ids!(tab_strip)).borrow_mut::<TabStrip>() {
strip.set_tabs(cx, summaries);
}
let shown = if theme::is_new_tab_url(&info.url) {
String::new()
} else {
info.url.clone()
};
// The omnibox follows the page unless the user is typing in it — but
// switching tabs always replaces what it shows.
let tab_changed = info.id != self.shown_tab;
self.shown_tab = info.id;
if shown != self.shown_url || tab_changed {
self.shown_url = shown.clone();
if !self.omnibox_focused || tab_changed {
self.ui.text_input(cx, ids!(omnibox)).set_text(cx, &shown);
}
}
let palette = palette();
let on = theme::parse_hex(&palette.foreground).unwrap_or_default();
let off = theme::parse_hex(&palette.muted).unwrap_or_default();
let mut back = self.ui.button(cx, ids!(back_btn));
let back_color = if info.can_go_back { on } else { off };
script_apply_eval!(cx, back, {
draw_icon +: {
color: #(back_color)
}
});
let mut forward = self.ui.button(cx, ids!(forward_btn));
let forward_color = if info.can_go_forward { on } else { off };
script_apply_eval!(cx, forward, {
draw_icon +: {
color: #(forward_color)
}
});
let title = if info.title.is_empty() {
"mpbrowser".to_string()
} else {
format!("{} — mpbrowser", info.title)
};
self.ui.window(cx, ids!(main_window)).set_title(cx, &title);
if info.render_mode != self.reported_mode && info.render_mode != "None" {
self.reported_mode = info.render_mode.clone();
log!(
"mpbrowser: page rendering is {} (accelerated frames so far: {}, last blit {}us)",
info.render_mode,
info.accelerated_frames,
info.last_blit_micros
);
}
self.ui.redraw(cx);
}
fn navigate_from_omnibox(&mut self, cx: &mut Cx, text: &str) {
let url = tabs::resolve_omnibox(text);
if url.is_empty() {
return;
}
self.with_webview(cx, |cx, wv| {
if wv.tab_count() == 0 {
wv.new_tab(cx, &url, true);
} else {
wv.navigate(cx, &url);
}
});
self.refresh_chrome(cx);
}
fn handle_shortcut(&mut self, cx: &mut Cx, ke: &KeyEvent) -> bool {
// Ctrl+Tab / Ctrl+Shift+Tab cycle tabs (Chrome), as do Cmd+Shift+] / [.
if ke.modifiers.control && ke.key_code == KeyCode::Tab {
let delta = if ke.modifiers.shift { -1 } else { 1 };
self.with_webview(cx, |cx, wv| wv.activate_offset(cx, delta));
self.refresh_chrome(cx);
return true;
}
if !ke.modifiers.logo {
return false;
}
if ke.modifiers.shift
&& matches!(ke.key_code, KeyCode::LBracket | KeyCode::RBracket)
{
let delta = if ke.key_code == KeyCode::LBracket { -1 } else { 1 };
self.with_webview(cx, |cx, wv| wv.activate_offset(cx, delta));
self.refresh_chrome(cx);
return true;
}
match ke.key_code {
KeyCode::KeyT | KeyCode::KeyN => self.new_tab(cx, None),
KeyCode::KeyW => self.close_active_tab(cx),
KeyCode::KeyL => self.focus_omnibox(cx),
KeyCode::KeyR => {
self.with_webview(cx, |cx, wv| wv.reload(cx));
}
KeyCode::LBracket => {
self.with_webview(cx, |cx, wv| wv.go_back(cx));
}
KeyCode::RBracket => {
self.with_webview(cx, |cx, wv| wv.go_forward(cx));
}
KeyCode::Key1
| KeyCode::Key2
| KeyCode::Key3
| KeyCode::Key4
| KeyCode::Key5
| KeyCode::Key6
| KeyCode::Key7
| KeyCode::Key8 => {
let index = match ke.key_code {
KeyCode::Key1 => 0,
KeyCode::Key2 => 1,
KeyCode::Key3 => 2,
KeyCode::Key4 => 3,
KeyCode::Key5 => 4,
KeyCode::Key6 => 5,
KeyCode::Key7 => 6,
_ => 7,
};
self.with_webview(cx, |cx, wv| wv.activate_index(cx, index));
self.refresh_chrome(cx);
}
KeyCode::Key9 => {
self.with_webview(cx, |cx, wv| {
let last = wv.tab_count().saturating_sub(1);
wv.activate_index(cx, last);
});
self.refresh_chrome(cx);
}
_ => return false,
}
true
}
}
impl MatchEvent for App {
fn handle_startup(&mut self, cx: &mut Cx) {
match makepad_cef::startup_phases() {
Some((bundle_ms, exec_gap_ms)) => log!(
"mpbrowser: window up at {} ms after main (app bundle prepared in {} ms, exec-to-main gap {} ms)",
uptime_ms(),
bundle_ms,
exec_gap_ms
),
None => log!("mpbrowser: window up at {} ms after main", uptime_ms()),
}
let urls = initial_urls();
if urls.is_empty() {
// Never boot empty: the first tab opens the web (Cmd+T tabs
// still get the themed New Tab page).
self.new_tab(cx, Some("https://www.google.com/".to_string()));
} else {
for url in urls {
self.new_tab(cx, Some(url));
}
}
}
fn handle_actions(&mut self, cx: &mut Cx, actions: &Actions) {
// Tab strip.
for action in actions {
let Some(action) = action.as_widget_action() else {
continue;
};
match action.cast::<TabStripAction>() {
TabStripAction::Activate(id) => {
self.with_webview(cx, |cx, wv| wv.activate(cx, id));
self.refresh_chrome(cx);
}
TabStripAction::Close(id) => {
let closed_last = self
.with_webview(cx, |cx, wv| {
wv.close_tab(cx, id);
wv.tab_count() == 0
})
.unwrap_or(false);
if closed_last {
self.new_tab(cx, None);
} else {
self.refresh_chrome(cx);
}
}
TabStripAction::New => self.new_tab(cx, None),
TabStripAction::None => {}
}
if let WebViewAction::TabsChanged = action.cast::<WebViewAction>() {
self.refresh_chrome(cx);
}
}
// Toolbar.
if self.ui.button(cx, ids!(back_btn)).clicked(actions) {
self.with_webview(cx, |cx, wv| wv.go_back(cx));
}
if self.ui.button(cx, ids!(forward_btn)).clicked(actions) {
self.with_webview(cx, |cx, wv| wv.go_forward(cx));
}
if self.ui.button(cx, ids!(reload_btn)).clicked(actions) {
// Chrome semantics: the button stops a page that is still loading.
self.with_webview(cx, |cx, wv| {
if wv.active_info().loading {
wv.stop(cx);
} else {
wv.reload(cx);
}
});
}
if self.ui.button(cx, ids!(star_btn)).clicked(actions) {
// Bookmarks are not wired yet; the star just re-focuses the page.
self.with_webview(cx, |cx, wv| wv.focus(cx));
}
if self.ui.button(cx, ids!(menu_btn)).clicked(actions) {
let open = !self.menu_open;
self.set_menu_open(cx, open);
}
// Menu.
if self.ui.button(cx, ids!(menu_new_tab)).clicked(actions) {
self.set_menu_open(cx, false);
self.new_tab(cx, None);
}
if self.ui.button(cx, ids!(menu_close_tab)).clicked(actions) {
self.set_menu_open(cx, false);
self.close_active_tab(cx);
}
if self.ui.button(cx, ids!(menu_reload)).clicked(actions) {
self.set_menu_open(cx, false);
self.with_webview(cx, |cx, wv| wv.reload(cx));
}
if self.ui.button(cx, ids!(menu_gpu)).clicked(actions) {
self.set_menu_open(cx, false);
self.new_tab(cx, Some("chrome://gpu".to_string()));
}
if self.ui.button(cx, ids!(menu_about)).clicked(actions) {
self.set_menu_open(cx, false);
let info = self
.with_webview(cx, |_cx, wv| wv.active_info())
.unwrap_or_default();
let html = format!(
"<!doctype html><title>About mpbrowser</title><body style=\"background:{bg};color:{fg};font:14px -apple-system,Helvetica,sans-serif;padding:32px\">\
<h2 style=\"font-weight:500\">mpbrowser</h2>\
<p>Makepad chrome, Chromium Embedded Framework {cef} page rendering.</p>\
<p>Page rendering path: <b>{mode}</b> (accelerated frames: {frames}, last GPU blit: {blit}µs)</p>\
<p>ANGLE backend: {angle}</p></body>",
bg = palette().darker_background,
fg = palette().foreground,
cef = makepad_cef::CEF_VERSION,
mode = info.render_mode,
frames = info.accelerated_frames,
blit = info.last_blit_micros,
angle = std::env::var("MAKEPAD_CEF_USE_ANGLE").unwrap_or_else(|_| "default".into()),
);
let url = format!("data:text/html;charset=utf-8,{}", theme::percent_encode(&html));
self.new_tab(cx, Some(url));
}
// Omnibox.
let omnibox = self.ui.text_input(cx, ids!(omnibox));
if let Some((text, _modifiers)) = omnibox.returned(actions) {
self.navigate_from_omnibox(cx, &text);
}
if omnibox.escaped(actions) {
let shown = self.shown_url.clone();
omnibox.set_text(cx, &shown);
self.with_webview(cx, |cx, wv| wv.focus(cx));
}
for action in actions {
let Some(widget_action) = action.as_widget_action() else {
continue;
};
if widget_action.widget_uid != omnibox.widget_uid() {
continue;
}
match widget_action.cast::<TextInputAction>() {
TextInputAction::KeyFocus => {
self.omnibox_focused = true;
if let Some(mut inner) = omnibox.borrow_mut() {
inner.select_all(cx);
}
}
TextInputAction::KeyFocusLost => {
if std::env::var_os("MAKEPAD_CEF_DEBUG").is_some() {
let now = cx.keyboard.key_focus();
let webview = self.ui.widget(cx, ids!(webview)).area();
log!(
"omnibox lost key focus to {}",
if now == webview { "the webview" } else if now == Area::Empty { "nothing (Area::Empty)" } else { "another widget" }
);
}
self.omnibox_focused = false;
let shown = self.shown_url.clone();
omnibox.set_text(cx, &shown);
}
_ => {}
}
}
}
}
impl AppMain for App {
fn script_mod(vm: &mut ScriptVm) -> ScriptValue {
crate::makepad_widgets::script_mod(vm);
// The family theme bridge retints the stock widgets from the WM
// theme; the chrome roles go into mod.mpb_theme.
mp_theme::apply(vm);
palette().apply(vm);
chrome::script_mod(vm);
webview::script_mod(vm);
self::script_mod(vm)
}
fn handle_event(&mut self, cx: &mut Cx, event: &Event) {
if let Event::KeyDown(ke) = event {
if self.handle_shortcut(cx, ke) {
return;
}
}
if self.focus_omnibox_pending && self.focus_frame.is_event(event).is_some() {
let omnibox = self.ui.text_input(cx, ids!(omnibox));
if omnibox.area().is_valid(cx) || self.focus_retries >= 60 {
self.focus_omnibox_pending = false;
self.focus_retries = 0;
omnibox.set_key_focus(cx);
omnibox.borrow_mut().map(|mut inner| inner.select_all(cx));
} else {
// At startup the first tab arrives before the chrome has been
// drawn: no area to focus yet, so try again next frame.
self.focus_retries += 1;
self.focus_frame = cx.new_next_frame();
}
}
if let Event::MouseDown(_) = event {
if self.menu_open {
// Any click outside the menu closes it; the menu's own
// buttons still get the event through the ui below.
let menu = self.ui.view(cx, ids!(menu)).area();
if let Event::MouseDown(md) = event {
if !(menu.is_valid(cx) && menu.rect(cx).contains(md.abs)) {
self.set_menu_open(cx, false);
}
}
}
}
self.match_event(cx, event);
self.ui.handle_event(cx, event, &mut Scope::empty());
}
}
fn main() {
app_main();
}
/// The CEF-aware entry point: helper-process bootstrap (`cef_execute_process`),
/// the app-bundle re-exec macOS needs for Chromium's subprocesses, then the
/// ordinary Makepad event loop with the `--remote` control surface.
#[cfg(not(any(target_arch = "wasm32", target_os = "android", target_env = "ohos")))]
pub fn app_main() {
let _ = uptime_ms();
if let Err(err) = makepad_cef::reexec_into_app_bundle_if_needed() {
panic!("CEF bundle re-exec failed: {err}");
}
match makepad_cef::bootstrap() {
Ok(BootstrapResult::Continue) => {}
Ok(BootstrapResult::Exit(code)) => std::process::exit(code),
Err(err) => panic!("CEF bootstrap failed: {err}"),
}
Cx::init_log();
if Cx::pre_start() {
return;
}
// Chromium composites the page on this colour. Left at CEF's default the
// first frames of every page are BLACK — a dark dip then a bright jump on
// open, and a black margin wherever a resize outruns the reflow.
makepad_cef::set_background_color(palette().page_background_argb());
// Only the cheap NSApp/pump preparation here: the window goes up first,
// the WebView runs `cef_initialize` on the frame after it is drawn.
if let Err(err) = makepad_cef::prepare() {
panic!("CEF prepare failed: {err}");
}
let cx = Rc::new(RefCell::new(Cx::new(
makepad_widgets::_app_main_event_closure!(App),
)));
let studio_http = makepad_widgets::resolve_studio_http();
cx.borrow_mut().init_websockets(&studio_http);
if makepad_widgets::should_run_stdin_loop_from_env() {
cx.borrow_mut().in_makepad_studio = true;
}
cx.borrow_mut().init_cx_os();
makepad_widgets::makepad_platform::remote::start_if_requested();
Cx::event_loop(cx.clone());
drop(cx);
makepad_cef::shutdown();
}
#[cfg(any(target_arch = "wasm32", target_os = "android", target_env = "ohos"))]
pub fn app_main() {
panic!("mpbrowser is desktop-only");
}

321
apps/mpbrowser/src/tabs.rs Normal file
View file

@ -0,0 +1,321 @@
//! The tab model. Each tab owns one CEF browser (created lazily when the
//! view first knows its size), the texture that browser paints into, and the
//! navigation state mirrored from CEF's display/load handlers.
use makepad_widgets::*;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct TabId(pub u64);
pub struct Tab {
pub id: TabId,
pub browser: Option<makepad_cef::Browser>,
/// The URL to load once the browser exists.
pub initial_url: String,
pub title: String,
pub url: String,
pub loading: bool,
pub can_go_back: bool,
pub can_go_forward: bool,
/// What the quad samples: an IOSurface-backed texture on the GPU path,
/// a BGRA upload texture on the software path. This is the LAST GOOD
/// frame — it is never dropped for a resize, only replaced once its
/// successor holds a page frame.
pub texture: Option<Texture>,
/// CEF has put at least one frame in `texture`. The GPU path hands the
/// browser an IOSurface up front, so the texture EXISTS long before the
/// page is on it; drawing it meanwhile paints an opaque black hole over
/// the themed ground — which, inside the WM, reads as the window
/// flashing black on the way in. Nothing samples the texture until this
/// is true.
pub painted: bool,
/// Pixel size `texture` is allocated at. The GPU surface is rounded up to
/// a coarse grid so a drag walks many sizes inside one surface.
pub texture_alloc: Option<(usize, usize)>,
/// The sub-rect of `texture` that actually holds page pixels (the last
/// blit's copy region). The quad samples exactly this and stretches it
/// over the current rect, so a frame from a slightly older size shows
/// scaled instead of cropped-with-a-blank-margin.
pub texture_valid: Option<(usize, usize)>,
/// A larger/smaller surface already handed to CEF that has NOT been
/// painted yet. It is not drawn until it holds a frame; `texture` keeps
/// covering the page area meanwhile.
pub pending_texture: Option<Texture>,
pub pending_alloc: Option<(usize, usize)>,
/// Allocation size of the surface CEF is painting into right now.
pub accel_target_size: Option<(usize, usize)>,
/// Page size last handed to `browser.resize`, and when — `was_resized` is
/// rate-limited, because a resize faster than CEF can paint starves the
/// paint callback completely (measured: 0 frames at ~2 kHz).
pub resized_to: Option<(usize, usize)>,
pub resized_at: Option<std::time::Instant>,
pub deferred_resize: Option<(usize, usize, f32)>,
/// Page size the layout last asked for, and when it last changed — the
/// settle detector behind shrinking a surface back down.
pub wanted_size: Option<(usize, usize)>,
pub wanted_at: Option<std::time::Instant>,
pub accel_frame_counter: u64,
pub nav_generation: u64,
pub favicon: Option<Texture>,
pub init_error: Option<String>,
pub render_mode: makepad_cef::RenderMode,
}
impl Tab {
fn new(id: TabId, url: &str) -> Self {
Self {
id,
browser: None,
initial_url: url.to_string(),
title: String::new(),
url: url.to_string(),
loading: true,
can_go_back: false,
can_go_forward: false,
texture: None,
painted: false,
texture_alloc: None,
texture_valid: None,
pending_texture: None,
pending_alloc: None,
accel_target_size: None,
resized_to: None,
resized_at: None,
deferred_resize: None,
wanted_size: None,
wanted_at: None,
accel_frame_counter: 0,
nav_generation: 0,
favicon: None,
init_error: None,
render_mode: makepad_cef::RenderMode::None,
}
}
/// Title for the strip: the page title, else the host, else "New Tab".
pub fn display_title(&self) -> String {
if !self.title.trim().is_empty() {
return self.title.clone();
}
if crate::theme::is_new_tab_url(&self.url) || self.url.is_empty() {
return "New Tab".to_string();
}
host_of(&self.url).unwrap_or_else(|| self.url.clone())
}
}
/// What the tab strip needs to draw one tab.
#[derive(Clone, Debug)]
pub struct TabSummary {
pub id: TabId,
pub title: String,
pub loading: bool,
pub active: bool,
pub favicon: Option<Texture>,
}
#[derive(Default)]
pub struct TabModel {
pub tabs: Vec<Tab>,
pub active: usize,
next_id: u64,
}
impl TabModel {
pub fn len(&self) -> usize {
self.tabs.len()
}
pub fn index_of(&self, id: TabId) -> Option<usize> {
self.tabs.iter().position(|t| t.id == id)
}
pub fn active(&self) -> Option<&Tab> {
self.tabs.get(self.active)
}
pub fn active_mut(&mut self) -> Option<&mut Tab> {
self.tabs.get_mut(self.active)
}
pub fn active_id(&self) -> Option<TabId> {
self.active().map(|t| t.id)
}
/// Insert a tab right after the active one (Chrome's placement), or at
/// the end when there is none.
pub fn insert(&mut self, url: &str, activate: bool) -> TabId {
self.next_id += 1;
let id = TabId(self.next_id);
let at = if self.tabs.is_empty() {
0
} else {
(self.active + 1).min(self.tabs.len())
};
self.tabs.insert(at, Tab::new(id, url));
if activate || self.tabs.len() == 1 {
self.active = at;
} else if at <= self.active {
self.active += 1;
}
id
}
/// Remove a tab. Returns the removed tab (dropping it closes its
/// browser) and whether it was the active one.
pub fn remove(&mut self, id: TabId) -> Option<(Tab, bool)> {
let index = self.index_of(id)?;
let was_active = index == self.active;
let tab = self.tabs.remove(index);
if self.tabs.is_empty() {
self.active = 0;
} else if index < self.active {
self.active -= 1;
} else if was_active {
// Chrome activates the tab to the right, else the new last one.
self.active = index.min(self.tabs.len() - 1);
}
Some((tab, was_active))
}
pub fn activate(&mut self, id: TabId) -> bool {
if let Some(index) = self.index_of(id) {
self.active = index;
true
} else {
false
}
}
pub fn activate_offset(&mut self, delta: isize) {
if self.tabs.is_empty() {
return;
}
let len = self.tabs.len() as isize;
let next = ((self.active as isize + delta) % len + len) % len;
self.active = next as usize;
}
pub fn activate_index(&mut self, index: usize) {
if index < self.tabs.len() {
self.active = index;
}
}
pub fn summaries(&self) -> Vec<TabSummary> {
self.tabs
.iter()
.enumerate()
.map(|(i, t)| TabSummary {
id: t.id,
title: t.display_title(),
loading: t.loading,
active: i == self.active,
favicon: t.favicon.clone(),
})
.collect()
}
}
pub fn host_of(url: &str) -> Option<String> {
let rest = url.split_once("://")?.1;
let host = rest.split(['/', '?', '#']).next()?;
if host.is_empty() {
return None;
}
Some(host.trim_start_matches("www.").to_string())
}
/// Turn omnibox input into a URL: keep explicit schemes, prefix `https://`
/// for things that look like hosts, otherwise search.
pub fn resolve_omnibox(input: &str) -> String {
let input = input.trim();
if input.is_empty() {
return String::new();
}
let lower = input.to_ascii_lowercase();
if lower.contains("://")
|| lower.starts_with("about:")
|| lower.starts_with("data:")
|| lower.starts_with("chrome:")
|| lower.starts_with("file:")
|| lower.starts_with("javascript:")
|| lower.starts_with("view-source:")
{
return input.to_string();
}
let has_space = input.contains(char::is_whitespace);
let first = input.split(['/', '?', '#']).next().unwrap_or("");
let (host, port) = match first.rsplit_once(':') {
Some((h, p)) if !p.is_empty() && p.chars().all(|c| c.is_ascii_digit()) => (h, Some(p)),
_ => (first, None),
};
let looks_like_host = !has_space
&& !host.is_empty()
&& (host == "localhost"
|| host.parse::<std::net::IpAddr>().is_ok()
|| (host.contains('.')
&& !host.starts_with('.')
&& !host.ends_with('.')
&& host
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '.')
&& host
.rsplit('.')
.next()
.map(|tld| tld.len() >= 2 && tld.chars().all(|c| c.is_ascii_alphabetic()))
.unwrap_or(false)));
let _ = port;
if looks_like_host {
format!("https://{input}")
} else {
format!(
"https://www.google.com/search?q={}",
crate::theme::percent_encode(input)
)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn omnibox_resolution() {
assert_eq!(resolve_omnibox("makepad.nl"), "https://makepad.nl");
assert_eq!(resolve_omnibox("http://x.org/a b"), "http://x.org/a b");
assert_eq!(resolve_omnibox("localhost:8080/x"), "https://localhost:8080/x");
assert_eq!(
resolve_omnibox("rust async traits"),
"https://www.google.com/search?q=rust%20async%20traits"
);
assert_eq!(
resolve_omnibox("hello"),
"https://www.google.com/search?q=hello"
);
assert_eq!(resolve_omnibox("about:blank"), "about:blank");
}
#[test]
fn model_insert_remove() {
let mut m = TabModel::default();
let a = m.insert("a", true);
let b = m.insert("b", true);
assert_eq!(m.active_id(), Some(b));
let c = m.insert("c", false);
assert_eq!(m.active_id(), Some(b));
assert_eq!(m.index_of(c), Some(2));
m.remove(b);
assert_eq!(m.active_id(), Some(c));
m.remove(c);
assert_eq!(m.active_id(), Some(a));
m.activate_offset(1);
assert_eq!(m.active_id(), Some(a));
}
#[test]
fn hosts() {
assert_eq!(host_of("https://www.makepad.nl/x"), Some("makepad.nl".into()));
assert_eq!(host_of("nope"), None);
}
}

221
apps/mpbrowser/src/theme.rs Normal file
View file

@ -0,0 +1,221 @@
//! The browser-chrome palette. Theming lives in splash: the chrome reads
//! `mod.mpb_theme.*` (tab strip, toolbar, omnibox, icon roles), which this
//! module evaluates into the VM before the UI modules.
//!
//! Under makepad-wm the roles come from the WM's theme.splash
//! (`MPWM_THEME_SPLASH`, line-scanned by `mp_theme`, the family bridge);
//! standalone runs get Chrome's own dark palette.
use makepad_widgets::*;
/// Chrome-dark roles, keyed like the mpwm theme so one mapping serves both.
#[derive(Clone, Debug)]
pub struct Palette {
/// Tab strip background (the "frame").
pub darker_background: String,
/// Active tab + toolbar.
pub background: String,
/// Hovered inactive tab.
pub dark_background: String,
/// Button hover squares, omnibox focus fill.
pub lighter_background: String,
pub foreground: String,
pub dark_foreground: String,
pub bright_foreground: String,
pub muted: String,
pub selection: String,
pub accent: String,
}
impl Palette {
pub fn chrome_dark() -> Self {
Self {
darker_background: "#202124".into(),
background: "#35363a".into(),
dark_background: "#2b2c2f".into(),
lighter_background: "#3c4043".into(),
foreground: "#e8eaed".into(),
dark_foreground: "#9aa0a6".into(),
bright_foreground: "#ffffff".into(),
muted: "#5f6368".into(),
selection: "#264f78".into(),
accent: "#8ab4f8".into(),
}
}
/// The WM palette when mpwm exported one, else Chrome dark.
pub fn current() -> Self {
let fallback = Self::chrome_dark();
let Some(p) = mp_theme::current() else {
return fallback;
};
Self {
darker_background: p.hex("darker_background", &fallback.darker_background),
background: p.hex("background", &fallback.background),
dark_background: p.hex("dark_background", &fallback.dark_background),
lighter_background: p.hex("lighter_background", &fallback.lighter_background),
foreground: p.hex("foreground", &fallback.foreground),
dark_foreground: p.hex("dark_foreground", &fallback.dark_foreground),
bright_foreground: p.hex("bright_foreground", &fallback.bright_foreground),
muted: p.hex("muted", &fallback.muted),
selection: p.hex("selection", &fallback.selection),
accent: p.hex("accent", &fallback.accent),
}
}
/// The `mod.mpb_theme = {...}` splash source. Runtime-evaluated, so plain
/// `#hex` (the `#x` escape is a proc-macro-only hazard).
pub fn splash_source(&self) -> String {
format!(
"mod.mpb_theme = {{\n\
\x20 darker_background: {}\n\
\x20 background: {}\n\
\x20 dark_background: {}\n\
\x20 lighter_background: {}\n\
\x20 foreground: {}\n\
\x20 dark_foreground: {}\n\
\x20 bright_foreground: {}\n\
\x20 muted: {}\n\
\x20 selection: {}\n\
\x20 accent: {}\n\
}}\n\
true\n",
self.darker_background,
self.background,
self.dark_background,
self.lighter_background,
self.foreground,
self.dark_foreground,
self.bright_foreground,
self.muted,
self.selection,
self.accent,
)
}
/// Evaluate `mod.mpb_theme` into the VM. Call after
/// `makepad_widgets::script_mod(vm)` and before the chrome modules.
pub fn apply(&self, vm: &mut ScriptVm) {
let script_mod_id = ScriptMod {
cargo_manifest_path: env!("CARGO_MANIFEST_DIR").to_string(),
module_path: "mpb_theme".to_string(),
file: "mpb_theme.splash".to_string(),
line: 0,
column: 0,
code: self.splash_source(),
values: vec![],
};
vm.eval(script_mod_id);
for e in vm.take_errors() {
log!("mpbrowser theme: {}", e);
}
}
/// What CEF fills the page with before the site has composited anything,
/// and behind area a reflow has not reached yet. The same colour as the
/// ground the page quad sits on, so neither a cold tab nor a mid-resize
/// page shows a black hole.
pub fn page_background_argb(&self) -> u32 {
parse_argb(&self.darker_background).unwrap_or(0xff20_2124)
}
/// The new-tab page: a data URL in the theme's colours, so a fresh tab
/// never flashes white.
pub fn new_tab_url(&self) -> String {
let html = format!(
"<!doctype html><html><head><meta charset=utf-8><title>New Tab</title>\
<style>html,body{{margin:0;height:100%;background:{bg};color:{fg};\
font:14px -apple-system,Helvetica,Arial,sans-serif}}\
.c{{display:flex;height:100%;align-items:center;justify-content:center;\
flex-direction:column;gap:10px}}.n{{font-size:28px;letter-spacing:1px;color:{fgb}}}\
.s{{color:{fgd}}}</style></head><body><div class=c>\
<div class=n>mpbrowser</div><div class=s>Type a URL or search in the box above</div>\
</div></body></html>",
bg = self.darker_background,
fg = self.foreground,
fgb = self.bright_foreground,
fgd = self.dark_foreground,
);
format!("data:text/html;charset=utf-8,{}", percent_encode(&html))
}
}
pub fn parse_hex(s: &str) -> Option<Vec4f> {
let s = s.trim().trim_start_matches('#');
if s.len() != 6 {
return None;
}
let r = u8::from_str_radix(&s[0..2], 16).ok()?;
let g = u8::from_str_radix(&s[2..4], 16).ok()?;
let b = u8::from_str_radix(&s[4..6], 16).ok()?;
Some(vec4(
r as f32 / 255.0,
g as f32 / 255.0,
b as f32 / 255.0,
1.0,
))
}
/// A `#rrggbb` role as opaque ARGB (`0xFFRRGGBB`) — the form CEF wants for
/// `cef_browser_settings_t::background_color`.
pub fn parse_argb(s: &str) -> Option<u32> {
let s = s.trim().trim_start_matches('#');
if s.len() != 6 {
return None;
}
let rgb = u32::from_str_radix(s, 16).ok()?;
Some(0xff00_0000 | rgb)
}
/// Minimal percent-encoding for a `data:` URL payload.
pub fn percent_encode(s: &str) -> String {
let mut out = String::with_capacity(s.len() * 3);
for b in s.bytes() {
match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
out.push(b as char)
}
_ => out.push_str(&format!("%{:02X}", b)),
}
}
out
}
/// Is this new-tab-page URL (never shown in the omnibox)?
pub fn is_new_tab_url(url: &str) -> bool {
url.starts_with("data:text/html;charset=utf-8,%3C%21doctype%20html%3E%3Chtml%3E%3Chead%3E%3Cmeta%20charset%3Dutf-8%3E%3Ctitle%3ENew%20Tab")
|| url == "about:blank"
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn new_tab_url_is_recognised() {
let p = Palette::chrome_dark();
assert!(is_new_tab_url(&p.new_tab_url()));
assert!(!is_new_tab_url("https://makepad.nl"));
}
#[test]
fn hex_parses() {
let c = parse_hex("#8ab4f8").unwrap();
assert!((c.x - 0x8a as f32 / 255.0).abs() < 1e-6);
assert!(parse_hex("#12345").is_none());
}
#[test]
fn argb_is_opaque() {
// CEF paints the page on this before the site composites; a
// transparent (or zero) colour is what makes a page open black.
assert_eq!(parse_argb("#202124"), Some(0xff20_2124));
assert_eq!(parse_argb("202124"), Some(0xff20_2124));
assert_eq!(parse_argb("#nope"), None);
assert_eq!(
Palette::chrome_dark().page_background_argb(),
0xff20_2124,
"the CEF fill must match the themed ground the page sits on"
);
}
}

View file

@ -0,0 +1,966 @@
//! The page area. One widget hosts every tab's CEF browser; the active
//! tab's texture is what gets drawn, input goes to the active browser,
//! background tabs are told they are hidden so they stop painting.
//!
//! Rendering: with `shared_texture_enabled` CEF paints on the GPU into pooled
//! IOSurfaces and `libs/cef` blits each one into a Makepad-owned IOSurface
//! texture (`Cx::create_iosurface_render_texture`) that the quad samples —
//! no CPU readback anywhere. The classic `on_paint` BGRA upload stays as the
//! fallback (`MAKEPAD_CEF_SOFTWARE=1`, or a CEF build without GPU paint).
use crate::tabs::{TabId, TabModel, TabSummary};
// The surface policy is the stock Browser widget's — one source of truth for
// how a CEF page survives a resize.
use makepad_widgets::browser::{
needs_new_surface, surface_alloc, Browser as BrowserKeys, RESIZE_INTERVAL, SETTLE,
};
use makepad_widgets::image::DrawImage;
use makepad_widgets::*;
script_mod! {
use mod.prelude.widgets_internal.*
use mod.widgets.*
mod.widgets.WebViewBase = #(WebView::register_widget(vm))
mod.widgets.WebView = set_type_default() do mod.widgets.WebViewBase{
width: Fill
height: Fill
draw_empty +: {
color: uniform(mod.mpb_theme.darker_background)
pixel: fn() {
return self.color
}
}
draw_status +: {
color: mod.mpb_theme.dark_foreground
text_style: theme.font_regular{
font_size: 11
}
}
}
}
#[derive(Clone, Debug, Default, PartialEq)]
pub enum WebViewAction {
#[default]
None,
/// Titles / URLs / loading state / favicons / the tab list changed.
TabsChanged,
}
/// What the chrome shows for the active tab.
#[derive(Clone, Debug, Default)]
pub struct ActiveInfo {
pub id: Option<TabId>,
pub url: String,
pub title: String,
pub loading: bool,
pub can_go_back: bool,
pub can_go_forward: bool,
pub render_mode: String,
pub accelerated_frames: u64,
pub last_blit_micros: u64,
}
#[derive(Script, ScriptHook, Widget)]
pub struct WebView {
#[uid]
uid: WidgetUid,
#[source]
source: ScriptObjectRef,
#[walk]
walk: Walk,
#[redraw]
#[live]
draw_bg: DrawImage,
#[live]
draw_empty: DrawQuad,
#[live]
draw_status: DrawText,
#[rust]
tabs: TabModel,
/// `cef_initialize` runs on the frame after the chrome first drew, so
/// the window is up before Chromium's processes spawn.
#[rust]
cef_ready: bool,
#[rust]
cef_init_frame: NextFrame,
#[rust]
cef_init_requested: bool,
#[rust]
cef_init_error: Option<String>,
#[rust]
first_frame_logged: bool,
#[rust]
pump_timer: Timer,
#[rust]
pressed_buttons: MouseButton,
#[rust]
suppress_next_paste_shortcut: bool,
#[rust]
pump_started: bool,
}
/// Env-gated resize tracing (`MPB_TRACE=1`): timestamps + sizes on every
/// draw, resize and target swap. Debug rig — not for committing.
pub fn trace_on() -> bool {
static ON: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*ON.get_or_init(|| std::env::var_os("MPB_TRACE").is_some())
}
macro_rules! trace {
($($arg:tt)*) => {
if crate::webview::trace_on() {
eprintln!("[mpb-trace {:>7}us] {}", crate::uptime_us(), format!($($arg)*));
}
};
}
impl WebView {
const PUMP_INTERVAL: f64 = 1.0 / 120.0;
// ------------------------------------------------------------------
// Tab operations (called by the app from chrome actions / shortcuts)
// ------------------------------------------------------------------
pub fn new_tab(&mut self, cx: &mut Cx, url: &str, activate: bool) -> TabId {
let id = self.tabs.insert(url, activate);
if activate {
self.apply_active_visibility();
self.focus(cx);
}
self.notify(cx);
id
}
pub fn close_tab(&mut self, cx: &mut Cx, id: TabId) {
if let Some((tab, _was_active)) = self.tabs.remove(id) {
// Dropping the tab closes its browser.
drop(tab);
}
self.apply_active_visibility();
self.notify(cx);
}
pub fn activate(&mut self, cx: &mut Cx, id: TabId) {
if self.tabs.activate(id) {
self.apply_active_visibility();
self.focus(cx);
self.notify(cx);
}
}
pub fn activate_offset(&mut self, cx: &mut Cx, delta: isize) {
self.tabs.activate_offset(delta);
self.apply_active_visibility();
self.focus(cx);
self.notify(cx);
}
pub fn activate_index(&mut self, cx: &mut Cx, index: usize) {
self.tabs.activate_index(index);
self.apply_active_visibility();
self.focus(cx);
self.notify(cx);
}
pub fn active_id(&self) -> Option<TabId> {
self.tabs.active_id()
}
pub fn tab_count(&self) -> usize {
self.tabs.len()
}
pub fn summaries(&self) -> Vec<TabSummary> {
self.tabs.summaries()
}
pub fn active_info(&self) -> ActiveInfo {
let Some(tab) = self.tabs.active() else {
return ActiveInfo::default();
};
let (accelerated_frames, last_blit_micros) = tab
.browser
.as_ref()
.map(|b| {
let s = b.accelerated_stats();
(s.frames, s.last_blit_micros)
})
.unwrap_or((0, 0));
ActiveInfo {
id: Some(tab.id),
url: tab.url.clone(),
title: tab.display_title(),
loading: tab.loading,
can_go_back: tab.can_go_back,
can_go_forward: tab.can_go_forward,
render_mode: format!("{:?}", tab.render_mode),
accelerated_frames,
last_blit_micros,
}
}
pub fn navigate(&mut self, cx: &mut Cx, url: &str) {
if let Some(tab) = self.tabs.active_mut() {
tab.url = url.to_string();
tab.title.clear();
tab.favicon = None;
tab.loading = true;
match &mut tab.browser {
Some(browser) => {
if let Err(err) = browser.set_url(url) {
log!("navigate failed: {err}");
}
}
None => tab.initial_url = url.to_string(),
}
}
self.focus(cx);
self.notify(cx);
}
pub fn go_back(&mut self, cx: &mut Cx) {
if let Some(browser) = self.active_browser() {
let _ = browser.go_back();
}
self.focus(cx);
}
pub fn go_forward(&mut self, cx: &mut Cx) {
if let Some(browser) = self.active_browser() {
let _ = browser.go_forward();
}
self.focus(cx);
}
pub fn reload(&mut self, cx: &mut Cx) {
if let Some(browser) = self.active_browser() {
let _ = browser.reload();
}
self.focus(cx);
}
pub fn stop(&mut self, cx: &mut Cx) {
if let Some(browser) = self.active_browser() {
let _ = browser.stop_load();
}
self.focus(cx);
}
pub fn focus(&mut self, cx: &mut Cx) {
let area = self.draw_bg.area();
if area.is_valid(cx) {
cx.set_key_focus(area);
}
if let Some(browser) = self.active_browser() {
let _ = browser.set_focus(true);
}
}
// ------------------------------------------------------------------
fn notify(&mut self, cx: &mut Cx) {
cx.widget_action(self.uid, WebViewAction::TabsChanged);
self.redraw(cx);
}
fn active_browser(&mut self) -> Option<&mut makepad_cef::Browser> {
self.tabs.active_mut().and_then(|t| t.browser.as_mut())
}
/// Background tabs stop painting; the active one is shown.
fn apply_active_visibility(&mut self) {
let active = self.tabs.active;
for (i, tab) in self.tabs.tabs.iter_mut().enumerate() {
if let Some(browser) = &mut tab.browser {
let _ = browser.set_hidden(i != active);
}
}
}
fn ensure_active_browser(&mut self, cx: &mut Cx, width: usize, height: usize, dpi: f32) {
if !self.cef_ready {
return;
}
let active = self.tabs.active;
let Some(tab) = self.tabs.tabs.get_mut(active) else {
return;
};
if tab.browser.is_none() && tab.init_error.is_none() {
match makepad_cef::Browser::new(&tab.initial_url, width, height, dpi) {
Ok(browser) => {
tab.browser = Some(browser);
}
Err(err) => {
let message = err.to_string();
log!("CEF browser creation failed: {message}");
tab.init_error = Some(message);
}
}
}
if tab.browser.is_none() {
return;
}
if let Some(browser) = &mut tab.browser {
let _ = browser.set_hidden(false);
}
let now = std::time::Instant::now();
if tab.wanted_size != Some((width, height)) {
tab.wanted_size = Some((width, height));
tab.wanted_at = Some(now);
}
Self::sync_browser_size(tab, width, height, dpi, now);
Self::sync_accel_surface(cx, tab, width, height, now);
}
/// Tell CEF the page size, at most once per `RESIZE_INTERVAL`; anything
/// faster is remembered and applied by the next pump, so the final size of
/// a drag always lands.
fn sync_browser_size(
tab: &mut crate::tabs::Tab,
width: usize,
height: usize,
dpi: f32,
now: std::time::Instant,
) {
if tab.resized_to == Some((width, height)) {
// `Browser::resize` no-ops on an unchanged size; this still lets a
// dpi change through.
if let Some(browser) = &mut tab.browser {
if let Err(err) = browser.resize(width, height, dpi) {
log!("CEF resize failed: {err}");
}
}
tab.deferred_resize = None;
return;
}
let due = tab
.resized_at
.is_none_or(|last| now.duration_since(last) >= RESIZE_INTERVAL);
if !due {
tab.deferred_resize = Some((width, height, dpi));
return;
}
if let Some(browser) = &mut tab.browser {
if let Err(err) = browser.resize(width, height, dpi) {
log!("CEF resize failed: {err}");
}
}
trace!("was_resized {}x{}", width, height);
tab.resized_to = Some((width, height));
tab.resized_at = Some(now);
tab.deferred_resize = None;
}
/// GPU path: hand the browser a Makepad-owned IOSurface to copy into.
///
/// The surface is over-allocated (grid-rounded) and grows only when the
/// page outgrows it; it shrinks only once the size has settled. A new
/// surface is BLANK, so it goes to `pending_texture` and the last good one
/// keeps being drawn until CEF has put a frame in the new one — that is
/// what keeps a drag-resize from flashing the themed ground.
fn sync_accel_surface(
cx: &mut Cx,
tab: &mut crate::tabs::Tab,
width: usize,
height: usize,
now: std::time::Instant,
) {
#[cfg(target_os = "macos")]
{
if !tab.browser.as_ref().is_some_and(|b| b.is_accelerated()) {
return;
}
let want = surface_alloc(width, height);
let settled = tab
.wanted_at
.is_some_and(|since| now.duration_since(since) >= SETTLE);
if !needs_new_surface(tab.accel_target_size, tab.pending_alloc, want, settled) {
return;
}
trace!(
"surface {:?} -> {}x{} for page {}x{}",
tab.accel_target_size,
want.0,
want.1,
width,
height
);
let (texture, iosurface, _id) = cx.create_iosurface_render_texture(want.0, want.1);
let Some(browser) = &mut tab.browser else {
return;
};
match browser.set_accelerated_target(iosurface, want.0, want.1) {
Ok(()) => {
tab.accel_target_size = Some(want);
if tab.painted && tab.texture.is_some() {
// Keep showing the last good frame until this one has
// one of its own.
tab.pending_texture = Some(texture);
tab.pending_alloc = Some(want);
} else {
tab.texture = Some(texture);
tab.texture_alloc = Some(want);
tab.texture_valid = None;
tab.pending_texture = None;
tab.pending_alloc = None;
}
}
Err(err) => {
log!("accelerated target failed, software frames only: {err}");
tab.accel_target_size = None;
tab.pending_texture = None;
tab.pending_alloc = None;
}
}
}
#[cfg(not(target_os = "macos"))]
{
let _ = (cx, tab, width, height, now);
}
}
fn apply_software_frame(cx: &mut Cx, tab: &mut crate::tabs::Tab, frame: makepad_cef::Frame) {
let size = (frame.width, frame.height);
match &tab.texture {
Some(texture)
if tab.accel_target_size.is_none()
&& texture.get_format(cx).vec_width_height() == Some(size) =>
{
texture.set_data_u32(cx, frame.width, frame.height, frame.pixels);
}
_ => {
tab.accel_target_size = None;
tab.pending_texture = None;
tab.pending_alloc = None;
tab.texture = Some(Texture::new_with_format(
cx,
TextureFormat::VecBGRAu8_32 {
data: Some(frame.pixels),
width: frame.width,
height: frame.height,
updated: TextureUpdated::Full,
},
));
}
}
// An upload texture is exactly the page: the whole thing is valid.
tab.texture_alloc = Some(size);
tab.texture_valid = Some(size);
tab.painted = true;
}
fn favicon_texture(cx: &mut Cx, frame: makepad_cef::Frame) -> Texture {
Texture::new_with_format(
cx,
TextureFormat::VecBGRAu8_32 {
data: Some(frame.pixels),
width: frame.width,
height: frame.height,
updated: TextureUpdated::Full,
},
)
}
/// One pump: run CEF's message loop, collect frames and navigation
/// changes from every tab, open queued popups as tabs.
fn pump(&mut self, cx: &mut Cx) {
makepad_cef::do_message_loop_work();
let active = self.tabs.active;
let mut changed = false;
let mut redraw = false;
let mut popups = Vec::new();
for (i, tab) in self.tabs.tabs.iter_mut().enumerate() {
let Some(browser) = &mut tab.browser else {
continue;
};
let mut latest = None;
while let Some(frame) = browser.take_frame() {
latest = Some(frame);
}
let counter = browser.accelerated_frame_counter();
if counter != tab.accel_frame_counter {
let stats = browser.accelerated_stats();
let valid = (stats.last_copy_width, stats.last_copy_height);
if stats.target_frames > 0 && valid.0 > 0 && valid.1 > 0 {
// The surface CEF is painting into now holds a real frame:
// if it was the pending one, this is the moment to swap —
// never before, so the page area never goes blank.
if let (Some(texture), Some(alloc)) =
(tab.pending_texture.take(), tab.pending_alloc.take())
{
trace!(
"promote pending {}x{} (valid {}x{})",
alloc.0,
alloc.1,
valid.0,
valid.1
);
tab.texture = Some(texture);
tab.texture_alloc = Some(alloc);
}
tab.texture_valid = Some(valid);
tab.painted = true;
}
if i == active {
trace!(
"paint landed {}x{} (+{} frames, surface {:?})",
stats.last_width,
stats.last_height,
counter - tab.accel_frame_counter,
tab.accel_target_size
);
}
tab.accel_frame_counter = counter;
if i == active {
redraw = true;
}
if !self.first_frame_logged {
self.first_frame_logged = true;
log!("mpbrowser: first page frame at {} ms", crate::uptime_ms());
}
}
let generation = browser.nav_generation();
if generation != tab.nav_generation {
tab.nav_generation = generation;
tab.title = browser.title();
let url = browser.url();
if !url.is_empty() {
tab.url = url;
}
tab.loading = browser.is_loading();
tab.can_go_back = browser.can_go_back();
tab.can_go_forward = browser.can_go_forward();
if let Some(favicon) = browser.take_favicon() {
tab.favicon = Some(Self::favicon_texture(cx, favicon));
}
popups.extend(browser.take_popup_requests());
changed = true;
}
let mode = browser.render_mode();
if mode != tab.render_mode {
tab.render_mode = mode;
log!("tab {:?} render mode: {:?}", tab.id, mode);
changed = true;
}
if let Some(frame) = latest {
Self::apply_software_frame(cx, tab, frame);
if i == active {
redraw = true;
}
}
// A resize that arrived faster than CEF can take them: apply the
// last one now, so the end of a drag always reaches the page.
if let Some((w, h, dpi)) = tab.deferred_resize {
Self::sync_browser_size(tab, w, h, dpi, std::time::Instant::now());
if i == active {
redraw = true;
}
}
}
for url in popups {
self.tabs.insert(&url, true);
self.apply_active_visibility();
changed = true;
redraw = true;
}
if changed {
cx.widget_action(self.uid, WebViewAction::TabsChanged);
}
if redraw {
self.redraw(cx);
}
}
// ------------------------------------------------------------------
// Input routing (same mapping as the stock Browser widget)
// ------------------------------------------------------------------
fn browser_rect(&self, cx: &mut Cx) -> Option<Rect> {
let area = self.draw_bg.area();
if area.is_valid(cx) {
Some(area.rect(cx))
} else {
None
}
}
/// CEF takes mouse coordinates in view points (it applies the device
/// scale factor itself).
fn cef_position(&self, cx: &mut Cx, abs: Vec2d) -> Option<(i32, i32)> {
let rect = self.browser_rect(cx)?;
let local = abs - rect.pos;
Some((local.x.round() as i32, local.y.round() as i32))
}
fn send_mouse_move(&mut self, cx: &mut Cx, abs: Vec2d, modifiers: KeyModifiers, leave: bool) {
let Some((x, y)) = self.cef_position(cx, abs) else {
return;
};
let m = BrowserKeys::cef_modifiers(modifiers, self.pressed_buttons);
if let Some(browser) = self.active_browser() {
let _ = browser.send_mouse_move(x, y, m, leave);
}
}
fn send_mouse_click(
&mut self,
cx: &mut Cx,
abs: Vec2d,
modifiers: KeyModifiers,
button: Option<MouseButton>,
mouse_up: bool,
click_count: i32,
) {
let Some((x, y)) = self.cef_position(cx, abs) else {
return;
};
let m = BrowserKeys::cef_modifiers(modifiers, self.pressed_buttons);
let b = BrowserKeys::cef_mouse_button(button);
if let Some(browser) = self.active_browser() {
let _ = browser.send_mouse_click(x, y, m, b, mouse_up, click_count.max(1));
}
}
fn send_mouse_wheel(&mut self, cx: &mut Cx, abs: Vec2d, modifiers: KeyModifiers, delta: Vec2d) {
let Some((x, y)) = self.cef_position(cx, abs) else {
return;
};
let m = BrowserKeys::cef_modifiers(modifiers, self.pressed_buttons)
| makepad_cef::EVENTFLAG_PRECISION_SCROLLING_DELTA;
if let Some(browser) = self.active_browser() {
let _ = browser.send_mouse_wheel(x, y, m, delta.x.round() as i32, delta.y.round() as i32);
}
}
fn send_key(&mut self, key_event: &KeyEvent, event_type: i32) {
let modifiers = BrowserKeys::key_event_modifiers(key_event);
let windows_key_code = BrowserKeys::windows_key_code(key_event.key_code);
let character = if key_event.modifiers.control
|| key_event.modifiers.alt
|| key_event.modifiers.logo
{
0
} else {
BrowserKeys::key_char(key_event.key_code, key_event.modifiers.shift)
.map(|ch| ch as u16)
.unwrap_or(0)
};
let send_char = event_type == makepad_cef::KEY_EVENT_KEYDOWN
&& character != 0
&& !key_event.modifiers.control
&& !key_event.modifiers.alt
&& !key_event.modifiers.logo
&& BrowserKeys::sends_char_on_keydown(key_event.key_code);
if let Some(browser) = self.active_browser() {
let _ = browser.send_key_event(
event_type,
modifiers,
windows_key_code,
windows_key_code,
character,
character,
false,
);
if send_char {
let _ = browser.send_key_event(
makepad_cef::KEY_EVENT_CHAR,
modifiers,
windows_key_code,
windows_key_code,
character,
character,
false,
);
}
}
}
fn update_ime_spot(&self, cx: &mut Cx, pos: Vec2d) {
let area = self.draw_bg.area();
if area.is_valid(cx) {
cx.show_text_ime(area, pos);
}
}
}
impl Widget for WebView {
fn handle_event(&mut self, cx: &mut Cx, event: &Event, _scope: &mut Scope) {
if let Event::Startup = event {
if !self.pump_started {
self.pump_started = true;
self.pump_timer = cx.start_interval(Self::PUMP_INTERVAL);
}
}
if let Event::Shutdown = event {
self.tabs.tabs.clear();
return;
}
if !self.cef_ready && self.cef_init_frame.is_event(event).is_some() {
// The chrome has been drawn: bring Chromium up now.
match makepad_cef::initialize() {
Ok(()) => {
self.cef_ready = true;
log!(
"mpbrowser: CEF {} initialized at {} ms",
makepad_cef::CEF_VERSION,
crate::uptime_ms()
);
}
Err(err) => {
let message = err.to_string();
log!("CEF initialize failed: {message}");
self.cef_init_error = Some(message);
}
}
self.redraw(cx);
}
if self.pump_timer.is_event(event).is_some() && self.cef_ready {
self.pump(cx);
}
match event.hits_with_capture_overload(cx, self.draw_bg.area(), true) {
Hit::KeyFocus(_) => {
if let Some(browser) = self.active_browser() {
let _ = browser.set_focus(true);
}
if let Some(rect) = self.browser_rect(cx) {
self.update_ime_spot(cx, rect.pos);
}
}
Hit::KeyFocusLost(_) => {
if let Some(browser) = self.active_browser() {
let _ = browser.set_focus(false);
}
cx.hide_text_ime();
self.suppress_next_paste_shortcut = false;
}
Hit::FingerDown(fe) => {
let button = fe.mouse_button().unwrap_or(MouseButton::PRIMARY);
self.pressed_buttons.insert(button);
cx.set_key_focus(self.draw_bg.area());
if let Some(browser) = self.active_browser() {
let _ = browser.set_focus(true);
}
self.update_ime_spot(cx, fe.abs);
self.send_mouse_move(cx, fe.abs, fe.modifiers, false);
self.send_mouse_click(
cx,
fe.abs,
fe.modifiers,
Some(button),
false,
fe.tap_count as i32,
);
}
Hit::FingerMove(fe) => {
self.send_mouse_move(cx, fe.abs, fe.modifiers, false);
}
Hit::FingerUp(fe) => {
let button = fe.mouse_button().unwrap_or(MouseButton::PRIMARY);
self.send_mouse_move(cx, fe.abs, fe.modifiers, false);
self.send_mouse_click(
cx,
fe.abs,
fe.modifiers,
Some(button),
true,
fe.tap_count as i32,
);
self.pressed_buttons.remove(button);
}
Hit::FingerHoverIn(fe) | Hit::FingerHoverOver(fe) => {
self.send_mouse_move(cx, fe.abs, fe.modifiers, false);
}
Hit::FingerHoverOut(fe) => {
self.send_mouse_move(cx, fe.abs, fe.modifiers, true);
}
Hit::FingerScroll(fe) => {
self.send_mouse_wheel(cx, fe.abs, fe.modifiers, fe.scroll);
}
Hit::KeyDown(key_event) => {
// Browser-level shortcuts (Cmd+T/W/L/R/[ ]) are the app's;
// they never reach the page.
if key_event.modifiers.logo && crate::is_app_shortcut(&key_event) {
return;
}
if self.suppress_next_paste_shortcut
&& key_event.key_code == KeyCode::KeyV
&& key_event.modifiers.is_primary()
{
self.suppress_next_paste_shortcut = false;
} else {
self.send_key(&key_event, makepad_cef::KEY_EVENT_KEYDOWN);
}
}
Hit::KeyUp(key_event) => {
if key_event.modifiers.logo && crate::is_app_shortcut(&key_event) {
return;
}
self.send_key(&key_event, makepad_cef::KEY_EVENT_KEYUP);
}
Hit::TextInput(text_event) => {
let ime_pos = self
.browser_rect(cx)
.map(|rect| rect.pos)
.unwrap_or_default();
self.update_ime_spot(cx, ime_pos);
if text_event.was_paste {
self.suppress_next_paste_shortcut = true;
}
let modifiers = BrowserKeys::cef_modifiers(cx.keyboard.modifiers(), MouseButton::empty());
let char_data = BrowserKeys::char_event_data(&text_event.input);
if let Some(browser) = self.active_browser() {
if text_event.was_paste || text_event.replace_last || char_data.is_none() {
let _ = browser.ime_commit_text(&text_event.input);
} else if let Some((windows_key_code, character)) = char_data {
let _ = browser.send_key_event(
makepad_cef::KEY_EVENT_CHAR,
modifiers,
windows_key_code,
windows_key_code,
character,
character,
false,
);
}
}
}
_ => {}
}
}
fn draw_walk(&mut self, cx: &mut Cx2d, _scope: &mut Scope, walk: Walk) -> DrawStep {
let rect = cx.peek_walk_turtle(walk);
let dpi = cx.current_dpi_factor() as f32;
let width = (rect.size.x.max(1.0) * dpi as f64).round().max(1.0) as usize;
let height = (rect.size.y.max(1.0) * dpi as f64).round().max(1.0) as usize;
self.ensure_active_browser(cx, width, height, dpi);
let shown = self.tabs.active().filter(|t| t.painted).and_then(|t| {
t.texture
.clone()
.map(|texture| (texture, t.texture_alloc, t.texture_valid))
});
// Themed ground under the page, so a tab that has not painted yet
// (or a transparent page) never shows black.
self.draw_empty.draw_abs(cx, rect);
if !self.cef_ready {
let status = match &self.cef_init_error {
Some(err) => format!("browser engine failed to start: {err}"),
None => "starting browser engine…".to_string(),
};
self.draw_status
.draw_abs(cx, rect.pos + dvec2(16.0, 14.0), &status);
if !self.cef_init_requested && self.cef_init_error.is_none() {
self.cef_init_requested = true;
self.cef_init_frame = cx.new_next_frame();
}
}
match shown {
Some((texture, alloc, valid)) => {
// Sample only the part of the surface that holds page pixels
// and stretch it over the current rect: mid-drag that is the
// last good frame at a slightly older size (a fraction of a
// percent of scale), never a blank margin.
let scale = match (alloc, valid) {
(Some((aw, ah)), Some((vw, vh))) if aw > 0 && ah > 0 && vw > 0 && vh > 0 => {
vec2(
(vw as f32 / aw as f32).min(1.0),
(vh as f32 / ah as f32).min(1.0),
)
}
_ => vec2(1.0, 1.0),
};
self.draw_bg.image_scale = scale;
self.draw_bg.draw_vars.set_texture(0, &texture);
self.draw_bg.opacity = 1.0;
}
None => {
self.draw_bg.image_scale = vec2(1.0, 1.0);
self.draw_bg.draw_vars.empty_texture(0);
self.draw_bg.opacity = 0.0;
}
}
if let Some(tab) = self.tabs.active() {
trace!(
"draw rect {}x{} px, surface {:?}, valid {:?}, pending {:?}, painted={}",
width,
height,
tab.accel_target_size,
tab.texture_valid,
tab.pending_alloc,
tab.painted
);
}
self.draw_bg.draw_walk(cx, walk);
cx.add_nav_stop(self.draw_bg.area(), NavRole::TextInput, Inset::default());
DrawStep::done()
}
}
#[cfg(test)]
mod tests {
use super::*;
use makepad_widgets::browser::SURFACE_GRID;
#[test]
fn surface_is_grid_rounded() {
assert_eq!(surface_alloc(1280, 860), (1280, 1024));
assert_eq!(surface_alloc(1281, 1025), (1536, 1280));
assert_eq!(surface_alloc(0, 0), (SURFACE_GRID, SURFACE_GRID));
}
/// The whole point of the grid: a drag walks many page sizes inside one
/// surface instead of handing CEF a blank one at every step (which is
/// what made the page area flash the themed ground for a whole drag).
#[test]
fn a_drag_walks_many_sizes_per_surface() {
let mut current = Some(surface_alloc(1600, 900));
let mut steps = 0;
let mut swaps = 0;
for w in (1600..2600).step_by(8) {
let want = surface_alloc(w, 900);
steps += 1;
if needs_new_surface(current, None, want, false) {
swaps += 1;
current = Some(want);
}
}
assert_eq!(steps, 125);
// 1600 -> 2592 crosses the 1792 / 2048 / 2304 / 2560 lines: four
// surfaces for 125 drag steps, instead of 125.
assert_eq!(swaps, 4);
}
#[test]
fn surface_grows_at_once_and_shrinks_only_when_settled() {
let current = Some((1536, 1024));
// Outgrowing the surface would clip the blit: no waiting.
assert!(needs_new_surface(current, None, (1792, 1024), false));
// Smaller page, still moving: keep the surface (and the frame in it).
assert!(!needs_new_surface(current, None, (1024, 1024), false));
// Settled: hand back the slack.
assert!(needs_new_surface(current, None, (1024, 1024), true));
// A bigger surface is already on its way; do not queue another.
assert!(!needs_new_surface(
current,
Some((1792, 1024)),
(1792, 1024),
true
));
assert!(needs_new_surface(
current,
Some((1792, 1024)),
(2048, 1024),
false
));
// Nothing yet: allocate.
assert!(needs_new_surface(None, None, (1280, 1024), false));
}
}

12
apps/mpfiles/Cargo.toml Normal file
View file

@ -0,0 +1,12 @@
[package]
name = "mpfiles"
version = "0.1.0"
edition = "2021"
[dependencies]
makepad-widgets = { path = "../../widgets" }
mp-theme = { path = "../../libs/mp_theme" }
mp-wm-api = { path = "../../libs/mp_wm_api" }
# The ask panel's local model: an in-process Qwen GGUF on makepad-ggml, loaded
# only when the panel is first opened.
makepad-ai-hub = { path = "../../libs/ai/hub", default-features = false, features = ["llm"] }

View file

@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#e0af68" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#f0d2a6" d="M29 2v12h12z"/>
<path fill="#3a2c14" d="M20 2h5v5h-5zm5 5h5v5h-5zm-5 5h5v5h-5zm5 5h5v5h-5zm-5 5h5v5h-5zm5 5h5v5h-5zm-6 6h7a2 2 0 0 1 2 2v8a2 2 0 0 1-2 2h-7a2 2 0 0 1-2-2v-8a2 2 0 0 1 2-2z"/>
<path fill="#f0d2a6" d="M22 37h4v5h-4z"/>
</svg>

After

Width:  |  Height:  |  Size: 427 B

View file

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#bb9af7" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#d5c0fb" d="M29 2v12h12z"/>
<path fill="#2a2140" d="M32 22v18a6 5 0 1 1-4-4.7V27l-11 2.6v13.9a6 5 0 1 1-4-4.7V27z"/>
</svg>

After

Width:  |  Height:  |  Size: 297 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="square" stroke-linejoin="miter" d="m14 5-7 7 7 7M7 12h11"/></svg>

After

Width:  |  Height:  |  Size: 199 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M6 2h12a1 1 0 0 1 1 1v19l-7-4.4L5 22V3a1 1 0 0 1 1-1"/></svg>

After

Width:  |  Height:  |  Size: 130 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M2 3h20v13H10l-5 5v-5H2zM4 5v9h3v3l3-3h10V5zM6.5 8h2v2h-2zm4 0h2v2h-2zm4 0h2v2h-2z"/></svg>

After

Width:  |  Height:  |  Size: 160 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.3 19.4 1.6 11.7l2.8-2.8 4.9 4.9L19.6 3.5l2.8 2.8z"/></svg>

After

Width:  |  Height:  |  Size: 130 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><circle cx="12" cy="12" r="9" fill="none" stroke="currentColor" stroke-width="2"/><path d="M12 6v6l4 2" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="square"/></svg>

After

Width:  |  Height:  |  Size: 247 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m5.6 3.5 6.4 6.4 6.4-6.4 2.1 2.1-6.4 6.4 6.4 6.4-2.1 2.1-6.4-6.4-6.4 6.4-2.1-2.1 6.4-6.4-6.4-6.4z"/></svg>

After

Width:  |  Height:  |  Size: 175 B

View file

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#7aa2f7" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#a9c2fb" d="M29 2v12h12z"/>
<path fill="none" stroke="#16161e" stroke-width="3" stroke-linecap="round" stroke-linejoin="round" d="m19 27-6 7 6 7m10-14 6 7-6 7"/>
</svg>

After

Width:  |  Height:  |  Size: 342 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 4h2.5v2.5H3zm4.5 0H21v2.5H7.5zM3 10.75h2.5v2.5H3zm4.5 0H21v2.5H7.5zM3 17.5h2.5V20H3zm4.5 0H21V20H7.5z"/></svg>

After

Width:  |  Height:  |  Size: 183 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 7h14l-1 14H6zM3 4h18v3H3zm6-3h6l1 3H8zM8.43 11.31 11.12 14l-2.69 2.69.88.88L12 14.88l2.69 2.69.88-.88L12.88 14l2.69-2.69-.88-.88L12 13.12 9.31 10.43z"/></svg>

After

Width:  |  Height:  |  Size: 230 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 5c5 0 9 4.2 10 7-1 2.8-5 7-10 7S3 14.8 2 12c1-2.8 5-7 10-7zm0 3a4 4 0 1 0 0 8 4 4 0 0 0 0-8zm0 2.5a1.5 1.5 0 1 1 0 3 1.5 1.5 0 0 1 0-3z"/></svg>

After

Width:  |  Height:  |  Size: 218 B

View file

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#8992ad" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#b4bdd2" d="M29 2v12h12z"/>
<path stroke="#5f6880" stroke-width="3" stroke-linecap="round" d="M13 25h20M13 33h20M13 41h14"/>
</svg>

After

Width:  |  Height:  |  Size: 305 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 4h18l-7 8.5V20l-4-2.8v-4.7z"/></svg>

After

Width:  |  Height:  |  Size: 109 B

View file

@ -0,0 +1,4 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 54">
<path fill="#4f86d9" d="M3 11c0-3 2-5 5-5h17l7 8h24c3 0 5 2 5 5v27c0 3-2 5-5 5H8c-3 0-5-2-5-5z"/>
<path fill="#72a7ee" d="M3 20h58l-5 27c-.4 2.3-2.4 4-4.8 4H8c-2.8 0-5-2.2-5-5z"/>
</svg>

After

Width:  |  Height:  |  Size: 252 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="square" stroke-linejoin="miter" d="m10 5 7 7-7 7m7-7H6"/></svg>

After

Width:  |  Height:  |  Size: 197 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 3h7v7H3zm11 0h7v7h-7zM3 14h7v7H3zm11 0h7v7h-7z"/></svg>

After

Width:  |  Height:  |  Size: 128 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m2 11 10-9 10 9-2 2-2-2v10h-5v-6h-2v6H6V11l-2 2z"/></svg>

After

Width:  |  Height:  |  Size: 127 B

View file

@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#8992ad" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#b4bdd2" d="M29 2v12h12z"/>
<path fill="#1f2335" d="M11 26h26v20H11z"/>
<path fill="#9ece6a" d="m11 41 7-8 6 6 6-7 7 9v5H11z"/>
<circle cx="18" cy="31" r="3" fill="#e0af68"/>
</svg>

After

Width:  |  Height:  |  Size: 359 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 1a11 11 0 1 0 0 22 11 11 0 0 0 0-22m0 3.2a1.7 1.7 0 1 1 0 3.4 1.7 1.7 0 0 1 0-3.4M10.3 9.6h3.4v9.2h-3.4z"/></svg>

After

Width:  |  Height:  |  Size: 186 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 4h3v3H3zm5 0h13v3H8zM3 10.5h3v3H3zm5 0h13v3H8zM3 17h3v3H3zm5 0h13v3H8z"/></svg>

After

Width:  |  Height:  |  Size: 152 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><circle cx="12" cy="5" r="2"/><circle cx="12" cy="12" r="2"/><circle cx="12" cy="19" r="2"/></svg>

After

Width:  |  Height:  |  Size: 159 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 3h5v5H5zm9 0h5v5h-5zM9.5 16h5v5h-5zM7.5 8v4h9V8M12 12v4" fill="none" stroke="currentColor" stroke-width="2"/></svg>

After

Width:  |  Height:  |  Size: 188 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M2 4h8l2 3h10v14H2zm2 5v10h16V9zm7 2h2v2h2v2h-2v2h-2v-2H9v-2h2z"/></svg>

After

Width:  |  Height:  |  Size: 141 B

View file

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#db4b4b" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#f0a0a0" d="M29 2v12h12z"/>
<path fill="#ffffff" d="M12 30h6a4 4 0 0 1 0 8h-2v6h-4zm4 3v2h2a1 1 0 0 0 0-2zm8-3h6a5 5 0 0 1 5 5v4a5 5 0 0 1-5 5h-6zm4 3v8h2a2 2 0 0 0 2-2v-4a2 2 0 0 0-2-2zM37 30h9v3h-5v2h4v3h-4v6h-4z"/>
</svg>

After

Width:  |  Height:  |  Size: 398 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M17.65 6.35A7.96 7.96 0 0 0 12 4a8 8 0 1 0 7.73 10h-2.08A6 6 0 1 1 12 6c1.66 0 3.14.69 4.22 1.78L13 11h8V3z"/></svg>

After

Width:  |  Height:  |  Size: 185 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><circle cx="10.5" cy="10.5" r="6.5" fill="none" stroke="currentColor" stroke-width="2"/><path d="m15.5 15.5 5 5" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="square"/></svg>

After

Width:  |  Height:  |  Size: 256 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m12 2.5 3 6.1 6.7 1-4.9 4.7 1.2 6.7-6-3.2L6 21l1.2-6.7-4.9-4.7 6.7-1z"/></svg>

After

Width:  |  Height:  |  Size: 148 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M2 3h20v18H2zm2 3v13h16V6zm2.7 2.3 1.4-1.4 3.9 3.9-3.9 3.9-1.4-1.4L9.2 12zM12 15h6v2h-6z"/></svg>

After

Width:  |  Height:  |  Size: 166 B

View file

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 56">
<path fill="#a9b1d6" d="M7 2h22l12 12v38c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2z"/>
<path fill="#d3d9f0" d="M29 2v12h12z"/>
<path stroke="#3b4261" stroke-width="3" stroke-linecap="round" d="M13 25h20M13 33h20M13 41h14"/>
</svg>

After

Width:  |  Height:  |  Size: 305 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 7h14l-1 14H6zM3 4h18v3H3zm6-3h6l1 3H8z"/></svg>

After

Width:  |  Height:  |  Size: 120 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M2 2h12v13H2zm14 0h6v6h-6zm0 8h6v5h-6zM2 17h7v5H2zm9 0h5v5h-5zm7 0h4v5h-4z"/></svg>

After

Width:  |  Height:  |  Size: 152 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M2 15h20v7H2zm4-9h7v6H6zm0 6h7v2H6zm9-3h5v3h-5zm0 3h5v1.5h-5z"/></svg>

After

Width:  |  Height:  |  Size: 140 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 1.8 20.5 6.4 12 11 3.5 6.4zM2.8 8.1l8 4.4v9.4l-8-4.4zm18.4 0v9.4l-8 4.4v-9.4z"/></svg>

After

Width:  |  Height:  |  Size: 160 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 8h18l-9 11z"/></svg>

After

Width:  |  Height:  |  Size: 92 B

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M8 3l11 9-11 9z"/></svg>

After

Width:  |  Height:  |  Size: 93 B

View file

@ -0,0 +1,15 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 54">
<rect x="2" y="6" width="60" height="42" rx="3" fill="#f7768e"/>
<rect x="12" y="6" width="40" height="42" fill="#2a1219"/>
<path fill="#fbb0be" d="m27 19 13 8-13 8z"/>
<g fill="#2a1219">
<rect x="4.5" y="9" width="5" height="5" rx="1"/>
<rect x="4.5" y="18" width="5" height="5" rx="1"/>
<rect x="4.5" y="27" width="5" height="5" rx="1"/>
<rect x="4.5" y="36" width="5" height="5" rx="1"/>
<rect x="54.5" y="9" width="5" height="5" rx="1"/>
<rect x="54.5" y="18" width="5" height="5" rx="1"/>
<rect x="54.5" y="27" width="5" height="5" rx="1"/>
<rect x="54.5" y="36" width="5" height="5" rx="1"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 713 B

View file

@ -0,0 +1,185 @@
//! The sidebar's bookmarks: folders the user keeps, in their own section.
//!
//! Bookmarks are the one piece of mpfiles state that outlives the process, so
//! the format is the one a person can fix in an editor when it goes wrong: one
//! absolute path per line, in the order the sidebar shows them. That is also
//! what GNOME Files stores (`~/.config/gtk-3.0/bookmarks`), minus the URI
//! scheme nobody here needs.
//!
//! Nothing in this module touches the UI, so all of it is unit-testable.
use std::{
fs,
path::{Path, PathBuf},
};
/// How many bookmarks the sidebar has room for. The section is a fixed set of
/// slots in the DSL, so the model has to agree with it: a bookmark past the
/// last slot would be saved and never shown, which is worse than refusing it.
pub const MAX_BOOKMARKS: usize = 12;
/// The bookmarks file for a given home directory.
pub fn config_path(home: &Path) -> PathBuf {
home.join(".config").join("mpfiles").join("bookmarks")
}
/// The bookmark list, in sidebar order, and where it is persisted.
#[derive(Clone, Debug, Default)]
pub struct Bookmarks {
file: PathBuf,
list: Vec<PathBuf>,
}
impl Bookmarks {
/// Read the user's bookmarks. A missing file is an empty list, not an
/// error: the first run of a fresh install must not look broken.
pub fn load(home: &Path) -> Self {
let file = config_path(home);
let list = fs::read_to_string(&file)
.map(|text| parse(&text))
.unwrap_or_default();
Self { file, list }
}
/// A list held in memory only — for tests, and for a home we cannot write.
pub fn in_memory(list: Vec<PathBuf>) -> Self {
Self {
file: PathBuf::new(),
list,
}
}
pub fn list(&self) -> &[PathBuf] {
&self.list
}
pub fn contains(&self, path: &Path) -> bool {
self.list.iter().any(|p| p == path)
}
/// Bookmark `path`. False when it is already there or the sidebar is full
/// — either way nothing was added and the caller should say so.
pub fn add(&mut self, path: &Path) -> bool {
if self.contains(path) || self.list.len() >= MAX_BOOKMARKS {
return false;
}
self.list.push(path.to_path_buf());
self.persist();
true
}
/// Drop `path` from the sidebar. False when it was never there.
pub fn remove(&mut self, path: &Path) -> bool {
let Some(at) = self.list.iter().position(|p| p == path) else {
return false;
};
self.list.remove(at);
self.persist();
true
}
/// Write the list back. A failure is silent by design: a read-only home
/// must not stop the user from using a bookmark for this session.
fn persist(&self) {
if self.file.as_os_str().is_empty() {
return;
}
if let Some(dir) = self.file.parent() {
let _ = fs::create_dir_all(dir);
}
let _ = fs::write(&self.file, render(&self.list));
}
}
/// One path per line; blank lines and `#` comments are skipped so a
/// hand-edited file with a note in it still loads.
fn parse(text: &str) -> Vec<PathBuf> {
let mut out = Vec::new();
for line in text.lines() {
let line = line.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let path = PathBuf::from(line);
if !out.contains(&path) {
out.push(path);
}
if out.len() >= MAX_BOOKMARKS {
break;
}
}
out
}
/// The file's text, newline-terminated so appending by hand works.
fn render(list: &[PathBuf]) -> String {
let mut out = String::new();
for path in list {
out.push_str(&path.to_string_lossy());
out.push('\n');
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_one_path_per_line_skipping_notes() {
let list = parse("# my folders\n/a/b\n\n/c/d\n/a/b\n");
assert_eq!(list, [PathBuf::from("/a/b"), PathBuf::from("/c/d")]);
}
#[test]
fn renders_what_it_parses() {
let list = vec![PathBuf::from("/a/b"), PathBuf::from("/c d/e")];
assert_eq!(parse(&render(&list)), list);
}
#[test]
fn adds_removes_and_refuses_duplicates() {
let mut marks = Bookmarks::in_memory(Vec::new());
assert!(marks.add(Path::new("/a")));
assert!(!marks.add(Path::new("/a")));
assert!(marks.contains(Path::new("/a")));
assert!(marks.remove(Path::new("/a")));
assert!(!marks.remove(Path::new("/a")));
assert!(marks.list().is_empty());
}
#[test]
fn stops_at_the_last_sidebar_slot() {
let mut marks = Bookmarks::in_memory(Vec::new());
for i in 0..MAX_BOOKMARKS {
assert!(marks.add(Path::new(&format!("/p{i}"))), "{i}");
}
assert!(!marks.add(Path::new("/one-too-many")));
assert_eq!(marks.list().len(), MAX_BOOKMARKS);
}
#[test]
fn survives_a_round_trip_through_a_real_file() {
let home = std::env::temp_dir().join("mpfiles-test-bookmarks");
let _ = fs::remove_dir_all(&home);
fs::create_dir_all(&home).unwrap();
let mut marks = Bookmarks::load(&home);
assert!(marks.list().is_empty(), "a fresh home has no bookmarks");
assert!(marks.add(Path::new("/tmp/one")));
assert!(marks.add(Path::new("/tmp/two")));
// A second process sees exactly what the first one saved.
let reread = Bookmarks::load(&home);
assert_eq!(
reread.list(),
[PathBuf::from("/tmp/one"), PathBuf::from("/tmp/two")]
);
assert!(config_path(&home).is_file(), "the list is on disk where it says");
marks.remove(Path::new("/tmp/one"));
assert_eq!(Bookmarks::load(&home).list(), [PathBuf::from("/tmp/two")]);
fs::remove_dir_all(&home).ok();
}
}

View file

@ -0,0 +1,85 @@
//! The file browser's thin adapter to the shared in-process local chat engine.
use makepad_ai_hub::{
hub::{AiHub, ChatConfig},
hub_chat::HubChatSession,
local_llm::{LocalLlmConfig, ToolSpec},
};
use makepad_widgets::makepad_platform::thread::SignalToUI;
use std::{
path::{Path, PathBuf},
sync::Arc,
};
pub use makepad_ai_hub::local_llm::ChatEvent;
/// Where the weights live, relative to the checkout this was built from.
pub const MODEL_FILE: &str = "local/models/Qwen3.5-9B-UD-Q4_K_XL.gguf";
/// The environment variable that overrides it.
pub const MODEL_ENV: &str = "MPFILES_CHAT_MODEL";
pub struct ChatAgent {
session: HubChatSession,
}
impl ChatAgent {
/// Start loading the model. Nothing blocks: the load happens on the hub's
/// worker and reports itself through [`ChatAgent::poll`].
pub fn start(model: PathBuf, system_prompt: String, tools: Vec<ToolSpec>) -> Self {
let config = ChatConfig {
llm: LocalLlmConfig::new(model),
system_prompt,
tools,
wake: Some(Arc::new(SignalToUI::set_ui_signal)),
};
Self {
session: AiHub::in_process().start_local_chat(config),
}
}
pub fn send_user_turn(&self, text: String) {
self.session.send_user_turn(text);
}
pub fn send_tool_results(&self, results: Vec<(String, bool)>) {
self.session.send_tool_results(results);
}
pub fn cancel(&self) {
self.session.cancel();
}
pub fn poll(&self) -> Vec<ChatEvent> {
self.session.poll()
}
}
/// Where the weights are, or `None` when this machine has none.
///
/// `MPFILES_CHAT_MODEL` wins; otherwise the file is looked for relative to the
/// working directory, then up from the binary (which finds `target/release`
/// runs from anywhere), then in the checkout this binary was compiled in.
pub fn model_path() -> Option<PathBuf> {
if let Some(from_env) = std::env::var_os(MODEL_ENV) {
let path = PathBuf::from(from_env);
return path.is_file().then_some(path);
}
let relative = Path::new(MODEL_FILE);
if relative.is_file() {
return Some(relative.to_path_buf());
}
if let Ok(exe) = std::env::current_exe() {
for base in exe.ancestors() {
let candidate = base.join(relative);
if candidate.is_file() {
return Some(candidate);
}
}
}
let checkout = Path::new(env!("CARGO_MANIFEST_DIR"))
.parent()
.and_then(Path::parent)
.map(|root| root.join(relative))?;
checkout.is_file().then_some(checkout)
}

Some files were not shown because too many files have changed in this diff Show more