8.1 KiB
Desktop tray capture
Workflow and ownership
One Makepad application, two workspace members, no additional platform projects.
budget-capture/src/desktop.rs owns the cancellable worker and tray facade;
budget-capture/src/sys/desktop owns OS capture/tray integration. app owns the
consent screen, editable review, product validation and durable confirmation.
Android tile/overlay and Share to Budget remain separate, unchanged paths.
- Start Budget normally (
cargo run -p budget --locked). - Choose Capture product… from the tray, or Capture in Budget. Merely opening this screen does not capture anything.
- Click Capture once — I consent. Windows, macOS and X11 minimize Budget for a three-second switching delay and capture the primary display, or the first display if the server declares no primary. Wayland leaves Budget open; switch to the product, then use the system portal to select one screen/window with fresh permission.
- A single frame is copied into memory, capture resources close, then the embedded offline OCR engine runs. Windows/macOS/X11 restore Budget afterward; on Wayland, return with the OS application switcher.
- Review/edit the suggested first-line title; enter and verify unit price,
currency, quantity, shipping, date, optional URL, expense category and item.
A single standalone
R/ZARprice line may prefill the ZAR quote; multiple prices or ambiguous text require manual entry. No currency conversion or unmarked-price inference is performed. The planner currently accepts ZAR only; it does not perform currency conversion. - Confirm — add planned purchase persists a validated candidate workbook before replacing the live model. This records a planned purchase, not a payment. A stable per-review UUID makes retry/double-confirm idempotent.
- Cancel / discard capture clears the draft without writing a purchase. If OCR is already running, its result is ignored; another worker cannot start until the current worker exits. Opening the tray again does not replace an unfinished review. Closing the application exits; this is not an autostart daemon or a close-to-tray implementation. The tray has explicit Open, Capture, Minimize and Quit commands (Minimize is disabled on Wayland).
Privacy and resource bounds
- No uploads, runtime OCR downloads, screenshot files, clipboard writes, continuous monitoring, automatic purchase creation or persistent permission tokens.
- Build-time OCR model download/verification remains in the existing build script.
- Native capture is limited to 33,554,432 pixels and OCR is resized to a maximum 1,920-pixel longest edge. OCR results are limited to 16,384 characters; the on-screen excerpt is limited to 2,000 characters.
- Wayland deliberately does not call XCap's Screenshot or VideoRecorder
paths. A private D-Bus connection owns a fresh ScreenCast portal session,
PersistMode::DoNot, and one PipeWire stream. The first valid packed RGB-family frame ends the loop; stream/session/connection teardown happens before OCR. Selection times out after 120 seconds; frame receipt after 20 seconds. A 100 ms cancellation check is used during selection and frame receipt. - Frame conversion validates channel format, dimensions, stride, offset and available bytes. Unsupported formats/negative strides fail instead of being silently misinterpreted. A cancelled result is rejected even if already queued.
- Window titles, screenshot pixels and recognized text are not intentionally written to logs. Only confirmed product fields enter the existing local store. Heap buffers are released, not claimed to be secure-memory/anti-swap storage.
Platform prerequisites and validation limits
| Target | Implementation | Required platform follow-up |
|---|---|---|
| Linux X11 | StatusNotifier tray (ksni), one-frame XCap capture, shared review |
Actual desktop panel interaction / multi-monitor acceptance |
| Linux Wayland | StatusNotifier tray, fresh portal + bounded PipeWire single-frame backend | Experimental: real compositor/portal tests and native activation-token integration still required |
| Windows | Native tray/menu, one-frame XCap capture, shared review; whole-app cross-target check passed | Native Windows link/package, permission/capture and menu runtime tests |
| macOS | Main-thread status item/menu, Screen Recording access check/request, one-frame XCap capture; whole-app ARM64 cross-target check passed | Native macOS link/package, signing/TCC permission and runtime tests |
| HarmonyOS PC | Blocked; not implemented or advertised as working | Huawei PC SDK/API version and device or simulator, desktop status-bar extension bridge and capture permission validation |
Linux needs a StatusNotifier/AppIndicator-compatible panel. GNOME may need an AppIndicator extension; tray construction errors are reported in the Capture screen and the app remains usable from its window. A missing panel/host may not cause a construction error, so use the Capture tab if the icon is absent. A tray library accepting registration is not proof that the desktop shell displays the icon.
Linux build dependencies (Debian/Ubuntu), in addition to Rust >= 1.90 and the matching Makepad checkout:
sudo apt-get install pkg-config libclang-dev libxcb1-dev libxrandr-dev \
libdbus-1-dev libpipewire-0.3-dev libwayland-dev libegl-dev libgbm-dev \
libdrm-dev libxkbcommon-dev libx11-dev libxcursor-dev libasound2-dev \
libpulse-dev libssl-dev libvulkan-dev libgl1-mesa-dev
Headless tests additionally need xvfb, xauth, dbus-x11, x11-utils, xfonts-base and a JDK for the
unchanged Android Java tests. The pinned Makepad backend does not provide Wayland activation tokens. Consequently
Budget does not auto-minimize on Wayland, and tray Open cannot reliably raise a
window hidden/minimized by the user; use the OS application switcher. This is a
known limitation, not a claim of full Wayland tray/window integration. macOS uses
AppKit activation plus deminiaturization, not Makepad's fullscreen-toggle restore.
Desktop dependencies explicitly exclude
target_env = "ohos"; a generic OpenHarmony build is not Huawei PC support.
HarmonyOS PC blocker
Huawei's official status-bar extension guide describes statusBarManager and
StatusBarViewExtensionAbility from @kit.DeskTopExtensionKit:
https://developer.huawei.com/consumer/cn/doc/HarmonyOS-Guides/statusbar-extension-guide
The PC extension lifecycle, module metadata, SDK-exposed APIs, native bridge and
allowed single-frame screen-capture permissions must be verified using the
actual Huawei SDK. No speculative ETS adapter, extra standalone application,
or uncompiled bridge is included. The future bridge belongs under the existing
native member; product review and confirmation stay in app.
Required Makepad revision
Validated against 9965c04940358ad664f190ea92f7b2c0c09fb1b4 on the existing
nigig-traffic-camera-boom-fix branch. This adds a two-line Windows build fix on
previously validated 326530539c5c66ed0d05c254e37fcb1e4b08212b: keep the manual
function-pointer-aware PartialEq implementations for EXCEPINFO and
WNDCLASSEXW, removing only their conflicting derives. The app could not pass
Windows type checking without this fix.
The source bundle includes MAKEPAD_DESKTOP_COMPILE_FIX.patch against 3265305.
Apply it to that matching Makepad checkout if the new commit is not available:
git -C ../rustdev/makepad apply "$PWD/docs/MAKEPAD_DESKTOP_COMPILE_FIX.patch"
For an older 2d732a5 checkout, first apply the existing
MAKEPAD_PACKAGER_COMPILE_FIX.patch as described in the Android validation report.
Neither repository's new desktop commits have been pushed by this task.
Reproducing validation
Use the owner-local disposable runner; it removes its Rust/toolchain/build state:
VALIDATED_LOCKFILE_OUTPUT="$PWD/docs/Cargo.lock" \
LIVE_X11_CAPTURE=1 app/tests/run-clean.sh
The live X11 gate explicitly captures an isolated Xvfb test display, not the
user's real desktop. Ordinary unit tests do not silently capture a real screen.
See DESKTOP_VALIDATION.md for the exact executed gates and remaining blockers.