137 lines
8.1 KiB
Markdown
137 lines
8.1 KiB
Markdown
# Desktop tray capture
|
|
|
|
## Workflow and ownership
|
|
|
|
One Makepad application, two workspace members, no additional platform projects.
|
|
`budget-capture/src/desktop.rs` owns the cancellable worker and tray facade;
|
|
`budget-capture/src/sys/desktop` owns OS capture/tray integration. `app` owns the
|
|
consent screen, editable review, product validation and durable confirmation.
|
|
Android tile/overlay and Share to Budget remain separate, unchanged paths.
|
|
|
|
1. Start Budget normally (`cargo run -p budget --locked`).
|
|
2. Choose **Capture product…** from the tray, or **Capture** in Budget. Merely
|
|
opening this screen does not capture anything.
|
|
3. Click **Capture once — I consent**. Windows, macOS and X11 minimize Budget for a three-second
|
|
switching delay and capture the primary display, or the
|
|
first display if the server declares no primary. Wayland leaves Budget open;
|
|
switch to the product, then use the system portal to select one screen/window with fresh permission.
|
|
4. A single frame is copied into memory, capture resources close, then the
|
|
embedded offline OCR engine runs. Windows/macOS/X11 restore Budget afterward;
|
|
on Wayland, return with the OS application switcher.
|
|
5. Review/edit the suggested first-line title; enter and verify unit price,
|
|
currency, quantity, shipping, date, optional URL, expense category and item.
|
|
A single standalone `R`/`ZAR` price line may prefill the ZAR quote; multiple
|
|
prices or ambiguous text require manual entry. No currency conversion or
|
|
unmarked-price inference is performed. The
|
|
planner currently accepts **ZAR only**; it does not perform currency conversion.
|
|
6. **Confirm — add planned purchase** persists a validated candidate workbook
|
|
before replacing the live model. This records a planned purchase, not a
|
|
payment. A stable per-review UUID makes retry/double-confirm idempotent.
|
|
7. **Cancel / discard capture** clears the draft without writing a purchase.
|
|
If OCR is already running, its result is ignored; another worker cannot start
|
|
until the current worker exits. Opening the tray again does not replace an
|
|
unfinished review. Closing the application exits; this is not an autostart
|
|
daemon or a close-to-tray implementation. The tray has explicit Open,
|
|
Capture, Minimize and Quit commands (Minimize is disabled on Wayland).
|
|
|
|
## Privacy and resource bounds
|
|
|
|
- No uploads, runtime OCR downloads, screenshot files, clipboard writes,
|
|
continuous monitoring, automatic purchase creation or persistent permission tokens.
|
|
- Build-time OCR model download/verification remains in the existing build script.
|
|
- Native capture is limited to 33,554,432 pixels and OCR is resized to a maximum
|
|
1,920-pixel longest edge. OCR results are limited to 16,384 characters; the
|
|
on-screen excerpt is limited to 2,000 characters.
|
|
- Wayland deliberately does **not** call XCap's Screenshot or VideoRecorder
|
|
paths. A private D-Bus connection owns a fresh ScreenCast portal session,
|
|
`PersistMode::DoNot`, and one PipeWire stream. The first valid packed RGB-family
|
|
frame ends the loop; stream/session/connection teardown happens before OCR.
|
|
Selection times out after 120 seconds; frame receipt after 20 seconds. A
|
|
100 ms cancellation check is used during selection and frame receipt.
|
|
- Frame conversion validates channel format, dimensions, stride, offset and
|
|
available bytes. Unsupported formats/negative strides fail instead of being
|
|
silently misinterpreted. A cancelled result is rejected even if already queued.
|
|
- Window titles, screenshot pixels and recognized text are not intentionally
|
|
written to logs. Only confirmed product fields enter the existing local store.
|
|
Heap buffers are released, not claimed to be secure-memory/anti-swap storage.
|
|
|
|
## Platform prerequisites and validation limits
|
|
|
|
| Target | Implementation | Required platform follow-up |
|
|
|---|---|---|
|
|
| Linux X11 | StatusNotifier tray (`ksni`), one-frame XCap capture, shared review | Actual desktop panel interaction / multi-monitor acceptance |
|
|
| Linux Wayland | StatusNotifier tray, fresh portal + bounded PipeWire single-frame backend | Experimental: real compositor/portal tests and native activation-token integration still required |
|
|
| Windows | Native tray/menu, one-frame XCap capture, shared review; whole-app cross-target check passed | Native Windows link/package, permission/capture and menu runtime tests |
|
|
| macOS | Main-thread status item/menu, Screen Recording access check/request, one-frame XCap capture; whole-app ARM64 cross-target check passed | Native macOS link/package, signing/TCC permission and runtime tests |
|
|
| HarmonyOS PC | **Blocked; not implemented or advertised as working** | Huawei PC SDK/API version and device or simulator, desktop status-bar extension bridge and capture permission validation |
|
|
|
|
Linux needs a StatusNotifier/AppIndicator-compatible panel. GNOME may need an
|
|
AppIndicator extension; tray construction errors are reported in the Capture screen and
|
|
the app remains usable from its window. A missing panel/host may not cause a
|
|
construction error, so use the Capture tab if the icon is absent. A tray library accepting registration
|
|
is not proof that the desktop shell displays the icon.
|
|
|
|
Linux build dependencies (Debian/Ubuntu), in addition to Rust >= 1.90 and the
|
|
matching Makepad checkout:
|
|
|
|
```sh
|
|
sudo apt-get install pkg-config libclang-dev libxcb1-dev libxrandr-dev \
|
|
libdbus-1-dev libpipewire-0.3-dev libwayland-dev libegl-dev libgbm-dev \
|
|
libdrm-dev libxkbcommon-dev libx11-dev libxcursor-dev libasound2-dev \
|
|
libpulse-dev libssl-dev libvulkan-dev libgl1-mesa-dev
|
|
```
|
|
|
|
Headless tests additionally need `xvfb`, `xauth`, `dbus-x11`, `x11-utils`, `xfonts-base` and a JDK for the
|
|
unchanged Android Java tests. The pinned Makepad backend does not provide Wayland activation tokens. Consequently
|
|
Budget does not auto-minimize on Wayland, and tray Open cannot reliably raise a
|
|
window hidden/minimized by the user; use the OS application switcher. This is a
|
|
known limitation, not a claim of full Wayland tray/window integration. macOS uses
|
|
AppKit activation plus deminiaturization, not Makepad's fullscreen-toggle restore.
|
|
|
|
Desktop dependencies explicitly exclude
|
|
`target_env = "ohos"`; a generic OpenHarmony build is **not** Huawei PC support.
|
|
|
|
## HarmonyOS PC blocker
|
|
|
|
Huawei's official status-bar extension guide describes `statusBarManager` and
|
|
`StatusBarViewExtensionAbility` from `@kit.DeskTopExtensionKit`:
|
|
https://developer.huawei.com/consumer/cn/doc/HarmonyOS-Guides/statusbar-extension-guide
|
|
|
|
The PC extension lifecycle, module metadata, SDK-exposed APIs, native bridge and
|
|
allowed single-frame screen-capture permissions must be verified using the
|
|
actual Huawei SDK. No speculative ETS adapter, extra standalone application,
|
|
or uncompiled bridge is included. The future bridge belongs under the existing
|
|
native member; product review and confirmation stay in `app`.
|
|
|
|
## Required Makepad revision
|
|
|
|
Validated against `9965c04940358ad664f190ea92f7b2c0c09fb1b4` on the existing
|
|
`nigig-traffic-camera-boom-fix` branch. This adds a two-line Windows build fix on
|
|
previously validated `326530539c5c66ed0d05c254e37fcb1e4b08212b`: keep the manual
|
|
function-pointer-aware `PartialEq` implementations for `EXCEPINFO` and
|
|
`WNDCLASSEXW`, removing only their conflicting derives. The app could not pass
|
|
Windows type checking without this fix.
|
|
|
|
The source bundle includes `MAKEPAD_DESKTOP_COMPILE_FIX.patch` against `3265305`.
|
|
Apply it to that matching Makepad checkout if the new commit is not available:
|
|
|
|
```sh
|
|
git -C ../rustdev/makepad apply "$PWD/docs/MAKEPAD_DESKTOP_COMPILE_FIX.patch"
|
|
```
|
|
|
|
For an older `2d732a5` checkout, first apply the existing
|
|
`MAKEPAD_PACKAGER_COMPILE_FIX.patch` as described in the Android validation report.
|
|
Neither repository's new desktop commits have been pushed by this task.
|
|
|
|
## Reproducing validation
|
|
|
|
Use the owner-local disposable runner; it removes its Rust/toolchain/build state:
|
|
|
|
```sh
|
|
VALIDATED_LOCKFILE_OUTPUT="$PWD/docs/Cargo.lock" \
|
|
LIVE_X11_CAPTURE=1 app/tests/run-clean.sh
|
|
```
|
|
|
|
The live X11 gate explicitly captures an isolated Xvfb test display, not the
|
|
user's real desktop. Ordinary unit tests do not silently capture a real screen.
|
|
See `DESKTOP_VALIDATION.md` for the exact executed gates and remaining blockers.
|