budget-app/docs/SHARED_CAPTURE_UI.md
Arena Assistant 1988457dd1 feat: add macOS capture panel and Android Makepad review
Reuse shared review and confirmation state. Android tile/setup and volatile OCR handoff route into Makepad; native floating controls remain. Android overlay rendering host is not implemented. Prior isolated checks: macOS ARM64 and Android ARM64 library checks, native Linux build/run and tests; native mobile/macOS runtime validation remains outstanding.
2026-10-28 05:02:41 +00:00

117 lines
12 KiB
Markdown

# Shared Makepad capture UI migration
## Shared access controls (published as 2bf18d8)
- The app's Capture product entry opens the existing Makepad capture page on every target instead of immediately opening Android's Java setup activity.
- That page contains shared access-status presentation, an explicit refresh action, Android overlay/notification permission requests, and enable/stop controls using the Rust capture facade.
- Requests are distinguished from grants. MediaProjection consent remains per-capture, never a persisted grant.
- Desktop capture and editable review continue to use the existing Makepad page. Desktop access wording does not pretend Android permissions apply to desktop platforms.
- The isolated native validation runner now recreates the manifest's actual sibling Makepad directory layout.
## Not implemented yet — do not call the migration complete
- Android Quick Settings now opens the Makepad capture page (see Android increment below).
- Android access explanations/requests are now Makepad-owned. The consent activity contains only Android system-consent lifecycle glue.
- Android floating controls and the small handoff/error prompt remain native Java views; editing and confirmation are Makepad-owned.
- A Makepad service-hosted Android rendering surface, input/focus routing, and lifecycle synchronization are not implemented.
- Android OCR delivery into the shared Makepad review model is implemented with a bounded volatile mailbox; Share to Budget remains independent and unchanged.
- macOS floating controls are implemented as described below. Inline review remains in the main Makepad window. Windows/Linux floating controls are not yet implemented.
- HarmonyOS PC remains unavailable pending the native SDK bridge and validation. Wayland must not promise unrestricted overlay placement.
- The missing dependency DEX packager fix discussed in chat is not applied to an SDK checkout here. It remains necessary for Android class loading.
## Required next implementation steps
1. Restore the exact owner Makepad SDK checkout, including local SDK 33/35 changes and dependency DEX packaging.
2. Build and run the app through app/tests/run-clean.sh before committing/publishing these UI changes.
3. Implement and test Android service-hosted Makepad surface lifecycle; do not reuse a single activity EGL surface for two live windows without backend support.
4. Route tile/overlay actions into a shared Rust state machine; keep consent tokens in the native capture backend and never persist them.
5. Deliver ephemeral OCR results to the existing Makepad review/transaction path, then remove duplicated Java presentation only after device tests pass.
6. Validate desktop floating-window behavior individually, exposing capability limitations (especially Wayland and HarmonyOS PC) to shared UI.
## Historical validation of the access-controls commit
- Existing Python Android contract tests: 2 passed.
- Existing Python app packaging tests: 2 passed.
- Native build/run attempted with the corrected isolated runner: blocked before toolchain setup because ../nigig-org/makepad-native-glue/makepad is absent.
- No Rust compilation, rendered UI test, Android device test, or native macOS/Windows/HarmonyOS test passed for this increment. Existing Python checks do not validate Makepad DSL or Rust type correctness.
- Published on the existing Budget main branch at the owner's explicit request. Publication does not imply Rust compilation or platform validation; the blockers above remain outstanding.
## macOS floating panel increment
`app/src/capture_panel.rs` adapts Makepad's `examples/floating_panel`:
- A macOS-only second Makepad window, configured as a borderless, non-activating floating NSPanel, with a draggable header.
- Capture, cancel/discard, and open-review controls reuse the existing BudgetApp capture state. No second job, workbook, or review model is created.
- Capture is disabled while a worker is active or a review is pending. Requests from the main page and panel share the same guard.
- All editing and financial confirmation remain on the existing main Makepad review page. The panel cannot confirm a purchase.
- Before starting a worker, the native backend hides the panel with AppKit orderOut. If it cannot find/hide the panel on the UI thread, capture does not start. Borderless panels are not assumed to be miniaturizable.
- The panel is restored on completion with orderFrontRegardless; tray open will not restore it while a worker is active.
- Android and other platforms retain their original single-window shell. No Android overlay support is claimed by this increment.
The native helper in budget-capture only controls window visibility. It draws no widgets. It identifies the single app-owned panel by its reserved title `Old Mutual Budget — Capture Panel`.
### Owner macOS acceptance test
With the matching Makepad SDK available:
```sh
cargo run -p budget
cargo test -p budget --test ui floating_capture_panel_opens_shared_review -- --nocapture
```
Manually verify: header drag; no unwanted app activation on panel click; hide before screenshot; restore on success/error/cancel; review blocks another capture; cancel adds nothing; tray reopen; Screen Recording denial; quitting with a worker active. A cross-target check does not prove any of these native runtime properties.
### Validation of this increment
- Exact upstream Makepad revision: 44ecb822b48106fa76bde4dfd309e5a8a7ff1932, which contains the floating_panel example. This is not the owner's locally modified Android packager.
- Full app macOS ARM64 cross-target cargo check: passed (2m01s). No macOS app was launched in this Linux environment.
- Python source/packaging contracts: 7 passed. These are structural checks, not rendered-panel tests.
- Native Linux isolated build and 15-second cargo run smoke: passed. Release UI suite: 7 passed in 16.96s, including Linux-rendered macOS/Omarchy style regression checks. These are not native macOS panel tests.
- Rust test summaries: 101 app library + 6 capture library + 5 review + 9 models + 4 desktop + 3 OCR + 7 UI = 135 passed. Opt-in live capture tests returned without running their live-capture branches (LIVE_X11_CAPTURE was not enabled).
- Python checks: 7 passed; Java model tests: 12 passed.
- Native macOS panel smoke test was added but not executed here. No Android, Windows, or HarmonyOS runtime test was performed.
- First Linux attempt failed for missing libclang. Installed the native dependency and reran successfully. No project code workaround was needed.
- Temporary SDKs, Rust toolchains/caches/build products and the temporary 3 GiB swap file were removed. System-native development libraries remain installed in the sandbox.
- Detailed logs are under /home/user/panel-validation; the resolved Linux validation lock is panel-validation/validated.lock. No dependency pins or repository lockfile were changed.
- This increment is included in the owner-authorized Budget main publication together with the Android setup/review increment.
## Android Makepad setup/review increment
### Implemented
1. BudgetTileService queues an in-process open request and launches the main Budget activity. It does not start screen capture.
2. The Makepad capture page consumes that request on startup/resume/actions and presents the shared access UI. Enabling floating capture checks overlay access and requests the native service; it no longer opens a Java setup form.
3. Tapping the existing floating control opens CapturePermissionActivity, which has no custom layout/buttons. It requests a fresh Android MediaProjection grant and passes that grant only to the native capture service.
4. OCR processing still stops projection before inference. Its native handoff prompt offers Open review in Budget or Discard.
5. CaptureReviewInbox holds at most one OCR string (16,384 UTF-16 units) in process memory. Reads/offers reject entries five minutes old. It uses no disk, clipboard, intent text extras, uploads, screenshot retention, or consent-token persistence. Process death loses the handoff; TTL cleanup is lazy on access.
6. JNI consumes the text only when Makepad has no pending review/job. Main-window fields, validation, UUID identity, persistence, and explicit confirmation use the same Review/product transaction path as desktop. Empty OCR allows manual entry on Android.
7. Cancel/discard stops the native capture generation and clears pending handoff text. Native stop, close, task removal and screen-off also clear unconsumed handoffs where the service is active. Successful handoff teardown deliberately does not clear text before Makepad consumes it.
8. Existing ShareToBudget Java activity and its external sharing workflow are unchanged.
### Still not a Makepad-rendered Android floating surface
The Android renderer inspected at Makepad 44ecb822 has one activity-owned rendering surface and global surface JNI callbacks; CreateWindow does not supply the independent service-window hosting used by macOS NSPanel. This increment does NOT reparent that surface, create a second Android renderer, or pretend the macOS configuration works on Android.
BudgetOverlayService still draws the floating +/close controls and handoff/error prompt with Android views. Those require a separate SDK-level rendering/lifecycle integration before they can be replaced safely. No claim of full Android floating-UI migration should be made.
### Packaging and device acceptance
The final APK must include CaptureReviewInbox plus the existing native classes. app/tests/verify_apk.py now checks it. The owner's dependency-DEX bundling fix remains required; this increment does not modify cargo-makepad.
On an Android device verify: tile opens Makepad setup; deny/grant overlay access; enable/stop; system capture denial; one successful capture; Open review routes into Makepad; edits survive reopening; another result cannot overwrite active edits; discard adds nothing; confirmation is idempotent; task/process death discards unconsumed handoff; Share to Budget still works. Fresh consent is mandatory on every capture.
### Validation
- Pure Java inbox/extraction tests: 18 passed.
- Python source/packaging contract tests: 9 passed.
- All production Java compiled against the real Android SDK 35 android.jar; the capture build script also generated DEX during the Android check.
- Android ARM64 `cargo check -p budget --lib --target aarch64-linux-android`: passed in 1m06s using upstream Makepad 44ecb822. This is type checking, not native linking or APK assembly.
- Linux workspace library tests (101 + 6), shared review (5), and models (9): 121 passed. Native cargo build and 15-second cargo run smoke passed.
- No fresh rendered UI suite was run for the Android increment; the prior macOS increment's Linux UI results are recorded above.
- First validation command incorrectly checked the desktop executable target on Android. It failed because Android exports the library entry point rather than shell::app_main. The runner was corrected to --lib; no app entrypoint workaround was introduced.
- JNI calls now use bounded local frames so repeated handoff/access polling does not leak local references on the permanently attached Rust thread.
- Logs and the resolved validation lock are in /home/user/android-review-validation. Temporary SDK archives, Rust toolchains, caches, and build products were removed. No repository lockfile was changed.
- The macOS and Android increments are included in the owner-authorized Budget main publication. The proposed Makepad-rendered Android overlay host is not implemented; no Makepad SDK changes are included.
- No Android device/emulator was available; no full APK installation or runtime behavior is claimed.