139 lines
6.7 KiB
Markdown
139 lines
6.7 KiB
Markdown
# AI Hub fleet security
|
|
|
|
Code: `makepad_network::fleet` (`platform/network/src/fleet/`): `auth.rs`
|
|
(credentials), `front.rs` (the TLS front, shared by every fleet service),
|
|
`lan.rs` (the private-LAN bind rule); the hub's route policy is
|
|
`libs/ai/hub/src/fleet_routes.rs`; `makepad_network::tls` (TLS on each OS's
|
|
own stack, known hosts). The asset store's sync listener (store sync,
|
|
`apps/commercial/engine/asset/store/src/host/sync.rs`) uses the same front
|
|
and credentials: roles `node` and `lan`, read-only sync routes.
|
|
|
|
## The model
|
|
|
|
- **Nodes are trusted the ssh way.** Each node makes its own self-signed
|
|
certificate on first start (`<cache>/fleet/tls`). A client records the
|
|
certificate fingerprint per `host:port` on first contact, in
|
|
`~/.makepad/ai-hub/known_nodes`. If it later changes, the client:
|
|
- prints a loud warning (host, old and new fingerprint);
|
|
- appends it to `known_nodes.log`;
|
|
- keeps it in `makepad_network::tls::fingerprint_changes()`, for status
|
|
displays;
|
|
- updates the record and continues.
|
|
|
|
Nothing about nodes is provisioned.
|
|
- **Clients are what is checked.**
|
|
- A fleet key (32 random bytes) is on the admin Mac and on every node
|
|
(`<cache>/fleet/fleet.key`, 0600).
|
|
- A client credential is `mkc2.<id>.<role>.<expiry>.<secret>`, where the
|
|
secret is derived from the fleet key.
|
|
- The credential is never sent. Each request carries
|
|
`Authorization: MKC2 <id>.<role>.<expiry>.<proof>`, where the proof is
|
|
HMAC(secret, `"mkfleet2 proof|" + certificate sha256`) for the
|
|
certificate the node actually presented.
|
|
- Something in the middle with its own certificate only gets a proof for
|
|
that certificate, which is useless against a real node.
|
|
- **Roles and ownership.**
|
|
- `lan` (people's apps) reaches the whole job API.
|
|
- `node` reaches model blobs, `/v1/model_inventory` and `/health` only.
|
|
- `device` reaches its own job routes only.
|
|
- Jobs belong to the credential that submitted them. Only that client can
|
|
cancel them, keep their leases alive or say `/bye` for them.
|
|
- **No open nodes.** A node bound to a network address serves only through
|
|
its TLS front, and refuses to start without `<cache>/fleet/fleet.key`. A
|
|
loopback-only node (the machine node, tests) is machine-local and needs
|
|
none of this.
|
|
- **Front limits.**
|
|
- 10 s for TLS.
|
|
- A request head of at most 16 KiB, within 15 s.
|
|
- 5 failed authentications from one address within 10 min ban it for
|
|
15 min.
|
|
- 32 connections per address, 256 in total (8 and 64 on the edge).
|
|
- **The node's HTTP server.** It listens on loopback only and trusts
|
|
identity headers only when they carry the front's per-process secret.
|
|
- **Discovery.** Beacons (`makepad-ai-hub-tls`) only say where a node's TLS
|
|
port is. A process with a credential reaches every node it hears; one
|
|
without a credential ignores them (logged).
|
|
- **Peers.** A node fetches model blobs from fleet endpoints only:
|
|
- request `peer_sources` must be discovered fleet endpoints;
|
|
- `MAKEPAD_AI_PEER_SOURCES` hosts are mapped onto the discovered nodes.
|
|
|
|
## Admin tasks (`ai-fleet`, admin Mac only)
|
|
|
|
```sh
|
|
cargo build --release -p makepad-ai-hub --bin ai-fleet
|
|
./target/release/ai-fleet init # ~/.makepad/ai-hub/admin/fleet.key
|
|
./target/release/ai-fleet issue rik-mac lan --out ~/.makepad/ai-hub/client.credential
|
|
./target/release/ai-fleet node-files w203 --out /tmp/w203 # fleet.key + node.credential
|
|
./target/release/ai-fleet revoke <client id>
|
|
```
|
|
|
|
**Enrol a node.** Copy `fleet.key` and `node.credential` (from
|
|
`node-files`) into `<cache>/fleet/` and `peer-secret` into `<cache>/`,
|
|
readable only by the node's account, and restart the node. `peer-secret`
|
|
signs model-blob tickets: without it the node serves no blobs to its peers.
|
|
|
|
**Keep every node's models complete.** `models = all` in the WM's
|
|
`ai-hub.conf` (or `MAKEPAD_AI_PREFETCH=all` for a bare hub) fetches every
|
|
registry model the node can run, from the peers that hold each file, then
|
|
Hugging Face; `models_licences` (default `none,community`) names the licence
|
|
classes it may fetch (`non-commercial`, `restricted`, `gated` are opt-in),
|
|
`models_mbps` paces it (default 60). Each pass writes `<cache>/prefetch.status`.
|
|
|
|
**Enrol a person or app.** Run `ai-fleet issue <id> lan --out <file>`. On
|
|
the client, the file becomes `~/.makepad/ai-hub/client.credential` (mode
|
|
600), or `MAKEPAD_AI_HUB_CREDENTIAL` holds its text.
|
|
|
|
**Revoke.** `ai-fleet revoke <id>` adds the id to
|
|
`~/.makepad/ai-hub/admin/revoked`. Copy that file to every node's
|
|
`<cache>/fleet/revoked` and restart the nodes. Credentials expire after
|
|
`--days` (default 365); to rotate one, issue a new one and revoke the old
|
|
id. To rotate the fleet key itself, run `ai-fleet init` on a fresh admin
|
|
directory, then re-issue everything.
|
|
|
|
**Scripts without the Rust client** (for example Python):
|
|
1. Open TLS without CA checking.
|
|
2. Take sha256 of the server's DER certificate. Compare it with your own
|
|
record if you keep one; warn on a change and continue.
|
|
3. Send `Authorization: MKC2 <id>.<role>.<expiry>.<hex HMAC-SHA256(bytes.fromhex(secret), b"mkfleet2 proof|" + fingerprint_hex)>`.
|
|
|
|
## Going on the internet later (not done; for the user)
|
|
|
|
The hub on 165 can serve an **edge**: a second TLS front with device routes
|
|
only. Set `MAKEPAD_AI_HUB_EDGE=0.0.0.0:8443` in its environment.
|
|
|
|
**Router (OpenWrt at 10.0.0.1).** Forward WAN tcp/8443 to the hub only:
|
|
|
|
```sh
|
|
uci add firewall redirect
|
|
uci set firewall.@redirect[-1].name='aihub-edge'
|
|
uci set firewall.@redirect[-1].src='wan'
|
|
uci set firewall.@redirect[-1].src_dport='8443'
|
|
uci set firewall.@redirect[-1].dest='lan'
|
|
uci set firewall.@redirect[-1].dest_ip='10.0.0.165'
|
|
uci set firewall.@redirect[-1].dest_port='8443'
|
|
uci set firewall.@redirect[-1].proto='tcp'
|
|
uci commit firewall && /etc/init.d/firewall reload
|
|
```
|
|
|
|
Never forward the nodes' ports or :8384 (tunnels).
|
|
|
|
**Address.** For a dynamic home IP, set up DDNS on the router
|
|
(`opkg install ddns-scripts` plus a provider package; LuCI → Services →
|
|
Dynamic DNS) for a hostname of your choice, or use a static IP.
|
|
|
|
**Devices.**
|
|
- On the LAN, issue a credential per device:
|
|
`ai-fleet issue <phone-name> device --out …`.
|
|
- Enrolment should carry it with the hub's address and certificate
|
|
fingerprint, e.g. a QR code with
|
|
`makepad-hub://<host>:8443?fp=<sha256>&cred=<credential>`.
|
|
- On the edge, devices must pin the hub certificate **strictly**: refuse a
|
|
change, unlike the LAN's warn-and-continue rule, because a device on the
|
|
internet has no other way to tell the hub apart. This is still to be
|
|
decided and built, with the enrolment UI.
|
|
|
|
**Checks before opening the port.** From outside the LAN (a phone hotspot):
|
|
- the forwarded port answers TLS only;
|
|
- a request without a credential gets 401, and repeated failures ban that
|
|
address;
|
|
- only device routes work.
|