# Local AI: run Makepad Code on a local model

Purpose: from a fresh checkout and no model, end with `makepad-code`
answering on a model running on this machine. Each step: command, what
success looks like, what to do if it fails. Run the steps in order.

## Pick a profile

| Machine | Model | Notes |
|---|---|---|
| Mac, 16 GB unified memory (M1 and later) | `qwen3.5-9b` | UNVERIFIED: not run on a 16 GB Mac; quant, context and memory use not measured |
| GPU with 24 GB+ VRAM (Linux/Windows, NVIDIA) | `qwen3.8-27b` | 17.1 GB Q4_K_M + 0.9 GB vision projector; 19.2 GB VRAM resident once loaded (32768 context, q8 KV). Verified end to end on an RTX PRO 6000 (Linux). Windows UNVERIFIED |
| No usable GPU | none | use a hub on the LAN: `MAKEPAD_AI_HUB=host:port makepad-code` |

The hub only serves models that are installed; install one model and that
is the one it loads. It never picks a model you did not install.

## 0. Prerequisites

- Rust via rustup (`rustup default stable`). Check: `cargo --version`.
- Disk: ~25 GB free for the 27B (17 GB of weights on disk, measured, plus about 1 GB of build output), the 9B is UNVERIFIED (not measured).
  Check: `df -h ~`.
- Linux/Windows with NVIDIA: the CUDA toolkit, and its root in
  `MAKEPAD_CUDA_ROOT` (for example `export MAKEPAD_CUDA_ROOT=/opt/cuda`).
  Without it the build succeeds but prints
  `makepad-ai-cuda: building WITHOUT CUDA — GPU inference is unavailable`.
  Add `MAKEPAD_GGML_REQUIRE_CUDA=1` to make that an error instead.
  Only the hub builds the kernels: they are makepad-ai-cuda's `cuda` feature, on in
  `makepad-app-ai-hub` (default) and the AI Hub app's `node`. Every other app is CPU-only;
  a model crate's own bin or test gets CUDA with `--features makepad-ai-cuda/cuda`.
- macOS: nothing extra (Metal).

## 1. Build

```sh
cd makepad
cargo build --release -p makepad-app-ai-hub -p makepad-code
cargo build --release -p makepad-ai-hub --features local --bin hub-install
export PATH="$PWD/target/release:$PATH"
```

Success: `target/release/makepad-app-ai-hub`, `makepad-code` and
`hub-install` exist (about 2.5 min on a 32-core Linux box with the CUDA kernels; the 23 kernels build twice, about 1 min each).
If `makepad-ai-cuda: building WITHOUT CUDA` appears on an NVIDIA machine,
set `MAKEPAD_CUDA_ROOT` (step 0) and build again.

## 2. Install the model

```sh
hub-install qwen3.8-27b --accept-license      # or qwen3.5-9b on a 16 GB Mac
```

`--accept-license` accepts the model's licence (Apache-2.0 for both). Files go
to `~/.makepad/weights` (the one place every Makepad hub on the machine reads) (`--weights-dir DIR` to change). Success: a `verified <file>` line per file, then the installed paths
(`llm-gguf:` and `mmproj:`). The 18 GB download takes from a few minutes to
over an hour depending on the link; progress lines print as it goes. A failed or
interrupted download can be re-run; finished files are kept.

## 3. Start the hub

```sh
makepad-app-ai-hub --machine &
echo $! > ~/hub.pid
```

Success: output ends with `listening on http 127.0.0.1:8765` and
`machine node: registered ~/.makepad/run/<key>.node (ttl 15m idle)`.
The machine node binds loopback only. It loads the model on the first request,
not at start, and exits by itself after 15 idle minutes
(`MAKEPAD_AI_HUB_MACHINE_TTL_MIN` changes that).

If it prints `Cannot bind http server port`, another hub already holds 8765
on this machine: use that one, or start this one with `--port 8799`
(`makepad-code` finds it through the registered node entry).

## 4. Check the model

```sh
cargo build --release -p makepad-ai-hub --bin hub-control
hub-control http://127.0.0.1:8765 get
```

Before the first request the `qwen3.8-27b` entry in `models` says
`"state":"unloaded"` and `llm.model` is `null`. That is normal. After step 5
it says `"state":"loaded"` with `resident_mb` about 19168, and
`nvidia-smi` shows about 19.7 GB more used than before. `curl
http://127.0.0.1:8765/health` also lists `models_loaded`.
(Use the port the hub printed.)

## 5. Run Makepad Code

```sh
cd ~/my-project
makepad-code -p "list the files here and say what each one is"
```

Success: a `model: qwen3.8-27b on 127.0.0.1` line, tool lines (`+ ls .`), then
an answer. The first request pays the model load. Run `makepad-code` with no
`-p` for the interactive session.

If it prints `No local AI model is installed yet` right after step 2 passed,
the hub is not running (or not on the port in its node entry): the message is
misleading in that case. Start it (step 3). Makepad Code does not start the
hub itself. The `[hub-chat] no model:` line above the error names the address
it tried.

## 6. Stop the hub

```sh
kill $(cat ~/hub.pid)
```

VRAM goes back to what it was before step 5 within seconds. A stale
`~/.makepad/run/<key>.node` file stays behind and is harmless; the next hub
start replaces it. Kill by the recorded PID only, never by process name.