5f81ebd2a8
LlamaSession gains append_image_embeddings: precomputed vision embeddings prefill through a second Embeddings-input graph spec sharing the same cache tensors (SessionGraphParams grows an embeddings_input discriminant). HybridDecodeBatchLayout gains an optional pre-expanded rope_positions override, threaded to encode_rope_positions — cache indices and attention masks stay linear while image tokens get qwen-vl 2D positions [pos0, pos0+y, pos0+x, 0] and the span advances rope position by max(w, h). Text after an image continues from the shifted position; pure text paths are byte-identical to before (verified via llama-generate). vlm-probe runs the whole thing: ppm -> vision tower -> chatml with vision_start/end -> greedy. Output is token-for-token identical to llama-mtmd-cli --temp 0 on both test images (radar scene description and 'A red circle.'). 229-token mixed prefill 0.98s, generation 27.9 tok/s on the 9B UD-Q4_K_XL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
37 lines
669 B
TOML
37 lines
669 B
TOML
[package]
|
|
name = "makepad-llama"
|
|
version = "0.1.0"
|
|
edition = "2021"
|
|
description = "Rust llama.cpp port scaffold built on makepad-ggml"
|
|
license = "MIT"
|
|
|
|
[dependencies]
|
|
makepad-ggml = { path = "../ggml" }
|
|
|
|
[[bin]]
|
|
name = "llama-load"
|
|
path = "src/bin/llama_load.rs"
|
|
|
|
[[bin]]
|
|
name = "llama-compare"
|
|
path = "src/bin/llama_compare.rs"
|
|
|
|
[[bin]]
|
|
name = "llama-generate"
|
|
path = "src/bin/llama_generate.rs"
|
|
|
|
[[bin]]
|
|
name = "llama-tokenize"
|
|
path = "src/bin/llama_tokenize.rs"
|
|
|
|
[[bin]]
|
|
name = "llama-batch-probe"
|
|
path = "src/bin/llama_batch_probe.rs"
|
|
|
|
[[bin]]
|
|
name = "vlm-vision-probe"
|
|
path = "src/bin/vlm_vision_probe.rs"
|
|
|
|
[[bin]]
|
|
name = "vlm-probe"
|
|
path = "src/bin/vlm_probe.rs"
|