makepad/libs/asset/store/tests/common/mod.rs
Admin e37c263b9c ai backbone: the hub era — makepad_ai deleted, every backend is a hub pipe; machine residency elections, job leases, ETA placement; the store only stores; creator pipelines run in the app (aicore)
Squashed from work:
- asset-ai: FastH3 4-step fast video backend; clip keyframes on the wire
- asset-ui: loop video chains — text→image→video that ends where it began
- h3: safetensors -> pruned-Q4_K GGUF quantizer for the 24GB DiT tiers
- h3_quant_gguf verify: row-error gates calibrated to the measured Q4_K floor
- asset-ai realtime: the feedback loop — the source anchors, the drifted frame inits
- asset-ai realtime: a feedback loop survives a resize and travels by default
- asset-ai realtime: the feedback loop frees itself from the feed handshake and pauses for its listener
- asset-ai realtime: the outbound encode leaves the loop's critical path
- asset-ai ocr: the ocr domain — Chandra 2 at page resolution, and the tower goes planner-owned
- llm slots: a lane can hold an image span — embedding prefill and a rope cursor of its own
- vision tower on CUDA: the encode leg gets its two missing kernels
- llm/ocr: one M-RoPE grid encoder for both image paths, and a livelock made an error
- vision tower on CUDA: the f16 GEMM keeps the precision it was throwing away
- live: a feed that moves box takes its trip with it — one seed image
- vision tower on CUDA: the tiled attention becomes bit-exact, and tensor cores go
- llm prefill on CUDA: the MMA attention kernel gets the tile a 4-to-1 model needs
- asset-ai ocr: the CUDA encode lane joins the integration — vision-parity sits beside run's three arms, and the kernels
- Merge branch 'ocr-perf-integration' into work
- asset-ai: the live anchor can follow the trip, and text leaves the 5090
- asset-ai: the camera moves the world, and the world starts still
- asset-import: the EA strategy classics, in the one 2D contract
- rtsmap: one seeded generator for tiled strategy maps
- asset-ui: one card for the strategy classics, with a pack dropdown
- asset-ai: music3 reference-audio path, ocr/h3 backends, registry
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- cnc: tiberium is twelve growth frames, not twelve empty variants
- platform: native file and save dialogs, in-house on all three desktops
- chat: the scan holds out for a lane home
- chat: a full home queues you — take the free lane
- chat: the preload has a percentage, and the boundless cap stops showing
- llm cuda: the 32x2 attention tile — even GQA ratios stay on MMA
- sa3 gets a bake path: the sfx model's tables precomputed by a diffusion-side bin
- sqlite_query: anti-join regression test
- td import: HARV's second frame block is its harvesting cycle, not a turret
- asset-ui: sprite enhancement runs on the 32B dev DiT — distillation, not the prompt, was the ceiling
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- asset-ui: test health fixtures grow the realtime field they were born without
- ai-hub: one home at ~/.makepad — weights/ run/ cache/ logs/, the service cache migrates from ai_content by a single re
- ai-hub: subprocess workers die with the node — process groups everywhere, PDEATHSIG on linux, one KILL_ON_JOB_CLOSE Jo
- ai-hub: the hub object — AiHub::in_process, pipes vocabulary, and the local LLM engine generalized out of mpfiles (aic
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- ai-hub: the machine layer — node entries, the 0600 machine token, and the residency election that IS the lock (aicore
- ai-hub: MPHUB1 — the fabric beacon only dedicated nodes can send (aicore §4)
- ai-hub: job leases — work lives only while it is renewed (aicore §8)
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- ai-hub: RAM residency facts — the CPU-side twin of residency.rs (aicore §3)
- ai-hub: ETA placement primitives — relative GPU throughput, the four-term estimate, and an observable breakdown (aicor
- ai-hub: leases go live on the wire — origin fields on submit, /job/<id>/keepalive, /bye, and the reaper that cancels w
- ai-hub: the chat providers move in — fleet qwen, openai, grok, claude/codex/grok CLIs, the responses driver, and the w
- asset-creator: the engine — one pipeline run against the hub, deps-gated, spliced, cancellable, resumable-by-construct
- ai-hub: the machine node mode — --machine binds loopback, registers in ~/.makepad/run, and exits on its own once idle
- asset-creator: makepad-creator-run — the detached client for runs that must outlive a window (aicore §9)
- ai-hub: a native Claude Messages-API provider — API-key or Claude Code OAuth, bounded SSE streaming, injected tools (a
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- asset-creator: the preset tables move in — fifteen chain-policy constants shared by every creator app (aicore §9 / P6)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- ai-hub: loads hold the machine residency election — set_model_state claims on Loaded and publishes the service port (a
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- ai-hub: pick_for_domain_eta — ETA-ranked placement over the shared hard-filter core (aicore §6 / P4)
- asset-creator: the engine picks a provider per stage at dispatch time — a chain's later stages see fresh fleet state (
- ai-hub: the fabric secret gates the service HTTP surface — bearer on everything but /health and the ticketed peer path
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- chat-ui: the session runs in the app — no broker anywhere on the chat path (aicore P8 / F5)
- asset-store: assets.query is a first-class query endpoint — the bounded SQL surface outlives the broker (aicore P8 / F
- asset-creator: CreatorTools — the chat tool pack for a store that only stores (aicore §9 / P8)
- asset-store: the shrink — the store stores (aicore P7)
- importer + asset-server host: the coordination era ends (aicore P7)
- store config purge + asset-ui goes fleet-direct; the derive protocol gets its route proof (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- ai-hub: 0.3.0 — the health version says which era a node runs
- ai-hub: the default fleet is 'gen' — apps hear the LAN without env plumbing
- ai-hub: the preload note percents the prefill, not the job bar
- ai-hub: conversations keep their KV — the wire mirror, the lane identity, the in-turn dynamic context (aicore §7)
- ai-hub: an open-think model is thinking from its first token
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades

Co-authored-by: Claude <info@makepad.nl>
2026-09-01 16:46:31 +02:00

510 lines
17 KiB
Rust

//! Shared test fixtures: unique on-disk roots and minimal valid manifests
//! whose blob references point at real bytes (the catalog refuses dangling
//! references, so fixtures upload what they reference).
#![allow(dead_code)]
use makepad_asset_store::variants::JobId;
use makepad_asset_store::*;
use makepad_asset_data::*;
use std::path::PathBuf;
use std::sync::atomic::{AtomicU64, Ordering};
/// Raw libsqlite3 shim: open a database file, run SQL, collect the first
/// column of every returned row. Panics on any failure — fixture SQL must
/// work. Used to fabricate byte-real legacy databases for migration tests and
/// to inject mid-transaction faults (triggers) for rollback tests.
pub mod raw {
use std::ffi::{CStr, CString};
use std::os::raw::{c_char, c_int, c_void};
use std::path::Path;
enum Sqlite3 {}
type ExecCallback =
unsafe extern "C" fn(*mut c_void, c_int, *mut *mut c_char, *mut *mut c_char) -> c_int;
#[link(name = "sqlite3")]
extern "C" {
fn sqlite3_open_v2(
filename: *const c_char,
db: *mut *mut Sqlite3,
flags: c_int,
vfs: *const c_char,
) -> c_int;
fn sqlite3_close(db: *mut Sqlite3) -> c_int;
fn sqlite3_exec(
db: *mut Sqlite3,
sql: *const c_char,
callback: Option<ExecCallback>,
arg: *mut c_void,
errmsg: *mut *mut c_char,
) -> c_int;
fn sqlite3_free(ptr: *mut c_void);
}
unsafe extern "C" fn capture_last_column(
arg: *mut c_void,
ncols: c_int,
vals: *mut *mut c_char,
_names: *mut *mut c_char,
) -> c_int {
if ncols > 0 {
let out = &mut *(arg as *mut Vec<String>);
let v = *vals.offset((ncols - 1) as isize);
out.push(if v.is_null() {
String::new()
} else {
CStr::from_ptr(v).to_string_lossy().into_owned()
});
}
0
}
unsafe extern "C" fn capture_first_column(
arg: *mut c_void,
ncols: c_int,
vals: *mut *mut c_char,
_names: *mut *mut c_char,
) -> c_int {
if ncols > 0 {
let out = &mut *(arg as *mut Vec<String>);
let v = *vals;
out.push(if v.is_null() {
String::new()
} else {
CStr::from_ptr(v).to_string_lossy().into_owned()
});
}
0
}
/// Like [`exec`] but collects the LAST column of every row — which is
/// `detail` for `EXPLAIN QUERY PLAN`, the only column that names the
/// index a statement actually uses.
pub fn exec_last(path: &Path, sql: &str) -> Vec<String> {
exec_with(path, sql, capture_last_column)
}
pub fn exec(path: &Path, sql: &str) -> Vec<String> {
exec_with(path, sql, capture_first_column)
}
fn exec_with(path: &Path, sql: &str, cb: ExecCallback) -> Vec<String> {
let cpath = CString::new(path.as_os_str().as_encoded_bytes()).unwrap();
let csql = CString::new(sql).unwrap();
let mut db: *mut Sqlite3 = std::ptr::null_mut();
// READWRITE | CREATE | FULLMUTEX, as in src/sqlite.rs.
let flags = 0x0000_0002 | 0x0000_0004 | 0x0001_0000;
let rc = unsafe { sqlite3_open_v2(cpath.as_ptr(), &mut db, flags, std::ptr::null()) };
assert_eq!(rc, 0, "fixture open failed for {path:?}");
let mut rows: Vec<String> = Vec::new();
let mut errmsg: *mut c_char = std::ptr::null_mut();
let rc = unsafe {
sqlite3_exec(
db,
csql.as_ptr(),
Some(cb),
&mut rows as *mut Vec<String> as *mut c_void,
&mut errmsg,
)
};
let msg = if errmsg.is_null() {
String::new()
} else {
unsafe {
let m = CStr::from_ptr(errmsg).to_string_lossy().into_owned();
sqlite3_free(errmsg as *mut c_void);
m
}
};
unsafe { sqlite3_close(db) };
assert_eq!(rc, 0, "fixture sql failed: {msg}");
rows
}
}
/// The search index generation as stored, via a raw side-channel read.
pub fn read_generation(db: &std::path::Path) -> i64 {
raw::exec(db, "SELECT generation FROM search_state WHERE id = 1")
.remove(0)
.parse()
.expect("generation is an integer")
}
static DIR_COUNTER: AtomicU64 = AtomicU64::new(0);
/// Deterministic test clock origin; tests advance it explicitly.
pub const NOW: u64 = 1_700_000_000_000;
/// A fresh unique root per call: pid + counter + name, under the OS temp dir.
pub fn test_root(name: &str) -> PathBuf {
let n = DIR_COUNTER.fetch_add(1, Ordering::Relaxed);
std::env::temp_dir().join(format!(
"mp_asset_server_test_{}_{}_{}",
std::process::id(),
n,
name
))
}
pub fn open_core(name: &str) -> (PathBuf, AssetServerCore) {
let root = test_root(name);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
(root, core)
}
pub fn asset_id_n(n: u8) -> AssetId {
AssetId::from_bytes([n; 16])
}
pub fn game_id_n(n: u8) -> GameId {
GameId::from_bytes([n; 16])
}
pub fn jid(n: u8) -> JobId {
JobId([n; 16])
}
pub fn pid_n(n: u8) -> PrincipalId {
PrincipalId([n; 16])
}
/// Full v3 rights fixtures: pinned terms digest, upstream provenance, and
/// source-determined policy.
pub fn cc0_rights(credits: &str, source: &str) -> Rights {
Rights {
license: "CC0-1.0".into(),
license_revision: String::new(),
terms_digest: Some(sha256(b"CC0-1.0 legal text")),
terms_url: "https://creativecommons.org/publicdomain/zero/1.0/".into(),
credits: credits.into(),
source: source.into(),
source_archive: None,
redistribution: Redistribution::Allowed,
derivatives: DerivativePolicy::Allowed,
}
}
pub fn kenney_terms() -> Rights {
Rights {
source_archive: Some(sha256(b"space-kit-1.0.zip")),
..cc0_rights("Kenney (kenney.nl)", "https://kenney.nl/assets/space-kit")
}
}
/// Minimal valid mesh-bearing manifest: one render GLB plus the mandatory
/// thumbnail, both referencing the digests of the supplied bytes.
pub fn prop_manifest(asset_id: AssetId, glb: &[u8], thumb: &[u8]) -> AssetManifest {
AssetManifest {
asset_id,
kind: AssetKind::Prop,
files: vec![AssetFile {
role: FileRole::RenderGlb,
tier: DeviceTier::Any,
lod: 0,
media: MediaType::Glb,
blob: BlobId::hash_of(glb),
byte_len: glb.len() as u64,
dims: None,
}],
dependencies: vec![],
thumbnail: Some(ThumbnailMeta {
blob: BlobId::hash_of(thumb),
media: ThumbnailMedia::Png,
width: 512,
height: 512,
byte_len: thumb.len() as u64,
views: Vec::new(),
}),
metrics: Metrics {
total_bytes: glb.len() as u64 + thumb.len() as u64,
triangles: 12,
vertices: 8,
joints: 0,
clips: 0,
max_texture_dim: 0,
media_millis: 0,
},
coordinate_system: CoordinateSystem {
units_per_meter: 1.0,
up: Axis::YPos,
forward: Axis::ZNeg,
pivot: Pivot::Origin,
},
bounds: Bounds {
min: Vec3::new(-1.0, -1.0, -1.0),
max: Vec3::new(1.0, 1.0, 1.0),
},
anchors: vec![],
capabilities: Capabilities {
rigged: false,
animated: false,
collidable: false,
loopable: false,
spawnable: false,
},
spawn_recipe: None,
provenance: None,
rights: cc0_rights("test", ""),
}
}
/// Upload blobs, register, stage and publish one prop in one call.
pub fn publish_prop(
core: &AssetServerCore,
ns: &str,
id_byte: u8,
glb: &[u8],
thumb: &[u8],
now: u64,
) -> (AssetId, AssetRevisionId) {
let id = asset_id_n(id_byte);
core.put_blob(glb, now).unwrap();
core.put_blob(thumb, now).unwrap();
core.catalog().register_asset(&id, ns, now).unwrap();
let manifest = prop_manifest(id, glb, thumb);
let bytes = manifest.to_canonical_bytes().unwrap();
let rev = core.catalog().stage_asset_revision(&bytes, now).unwrap();
core.catalog().publish_asset(&id, &rev, now).unwrap();
(id, rev)
}
/// Fixed pack source bytes for the Kenney-style import fixture. Content is
/// deterministic so import/derivation identities are identical across clean
/// servers.
pub const PACK_GLB: &[u8] = b"KENNEY-WATCHTOWER-GLB-v1";
pub const PACK_COLLIDER: &[u8] = b"KENNEY-WATCHTOWER-COLLIDER-v1";
pub const PACK_PREVIEW: &[u8] = b"KENNEY-WATCHTOWER-PREVIEW-PNG-v1";
pub const PACK_TEXTURE: &[u8] = b"KENNEY-HULL-PANEL-PNG-v1";
/// CC-BY attribution terms for hostile-rights tests.
pub fn cc_by_terms() -> Rights {
Rights {
license: "CC-BY-4.0".into(),
license_revision: "4.0".into(),
terms_digest: Some(sha256(b"CC-BY-4.0 legal text")),
terms_url: "https://creativecommons.org/licenses/by/4.0/legalcode".into(),
credits: "Example Author".into(),
source: "https://example.com/pack".into(),
source_archive: Some(sha256(b"example-pack.zip")),
redistribution: Redistribution::AttributionRequired,
derivatives: DerivativePolicy::AttributionRequired,
}
}
/// A registered collection with explicit terms under an explicit id.
pub fn collection_with_terms(id: &str, terms: Rights) -> SourceCollection {
SourceCollection {
id: id.into(),
title: "Terms fixture".into(),
origin: SourceOrigin::Upload,
terms,
}
}
/// The Kenney pack re-homed under another registered collection: same
/// entries, different source identity and claimed rights.
pub fn pack_with_terms(collection: &SourceCollection) -> ImportManifest {
let mut pack = kenney_pack("1.0");
pack.source_id = collection.id.clone();
pack.source_collection = collection.digest().unwrap();
pack.rights = collection.terms.clone();
pack.canonicalize();
pack
}
/// The approved Kenney source collection fixture. Its terms are the
/// authoritative rights of everything imported under it.
pub fn kenney_collection() -> SourceCollection {
SourceCollection {
id: "kenney".into(),
title: "Kenney game assets".into(),
origin: SourceOrigin::Upload,
terms: kenney_terms(),
}
}
/// A tiny pinned two-asset Kenney-style pack over the fixed source bytes:
/// a mesh prop (render GLB + collider + preview thumbnail) and a texture.
pub fn kenney_pack(version: &str) -> ImportManifest {
let mut manifest = ImportManifest {
source_collection: kenney_collection().digest().unwrap(),
source_id: "kenney".into(),
pack_name: "space-kit".into(),
pack_version: version.into(),
policy_version: IMPORT_ASSET_ID_POLICY_V1,
assets: vec![
ImportAsset {
key: "models/watchtower".parse().unwrap(),
kind: AssetKind::Prop,
files: vec![
ImportFile {
path: "models/watchtower.glb".into(),
file: AssetFile {
role: FileRole::RenderGlb,
tier: DeviceTier::Any,
lod: 0,
media: MediaType::Glb,
blob: BlobId::hash_of(PACK_GLB),
byte_len: PACK_GLB.len() as u64,
dims: None,
},
},
ImportFile {
path: "colliders/watchtower.bin".into(),
file: AssetFile {
role: FileRole::Collider,
tier: DeviceTier::Any,
lod: 0,
media: MediaType::Bin,
blob: BlobId::hash_of(PACK_COLLIDER),
byte_len: PACK_COLLIDER.len() as u64,
dims: None,
},
},
],
thumbnail: Some(ImportThumbnail {
path: "previews/watchtower.png".into(),
meta: ThumbnailMeta {
blob: BlobId::hash_of(PACK_PREVIEW),
media: ThumbnailMedia::Png,
width: 512,
height: 512,
byte_len: PACK_PREVIEW.len() as u64,
views: Vec::new(),
},
}),
metrics: Metrics {
total_bytes: (PACK_GLB.len() + PACK_COLLIDER.len() + PACK_PREVIEW.len())
as u64,
triangles: 500,
vertices: 300,
joints: 0,
clips: 0,
max_texture_dim: 512,
media_millis: 0,
},
coordinate_system: CoordinateSystem {
units_per_meter: 1.0,
up: Axis::YPos,
forward: Axis::ZNeg,
pivot: Pivot::BoundsBottom,
},
bounds: Bounds {
min: Vec3::new(-1.0, 0.0, -1.0),
max: Vec3::new(1.0, 3.0, 1.0),
},
anchors: vec![],
capabilities: Capabilities {
collidable: true,
..Default::default()
},
spawn_recipe: None,
},
ImportAsset {
key: "textures/hull-panel".parse().unwrap(),
kind: AssetKind::Texture,
files: vec![ImportFile {
path: "textures/hull_panel.png".into(),
file: AssetFile {
role: FileRole::Texture,
tier: DeviceTier::Any,
lod: 0,
media: MediaType::Png,
blob: BlobId::hash_of(PACK_TEXTURE),
byte_len: PACK_TEXTURE.len() as u64,
dims: Some(ImageDims {
width: 2048,
height: 2048,
}),
},
}],
thumbnail: None,
metrics: Metrics {
total_bytes: PACK_TEXTURE.len() as u64,
max_texture_dim: 2048,
..Default::default()
},
coordinate_system: CoordinateSystem {
units_per_meter: 1.0,
up: Axis::YPos,
forward: Axis::ZNeg,
pivot: Pivot::Origin,
},
bounds: Bounds {
min: Vec3::ZERO,
max: Vec3::ONE,
},
anchors: vec![],
capabilities: Capabilities::default(),
spawn_recipe: None,
},
],
rights: kenney_terms(),
};
manifest.canonicalize();
manifest
}
/// Upload the pack's source bytes, register the collection, and run the
/// import. Returns the report.
pub fn run_kenney_import(core: &AssetServerCore, version: &str, now: u64) -> ImportReport {
for bytes in [PACK_GLB, PACK_COLLIDER, PACK_PREVIEW, PACK_TEXTURE] {
core.put_blob(bytes, now).unwrap();
}
let collection_bytes = kenney_collection().to_canonical_bytes().unwrap();
core.imports().register_source(&collection_bytes, now).unwrap();
let manifest_bytes = kenney_pack(version).to_canonical_bytes().unwrap();
core.imports().run_import(&manifest_bytes, now).unwrap()
}
/// Minimal valid game revision manifest over real uploaded bytes.
pub fn game_manifest(
game_id: GameId,
splash: &[u8],
toml: &[u8],
lock_bytes: &[u8],
thumb: &[u8],
) -> GameRevisionManifest {
GameRevisionManifest {
game_id,
name: "Test Game".into(),
description: "integration fixture".into(),
author: "rik".into(),
splash_blob: BlobId::hash_of(splash),
manifest_blob: BlobId::hash_of(toml),
lock_blob: BlobId::hash_of(lock_bytes),
thumbnail: ThumbnailMeta {
blob: BlobId::hash_of(thumb),
media: ThumbnailMedia::Png,
width: 512,
height: 512,
byte_len: thumb.len() as u64,
views: Vec::new(),
},
catalog_snapshot: None,
search_algorithm_version: 1,
engine_version: 1,
protocol_version: 1,
splash_byte_len: splash.len() as u64,
}
}
/// Canonical lock pinning the given published refs under fixed aliases.
pub fn lock_for(game_id: GameId, refs: &[(&str, AssetRevisionRef)]) -> Vec<u8> {
let mut lock = ContentLock {
game_id,
entries: refs
.iter()
.map(|(alias, r)| LockEntry {
alias: alias.parse().unwrap(),
asset_id: r.asset_id,
revision: r.revision,
})
.collect(),
closure: refs.iter().map(|(_, r)| *r).collect(),
variant_sets: vec![],
};
lock.canonicalize();
lock.to_canonical_bytes().unwrap()
}