makepad/libs/asset/store/tests/migration.rs
Admin e37c263b9c ai backbone: the hub era — makepad_ai deleted, every backend is a hub pipe; machine residency elections, job leases, ETA placement; the store only stores; creator pipelines run in the app (aicore)
Squashed from work:
- asset-ai: FastH3 4-step fast video backend; clip keyframes on the wire
- asset-ui: loop video chains — text→image→video that ends where it began
- h3: safetensors -> pruned-Q4_K GGUF quantizer for the 24GB DiT tiers
- h3_quant_gguf verify: row-error gates calibrated to the measured Q4_K floor
- asset-ai realtime: the feedback loop — the source anchors, the drifted frame inits
- asset-ai realtime: a feedback loop survives a resize and travels by default
- asset-ai realtime: the feedback loop frees itself from the feed handshake and pauses for its listener
- asset-ai realtime: the outbound encode leaves the loop's critical path
- asset-ai ocr: the ocr domain — Chandra 2 at page resolution, and the tower goes planner-owned
- llm slots: a lane can hold an image span — embedding prefill and a rope cursor of its own
- vision tower on CUDA: the encode leg gets its two missing kernels
- llm/ocr: one M-RoPE grid encoder for both image paths, and a livelock made an error
- vision tower on CUDA: the f16 GEMM keeps the precision it was throwing away
- live: a feed that moves box takes its trip with it — one seed image
- vision tower on CUDA: the tiled attention becomes bit-exact, and tensor cores go
- llm prefill on CUDA: the MMA attention kernel gets the tile a 4-to-1 model needs
- asset-ai ocr: the CUDA encode lane joins the integration — vision-parity sits beside run's three arms, and the kernels
- Merge branch 'ocr-perf-integration' into work
- asset-ai: the live anchor can follow the trip, and text leaves the 5090
- asset-ai: the camera moves the world, and the world starts still
- asset-import: the EA strategy classics, in the one 2D contract
- rtsmap: one seeded generator for tiled strategy maps
- asset-ui: one card for the strategy classics, with a pack dropdown
- asset-ai: music3 reference-audio path, ocr/h3 backends, registry
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- cnc: tiberium is twelve growth frames, not twelve empty variants
- platform: native file and save dialogs, in-house on all three desktops
- chat: the scan holds out for a lane home
- chat: a full home queues you — take the free lane
- chat: the preload has a percentage, and the boundless cap stops showing
- llm cuda: the 32x2 attention tile — even GQA ratios stay on MMA
- sa3 gets a bake path: the sfx model's tables precomputed by a diffusion-side bin
- sqlite_query: anti-join regression test
- td import: HARV's second frame block is its harvesting cycle, not a turret
- asset-ui: sprite enhancement runs on the 32B dev DiT — distillation, not the prompt, was the ceiling
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- asset-ui: test health fixtures grow the realtime field they were born without
- ai-hub: one home at ~/.makepad — weights/ run/ cache/ logs/, the service cache migrates from ai_content by a single re
- ai-hub: subprocess workers die with the node — process groups everywhere, PDEATHSIG on linux, one KILL_ON_JOB_CLOSE Jo
- ai-hub: the hub object — AiHub::in_process, pipes vocabulary, and the local LLM engine generalized out of mpfiles (aic
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- ai-hub: the machine layer — node entries, the 0600 machine token, and the residency election that IS the lock (aicore
- ai-hub: MPHUB1 — the fabric beacon only dedicated nodes can send (aicore §4)
- ai-hub: job leases — work lives only while it is renewed (aicore §8)
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- ai-hub: RAM residency facts — the CPU-side twin of residency.rs (aicore §3)
- ai-hub: ETA placement primitives — relative GPU throughput, the four-term estimate, and an observable breakdown (aicor
- ai-hub: leases go live on the wire — origin fields on submit, /job/<id>/keepalive, /bye, and the reaper that cancels w
- ai-hub: the chat providers move in — fleet qwen, openai, grok, claude/codex/grok CLIs, the responses driver, and the w
- asset-creator: the engine — one pipeline run against the hub, deps-gated, spliced, cancellable, resumable-by-construct
- ai-hub: the machine node mode — --machine binds loopback, registers in ~/.makepad/run, and exits on its own once idle
- asset-creator: makepad-creator-run — the detached client for runs that must outlive a window (aicore §9)
- ai-hub: a native Claude Messages-API provider — API-key or Claude Code OAuth, bounded SSE streaming, injected tools (a
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- asset-creator: the preset tables move in — fifteen chain-policy constants shared by every creator app (aicore §9 / P6)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- ai-hub: loads hold the machine residency election — set_model_state claims on Loaded and publishes the service port (a
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- ai-hub: pick_for_domain_eta — ETA-ranked placement over the shared hard-filter core (aicore §6 / P4)
- asset-creator: the engine picks a provider per stage at dispatch time — a chain's later stages see fresh fleet state (
- ai-hub: the fabric secret gates the service HTTP surface — bearer on everything but /health and the ticketed peer path
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- chat-ui: the session runs in the app — no broker anywhere on the chat path (aicore P8 / F5)
- asset-store: assets.query is a first-class query endpoint — the bounded SQL surface outlives the broker (aicore P8 / F
- asset-creator: CreatorTools — the chat tool pack for a store that only stores (aicore §9 / P8)
- asset-store: the shrink — the store stores (aicore P7)
- importer + asset-server host: the coordination era ends (aicore P7)
- store config purge + asset-ui goes fleet-direct; the derive protocol gets its route proof (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- ai-hub: 0.3.0 — the health version says which era a node runs
- ai-hub: the default fleet is 'gen' — apps hear the LAN without env plumbing
- ai-hub: the preload note percents the prefill, not the job bar
- ai-hub: conversations keep their KV — the wire mirror, the lane identity, the in-turn dynamic context (aicore §7)
- ai-hub: an open-think model is thinking from its first token
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades

Co-authored-by: Claude <info@makepad.nl>
2026-09-01 16:46:31 +02:00

624 lines
26 KiB
Rust

//! Schema migration: fresh roots land on the current version, v1 roots (with
//! and without the ad-hoc search tables that predate v2) migrate forward with
//! their data intact, and versions this build does not speak refuse to open.
//!
//! Fixture databases are fabricated through the minimal raw-sqlite shim in
//! tests/common (the test-side twin of src/sqlite.rs) so the legacy layouts
//! are byte-real, not simulated through current-code hooks.
/// The job queue left the store (aicore P7); migration tests still build
/// faithful HISTORICAL roots, so its DDL is pinned here as data.
const HISTORICAL_JOBS_SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS jobs(
job_id BLOB PRIMARY KEY,
parent_job BLOB,
kind TEXT NOT NULL,
payload BLOB NOT NULL,
state TEXT NOT NULL CHECK(state IN ('pending','running','succeeded','failed','cancelled')),
attempts_used INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL,
priority INTEGER NOT NULL,
not_before_ms INTEGER NOT NULL,
created_ms INTEGER NOT NULL,
updated_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS jobs_by_state ON jobs(state, not_before_ms);
CREATE INDEX IF NOT EXISTS jobs_by_parent ON jobs(parent_job);
CREATE TABLE IF NOT EXISTS job_deps(
job_id BLOB NOT NULL,
depends_on BLOB NOT NULL,
PRIMARY KEY(job_id, depends_on)
);
CREATE TABLE IF NOT EXISTS job_attempts(
job_id BLOB NOT NULL,
attempt INTEGER NOT NULL,
worker TEXT NOT NULL,
started_ms INTEGER NOT NULL,
ended_ms INTEGER,
outcome TEXT NOT NULL DEFAULT 'running',
PRIMARY KEY(job_id, attempt)
);
CREATE TABLE IF NOT EXISTS job_leases(
job_id BLOB PRIMARY KEY,
worker TEXT NOT NULL,
attempt INTEGER NOT NULL,
expires_ms INTEGER NOT NULL,
heartbeat_ms INTEGER NOT NULL
);
";
const HISTORICAL_OPERATIONS_SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS operations(
operation_id BLOB PRIMARY KEY,
owner BLOB NOT NULL,
namespace TEXT NOT NULL,
kind TEXT NOT NULL,
def_revision INTEGER NOT NULL,
idempotency_key TEXT NOT NULL,
spec_digest BLOB NOT NULL,
spec BLOB NOT NULL,
state TEXT NOT NULL CHECK(state IN ('queued','succeeded','failed','cancelled')),
round INTEGER NOT NULL DEFAULT 0,
job_id BLOB NOT NULL,
error TEXT,
result_asset BLOB,
result_revision BLOB,
created_ms INTEGER NOT NULL,
updated_ms INTEGER NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS operations_idem
ON operations(owner, namespace, idempotency_key);
CREATE INDEX IF NOT EXISTS operations_by_job ON operations(job_id);
CREATE TABLE IF NOT EXISTS operation_events(
operation_id BLOB NOT NULL,
seq INTEGER NOT NULL,
kind TEXT NOT NULL,
detail TEXT NOT NULL,
created_ms INTEGER NOT NULL,
PRIMARY KEY(operation_id, seq)
);
CREATE TABLE IF NOT EXISTS operation_worker_seen(
kind TEXT PRIMARY KEY,
last_seen_ms INTEGER NOT NULL
);
";
mod common;
use common::*;
use makepad_asset_store::{
auth::AUTH_SCHEMA, catalog::CATALOG_SCHEMA, AssetAnnotation,
AssetServerCore, Budgets, SearchFilters, SearchQuery, SearchViewer, ServerError, ViewerScope,
Visibility, SERVER_SCHEMA_VERSION,
};
use makepad_asset_data::AssetKind;
use std::path::{Path, PathBuf};
/// The v1 search DDL exactly as it shipped: no kind column, no kind index.
const LEGACY_SEARCH_V1: &str = "
CREATE TABLE IF NOT EXISTS search_annotations(
asset_id BLOB PRIMARY KEY,
namespace TEXT NOT NULL,
visibility TEXT NOT NULL CHECK(visibility IN ('public','private')),
owner BLOB,
title TEXT NOT NULL,
description TEXT NOT NULL,
creator TEXT NOT NULL,
generator TEXT NOT NULL,
backend TEXT NOT NULL,
model TEXT NOT NULL,
prompt TEXT NOT NULL,
provenance TEXT NOT NULL,
live INTEGER NOT NULL,
updated_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS search_annotations_by_ns ON search_annotations(namespace);
CREATE TABLE IF NOT EXISTS search_labels(
asset_id BLOB NOT NULL,
kind TEXT NOT NULL CHECK(kind IN ('category','tag')),
label TEXT NOT NULL,
PRIMARY KEY(asset_id, kind, label)
);
CREATE INDEX IF NOT EXISTS search_labels_by_label ON search_labels(kind, label);
CREATE TABLE IF NOT EXISTS search_postings(
term TEXT NOT NULL,
asset_id BLOB NOT NULL,
weight_public INTEGER NOT NULL,
weight_owner INTEGER NOT NULL,
PRIMARY KEY(term, asset_id)
);
CREATE INDEX IF NOT EXISTS search_postings_by_asset ON search_postings(asset_id);
";
/// [1; 16] as a SQLite blob literal — `asset_id_n(1)` in fixture SQL.
const ID1_HEX: &str = "X'01010101010101010101010101010101'";
fn fixture_root(name: &str, sql: &str) -> (PathBuf, PathBuf) {
let root = test_root(name);
std::fs::create_dir_all(&root).unwrap();
let db = root.join("catalog.sqlite3");
raw::exec(&db, sql);
(root, db)
}
fn user_version(db: &Path) -> String {
raw::exec(db, "PRAGMA user_version").remove(0)
}
fn ann(title: &str) -> AssetAnnotation {
AssetAnnotation {
title: title.into(),
description: String::new(),
kind: None,
categories: Vec::new(),
tags: Vec::new(),
creator: String::new(),
owner: None,
generator: String::new(),
backend: String::new(),
model: String::new(),
prompt: String::new(),
provenance: String::new(),
visibility: Visibility::Public,
}
}
fn q(text: &str) -> SearchQuery<'_> {
SearchQuery {
text,
filters: SearchFilters::default(),
expand: false,
page_size: 10,
facets: 0,
}
}
const ANYONE: SearchViewer<'static> = SearchViewer { principal: None, scope: ViewerScope::All };
#[test]
fn fresh_root_lands_at_current_version() {
let root = test_root("fresh_version");
{
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
core.catalog().register_asset(&asset_id_n(1), "rik2", NOW).unwrap();
core.search().set_annotation(&asset_id_n(1), &ann("fresh thing"), NOW).unwrap();
}
assert_eq!(user_version(&root.join("catalog.sqlite3")), SERVER_SCHEMA_VERSION.to_string());
// Reopening a current-version root is a no-op migrate.
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(core.search().search(&q("fresh"), &ANYONE, None).unwrap().total, 1);
}
#[test]
fn presearch_v1_root_migrates_to_current() {
// A faithful pre-search v1 root: catalog/jobs/auth tables only.
let (root, db) = fixture_root(
"presearch_v1",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
PRAGMA user_version=1;"
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
// The migrated root has working search including the kind column.
let mut a = ann("migrated relic");
a.kind = Some(AssetKind::Prop);
core.search().set_annotation(&asset_id_n(1), &a, NOW).unwrap();
let mut query = q("");
query.filters = SearchFilters { kind: Some(AssetKind::Prop), ..Default::default() };
assert_eq!(core.search().search(&query, &ANYONE, None).unwrap().total, 1);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
#[test]
fn legacy_search_v1_root_is_retrofitted_with_data_intact() {
// A v1 root that already carried the ad-hoc search tables (pre-kind),
// with a live-looking annotation, postings and a label in place.
let (root, db) = fixture_root(
"legacy_search_v1",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{LEGACY_SEARCH_V1}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
INSERT INTO search_annotations(asset_id, namespace, visibility, owner,\
title, description, creator, generator, backend, model,\
prompt, provenance, live, updated_ms)\
VALUES({ID1_HEX},'rik2','public',NULL,'Legacy Lantern',\
'an oil lantern admitted before the kind column existed',\
'rik','trellis','cuda','xl','','',0,1);\
INSERT INTO search_labels(asset_id, kind, label) VALUES({ID1_HEX},'tag','durable');\
INSERT INTO search_postings(term, asset_id, weight_public, weight_owner)\
VALUES('legacy',{ID1_HEX},100,100),('lantern',{ID1_HEX},120,120),\
('oil',{ID1_HEX},20,20);\
PRAGMA user_version=1;"
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
let search = core.search();
// Pre-migration data is fully searchable; the retrofitted kind reads None.
let page = search.search(&q("lantern"), &ANYONE, None).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].title, "Legacy Lantern");
assert_eq!(page.hits[0].kind, None);
assert!(!page.hits[0].live);
let mut tagged = q("");
tagged.filters = SearchFilters { tag: Some("durable"), ..Default::default() };
assert_eq!(search.search(&tagged, &ANYONE, None).unwrap().total, 1);
let read = search.annotation(&asset_id_n(1)).unwrap().unwrap();
assert_eq!((read.title.as_str(), read.kind), ("Legacy Lantern", None));
// Legacy rows match no kind filter until re-annotated with one.
let mut by_kind = q("");
by_kind.filters = SearchFilters { kind: Some(AssetKind::Prop), ..Default::default() };
assert_eq!(search.search(&by_kind, &ANYONE, None).unwrap().total, 0);
let mut a = ann("Legacy Lantern");
a.kind = Some(AssetKind::Prop);
search.set_annotation(&asset_id_n(1), &a, NOW).unwrap();
let page = search.search(&by_kind, &ANYONE, None).unwrap();
assert_eq!((page.total, page.hits[0].kind), (1, Some(AssetKind::Prop)));
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
// A second open of the migrated root is a clean no-op.
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
/// The v2 search DDL exactly as it shipped: kind column present, but no
/// canon_alias column, no alias postings, no search_state row.
const LEGACY_SEARCH_V2: &str = "
CREATE TABLE IF NOT EXISTS search_annotations(
asset_id BLOB PRIMARY KEY,
namespace TEXT NOT NULL,
kind TEXT CHECK(kind IS NULL OR kind IN \
('mesh','character','weapon','vehicle','prop','texture','material',\
'audio','video','skybox','world','prefab')),
visibility TEXT NOT NULL CHECK(visibility IN ('public','private')),
owner BLOB,
title TEXT NOT NULL,
description TEXT NOT NULL,
creator TEXT NOT NULL,
generator TEXT NOT NULL,
backend TEXT NOT NULL,
model TEXT NOT NULL,
prompt TEXT NOT NULL,
provenance TEXT NOT NULL,
live INTEGER NOT NULL,
updated_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS search_annotations_by_ns ON search_annotations(namespace);
CREATE INDEX IF NOT EXISTS search_annotations_by_kind ON search_annotations(kind);
CREATE TABLE IF NOT EXISTS search_labels(
asset_id BLOB NOT NULL,
kind TEXT NOT NULL CHECK(kind IN ('category','tag')),
label TEXT NOT NULL,
PRIMARY KEY(asset_id, kind, label)
);
CREATE INDEX IF NOT EXISTS search_labels_by_label ON search_labels(kind, label);
CREATE TABLE IF NOT EXISTS search_postings(
term TEXT NOT NULL,
asset_id BLOB NOT NULL,
weight_public INTEGER NOT NULL,
weight_owner INTEGER NOT NULL,
PRIMARY KEY(term, asset_id)
);
CREATE INDEX IF NOT EXISTS search_postings_by_asset ON search_postings(asset_id);
";
#[test]
fn v2_root_gains_alias_index_with_data_backfilled() {
// A byte-real v2 root: search tables in their v2 shape, one annotated
// asset with an alias head already pointing at it (v2 kept `live`
// maintained but knew nothing of canonical aliases or alias terms).
let (root, db) = fixture_root(
"search_v2",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{LEGACY_SEARCH_V2}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
INSERT INTO asset_aliases(alias, asset_id, head_revision, updated_ms)\
VALUES('rik2/props/lamp',{ID1_HEX},X'{rev}',1);\
INSERT INTO search_annotations(asset_id, namespace, kind, visibility, owner,\
title, description, creator, generator, backend, model,\
prompt, provenance, live, updated_ms)\
VALUES({ID1_HEX},'rik2','prop','public',NULL,'Brass Lantern',\
'a lantern annotated under schema v2','rik','trellis','cuda','xl',\
'','',1,1);\
INSERT INTO search_labels(asset_id, kind, label) VALUES({ID1_HEX},'tag','durable');\
INSERT INTO search_postings(term, asset_id, weight_public, weight_owner)\
VALUES('brass',{ID1_HEX},100,100),('lantern',{ID1_HEX},120,120);\
PRAGMA user_version=2;",
rev = "22".repeat(32),
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
let search = core.search();
// The alias head became searchable terms and the canonical alias.
let page = search.search(&q("lamp"), &ANYONE, None).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].title, "Brass Lantern");
assert_eq!(page.hits[0].alias.as_deref(), Some("rik2/props/lamp"));
assert!(page.hits[0].live);
// Pre-migration annotation data is intact: postings, labels, kind, text.
assert_eq!(search.search(&q("brass"), &ANYONE, None).unwrap().total, 1);
let mut tagged = q("");
tagged.filters = SearchFilters { tag: Some("durable"), ..Default::default() };
assert_eq!(search.search(&tagged, &ANYONE, None).unwrap().total, 1);
let mut by_kind = q("");
by_kind.filters = SearchFilters { kind: Some(AssetKind::Prop), ..Default::default() };
assert_eq!(search.search(&by_kind, &ANYONE, None).unwrap().total, 1);
let read = search.annotation(&asset_id_n(1)).unwrap().unwrap();
assert_eq!((read.title.as_str(), read.kind), ("Brass Lantern", Some(AssetKind::Prop)));
// The index generation row exists and mutations advance it: cursors are
// fully live on a migrated root.
core.catalog().register_asset(&asset_id_n(2), "rik2", NOW).unwrap();
search.set_annotation(&asset_id_n(2), &ann("Brass Kettle"), NOW).unwrap();
let mut one = q("brass");
one.page_size = 1;
let cursor = search.search(&one, &ANYONE, None).unwrap().cursor.expect("two brass hits");
let page2 = search.search(&one, &ANYONE, Some(&cursor)).unwrap();
assert_eq!(page2.hits.len(), 1);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
assert!(read_generation(&db) >= 1, "state row seeded");
// A second open of the migrated root is a clean no-op.
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
#[test]
fn v3_root_gains_import_and_variant_tables() {
// A faithful v3 root: the current catalog/jobs/auth/search layout (v4
// added nothing to those), no import/variant tables, user_version=3.
let (root, db) = fixture_root(
"import_v3",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{search}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
PRAGMA user_version=3;",
search = makepad_asset_store::search::SEARCH_SCHEMA,
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
// The migrated root serves the new subsystems and kept its data.
assert!(core.imports().sources().unwrap().is_empty());
assert_eq!(
core.catalog().asset_namespace(&asset_id_n(1)).unwrap().as_deref(),
Some("rik2")
);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
let tables = raw::exec(
&db,
"SELECT name FROM sqlite_master WHERE type='table' AND name IN \
('import_sources','imports','import_entries','recipes','derivations',\
'derived_variants','variant_sets') ORDER BY name",
);
assert_eq!(tables.len(), 7, "v4 tables present: {tables:?}");
// A second open of the migrated root is a clean no-op.
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
#[test]
fn v4_root_gains_operation_tables() {
// A faithful v4 root: the current catalog/jobs/auth/search/import/variant
// layout, no operation tables, user_version=4.
let (root, db) = fixture_root(
"operations_v4",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{search}{import}{variant}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
PRAGMA user_version=4;",
search = makepad_asset_store::search::SEARCH_SCHEMA,
import = makepad_asset_store::imports::IMPORT_SCHEMA,
variant = makepad_asset_store::variants::VARIANT_SCHEMA,
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
// The queue subsystems are gone (aicore P7); the migrated root still
// opens and its CATALOG data survived — the only promise that remains.
assert_eq!(
core.catalog().asset_namespace(&asset_id_n(1)).unwrap().as_deref(),
Some("rik2")
);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
let tables = raw::exec(
&db,
"SELECT name FROM sqlite_master WHERE type='table' AND name IN \
('operations','operation_events','operation_worker_seen') ORDER BY name",
);
assert_eq!(tables.len(), 3, "v5 tables present: {tables:?}");
// A second open of the migrated root is a clean no-op.
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
#[test]
fn v7_root_gains_scale_indices_and_a_sharded_cas() {
// A faithful v7 root: today's tables minus the three indices v8 adds,
// plus a CAS object at the pre-v8 one-level path.
let (root, db) = fixture_root(
"scale_v7",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{search}{import}{variant}{operations}\
DROP INDEX IF EXISTS asset_aliases_by_asset;\
DROP INDEX IF EXISTS game_aliases_by_game;\
DROP INDEX IF EXISTS search_annotations_by_canon;\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
PRAGMA user_version=7;",
search = makepad_asset_store::search::SEARCH_SCHEMA,
import = makepad_asset_store::imports::IMPORT_SCHEMA,
variant = makepad_asset_store::variants::VARIANT_SCHEMA,
operations = HISTORICAL_OPERATIONS_SCHEMA,
),
);
let payload = b"blob written by a v7 server".to_vec();
let blob_id = makepad_asset_data::BlobId::hash_of(&payload);
let hex: String = blob_id.as_bytes().iter().map(|b| format!("{b:02x}")).collect();
let legacy_dir = root.join("cas/objects").join(&hex[..2]);
std::fs::create_dir_all(&legacy_dir).unwrap();
std::fs::write(legacy_dir.join(&hex), &payload).unwrap();
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(
core.catalog().asset_namespace(&asset_id_n(1)).unwrap().as_deref(),
Some("rik2"),
"migration kept the catalog data"
);
// The object moved into its two-level hash path and still verifies.
assert!(root
.join("cas/objects")
.join(&hex[..2])
.join(&hex[2..4])
.join(&hex)
.is_file());
assert!(!legacy_dir.join(&hex).is_file(), "one-level copy is gone");
assert_eq!(core.cas().read_verified(&blob_id).unwrap(), payload);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
let indices = raw::exec(
&db,
"SELECT name FROM sqlite_master WHERE type='index' AND name IN \
('asset_aliases_by_asset','game_aliases_by_game','search_annotations_by_canon') \
ORDER BY name",
);
assert_eq!(indices.len(), 3, "v8 indices present: {indices:?}");
// A second open of the migrated root is a clean no-op.
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
/// The plans the v8 indices exist to produce. A schema that grows a new
/// ordering column or filter without an index would show up here as a
/// re-appearing SCAN or TEMP B-TREE.
#[test]
fn hot_paths_use_the_scale_indices() {
let root = test_root("scale_plans");
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
let db = root.join("catalog.sqlite3");
let plan = |sql: &str| raw::exec_last(&db, &format!("EXPLAIN QUERY PLAN {sql}")).join(" | ");
// Browse page: index-ordered, no sort, no scan.
let browse = plan(
"SELECT a.asset_id FROM search_annotations a WHERE 1=1 \
AND (a.visibility = 'public' OR (a.owner IS NOT NULL AND a.owner = NULL)) \
ORDER BY a.canon_alias ASC, a.asset_id ASC LIMIT 61",
);
assert!(browse.contains("search_annotations_by_canon"), "{browse}");
assert!(!browse.contains("TEMP B-TREE"), "browse page still sorts: {browse}");
// Alias-head reads by asset: seek, not scan.
for sql in [
"SELECT alias FROM asset_aliases WHERE asset_id = X'01' ORDER BY alias",
"SELECT MIN(alias) FROM asset_aliases WHERE asset_id = X'01'",
"DELETE FROM asset_aliases WHERE asset_id = X'01' AND head_revision = X'02'",
] {
let p = plan(sql);
assert!(p.contains("asset_aliases_by_asset"), "{sql} => {p}");
}
}
/// The v11 annotation table exactly before `data` widened its kind CHECK.
const LEGACY_SEARCH_V11_ANNOTATIONS: &str = "
CREATE TABLE search_annotations(
asset_id BLOB PRIMARY KEY,
namespace TEXT NOT NULL,
kind TEXT CHECK(kind IS NULL OR kind IN
('mesh','character','weapon','vehicle','prop','texture','material',
'audio','video','skybox','world','prefab','billboard','game','vjeffect')),
visibility TEXT NOT NULL CHECK(visibility IN ('public','private')),
owner BLOB,
title TEXT NOT NULL,
description TEXT NOT NULL,
creator TEXT NOT NULL,
generator TEXT NOT NULL,
backend TEXT NOT NULL,
model TEXT NOT NULL,
prompt TEXT NOT NULL,
provenance TEXT NOT NULL,
live INTEGER NOT NULL,
updated_ms INTEGER NOT NULL,
canon_alias TEXT NOT NULL DEFAULT ''
);
CREATE INDEX search_annotations_by_ns ON search_annotations(namespace);
CREATE INDEX search_annotations_by_kind ON search_annotations(kind);
CREATE INDEX search_annotations_by_canon ON search_annotations(canon_alias, asset_id);
";
#[test]
fn v11_root_migrates_kind_check_to_appended_kinds_with_rows_intact() {
let (root, db) = fixture_root(
"data_kind_v11",
&format!(
"{CATALOG_SCHEMA}{HISTORICAL_JOBS_SCHEMA}{AUTH_SCHEMA}{LEGACY_SEARCH_V11_ANNOTATIONS}\
INSERT INTO assets(asset_id, namespace, created_ms) VALUES({ID1_HEX},'rik2',1);\
INSERT INTO search_annotations(asset_id, namespace, kind, visibility, owner,\
title, description, creator, generator, backend, model, prompt, provenance,\
live, updated_ms, canon_alias)\
VALUES({ID1_HEX},'rik2','vjeffect','public',NULL,'Legacy document','','','','',\
'','','',0,1,'');\
PRAGMA user_version=11;"
),
);
let core = AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
let before = core.search().annotation(&asset_id_n(1)).unwrap().unwrap();
assert_eq!((before.title.as_str(), before.kind), ("Legacy document", Some(AssetKind::VjEffect)));
let mut data = ann("Migrated data document");
data.kind = Some(AssetKind::Data);
core.search().set_annotation(&asset_id_n(1), &data, NOW).unwrap();
let mut by_data = q("");
by_data.filters = SearchFilters { kind: Some(AssetKind::Data), ..Default::default() };
let page = core.search().search(&by_data, &ANYONE, None).unwrap();
assert_eq!((page.total, page.hits[0].kind), (1, Some(AssetKind::Data)));
let mut model_program = ann("Migrated editable CSG model");
model_program.kind = Some(AssetKind::ModelProgram);
core.search().set_annotation(&asset_id_n(1), &model_program, NOW + 1).unwrap();
let mut by_model_program = q("");
by_model_program.filters = SearchFilters {
kind: Some(AssetKind::ModelProgram),
..Default::default()
};
let page = core.search().search(&by_model_program, &ANYONE, None).unwrap();
assert_eq!(
(page.total, page.hits[0].kind),
(1, Some(AssetKind::ModelProgram))
);
drop(core);
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
let ddl = raw::exec(
&db,
"SELECT sql FROM sqlite_master WHERE type='table' AND name='search_annotations'",
);
assert!(ddl[0].contains("'data'"), "migrated CHECK: {}", ddl[0]);
assert!(ddl[0].contains("'model-program'"), "migrated CHECK: {}", ddl[0]);
AssetServerCore::open(&root, Budgets::default_v1()).unwrap();
assert_eq!(user_version(&db), SERVER_SCHEMA_VERSION.to_string());
}
#[test]
fn future_and_negative_schema_versions_refuse_to_open() {
let (root, db) = fixture_root("future_version", "PRAGMA user_version=99;");
match AssetServerCore::open(&root, Budgets::default_v1()).err() {
Some(ServerError::UnsupportedSchema { found: 99 }) => {}
other => panic!("expected UnsupportedSchema 99, got {other:?}"),
}
// The refused open left no schema behind.
assert_eq!(raw::exec(&db, "SELECT name FROM sqlite_master WHERE type='table'"), Vec::<String>::new());
let (root, _db) = fixture_root("negative_version", "PRAGMA user_version=-3;");
match AssetServerCore::open(&root, Budgets::default_v1()).err() {
Some(ServerError::UnsupportedSchema { found: -3 }) => {}
other => panic!("expected UnsupportedSchema -3, got {other:?}"),
}
}