makepad/tools/map_tiles/README.md

Makepad map tile tools

makepad-map-tiles prepares two complementary, disk-streamable map archives:

• a compact Shortbread base pyramid copied from VersaTiles; and
• an all-tag street-detail overlay built directly from a sorted OSM PBF.

Both outputs are standard gzip-MVT MBTiles files. The pure-Rust MBTiles writer orders rows in 256×256 tile blocks and records a makepad_rowid_scheme marker, so Makepad can seek directly to a requested tile without enumerating a zoom level. The raw PBF remains the canonical copy of the complete OSM graph.

Native OSM detail conversion

cargo build --release --package makepad-map-tiles

target/release/makepad-map-tiles pbf-detail \
    local/maps/europe-260726.osm.pbf \
    local/maps/europe-osm-detail.partial.mbtiles \
    --store local/maps/native-detail-europe.store \
    --zoom 14 \
    --sort-memory-mib 256

The input must advertise the standard PBF Sort.Type_then_ID optional feature. The converter performs bounded-memory sequential passes:

1. build a bounded relation-way bitset and persist it for the later passes;
2. project each node once and build a compressed, indexed coordinate store;
3. resolve ways, retain relation-member way references, and clip tagged ways;
4. assemble tagged relation geometry, including multipolygon rings; and
5. externally sort 256×256 tile-block spools, encode MVT, gzip each tile, and stream the tiles into MBTiles.

No feature/tag allow-list or geometry simplification is applied. Every tagged node is emitted as a point. Every tagged way is emitted as a line; a closed way is also emitted as a polygon candidate so the conversion does not make an irreversible area interpretation. Tagged relation members are emitted as relation point/line geometry, and type=multipolygon relations also receive assembled polygon geometry.

The six MVT layers are:

• osm_points
• osm_lines
• osm_polygons
• osm_relation_points
• osm_relation_lines
• osm_relation_polygons

Each feature retains every original OSM tag as a string plus its MVT feature ID and the internal fields __makepad_osm_id, __makepad_osm_type, and __makepad_osm_closed.

Simple 3D Buildings tags are copied verbatim, including building, building:part, height, min_height, building:levels, building:min_level, roof shape/height/levels/direction/orientation/angle, building and roof materials, and building and roof colours. Keeping the raw strings avoids silently changing values with units; the renderer can normalize them once when preparing a 2.5D building mesh and fall back from explicit height to level counts.

The scratch directory is deliberately retained after a successful build. It contains compressed node/way indexes, the relation-way bitset, source block spools, native-detail-audit.txt, and an atomic spool.complete.json marker. If final MBTiles writing is interrupted, move the partial MBTiles aside and repeat the same command: the converter validates the source size and zoom, then restarts only the external-sort/MBTiles pass. An incomplete earlier pass is never guessed at or silently reused.

Performance notes

The hot path is designed around the scale of the pinned Europe extract: 3.79 billion nodes, 464 million ways, and 4.79 billion way-node references.

• node/way group indexes use bounded direct lookup tables;
• compressed coordinate groups are decoded once into an O(1) cache;
• Web Mercator projection happens once per node, not once per way reference;
• ordinary ways resolve directly from the PBF iterator without allocating a second node-reference vector;
• source tags borrow the PBF string table and use inline storage for the common case, avoiding per-tag string copies and most small heap allocations;
• tile-block writers use hash lookup and retain enough buffered handles to cover the Europe block set without per-feature reopen churn;
• one reusable scratch buffer encodes records without allocating a payload vector for every tile record;
• relation-member way references are borrowed from decoded groups rather than cloned; and
• MVT dictionaries use deterministic insertion order with fast lookup.

On the same machine used for development, the release converter processed the 47,228,875-byte Luxembourg extract (4,208,128 nodes, 6,313,908 way-node references, and 1,617,855 tile records) into 3,953 checked tiles in 12.3 seconds. Pass 3 sustained about 1.29 million resolved references per second. Scaling the measured stages by the audited Europe object/reference counts gives a practical preflight estimate of roughly 2–3 hours, rather than four-plus, though final time depends on SSD throughput and tile density.

Other commands

# Inspect source ordering and bounds.
target/release/makepad-map-tiles inspect-pbf source.osm.pbf

# Count all source tags plus benches, playgrounds, buildings and 2.5D fields.
target/release/makepad-map-tiles audit-pbf source.osm.pbf

# Verify direct MBTiles lookup, gzip/MVT validity, layers, and detail tag keys.
target/release/makepad-map-tiles probe-mbtiles map.mbtiles 14/8414/5384

# Extract the Europe rectangle from a VersaTiles Shortbread planet archive.
target/release/makepad-map-tiles versatiles \
    source.versatiles output.mbtiles \
    --bbox=-32.683233,29.635548,46.753480,81.472990 \
    --max-zoom 14

download_map.sh pins and verifies the large upstream snapshots, builds the release utility, validates SQLite, probes a real MVT tile, and only then atomically renames a .partial.mbtiles output.