fc63b33a84
Conversion tooling (tools/map_tiles): curated Shortbread base pyramid
from a VersaTiles planet archive (bbox extract, brotli->gzip transcode
parallelized per 256x256 block), all-tag native OSM pbf detail
converter, pbf audit, and nav-build/nav-probe producing the routing
graph + search index artifacts. download_map.sh orchestrates pinned
downloads and conversions. mbtile_reader writer now skips SQLite's
lock-byte page at byte offset 1 GiB — allocating through it corrupted
every database over 1 GiB ("2nd reference to page 16385"); the 31 GB
Europe conversion passes integrity checks.
Navigation (libs/map_nav, new): region.search place/POI/street/address
index (prefix autocomplete, NL/EN category synonyms, proximity
ranking) and region.graph routing graph (CSR directed edges,
car/bike/foot speeds, oneway, turn restrictions, snap grid, A*),
maneuver generation and the NavSession map-matching state machine;
26 unit tests.
MapView interaction layer: MapViewAction, camera API + animated
fly_to, overlay.rs (route polyline with traveled dimming, markers,
position puck), per-widget mbtiles_path override, archive
minzoom/maxzoom honored for tile requests, zoom-level cross-fade over
the previous level's imagery, floating panels win hit-testing.
examples/map is the navigator app: worker-thread search, Drive/Bike/
Walk routing, simulated turn-by-turn with banner, follow camera.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
121 lines
5.4 KiB
Markdown
121 lines
5.4 KiB
Markdown
# Makepad map tile tools
|
||
|
||
`makepad-map-tiles` prepares two complementary, disk-streamable map archives:
|
||
|
||
- a compact Shortbread base pyramid copied from VersaTiles; and
|
||
- an all-tag street-detail overlay built directly from a sorted OSM PBF.
|
||
|
||
Both outputs are standard gzip-MVT MBTiles files. The pure-Rust MBTiles writer
|
||
orders rows in 256×256 tile blocks and records a `makepad_rowid_scheme` marker,
|
||
so Makepad can seek directly to a requested tile without enumerating a zoom
|
||
level. The raw PBF remains the canonical copy of the complete OSM graph.
|
||
|
||
## Native OSM detail conversion
|
||
|
||
```sh
|
||
cargo build --release --package makepad-map-tiles
|
||
|
||
target/release/makepad-map-tiles pbf-detail \
|
||
local/maps/europe-260726.osm.pbf \
|
||
local/maps/europe-osm-detail.partial.mbtiles \
|
||
--store local/maps/native-detail-europe.store \
|
||
--zoom 14 \
|
||
--sort-memory-mib 256
|
||
```
|
||
|
||
The input must advertise the standard PBF `Sort.Type_then_ID` optional feature.
|
||
The converter performs bounded-memory sequential passes:
|
||
|
||
1. build a bounded relation-way bitset and persist it for the later passes;
|
||
2. project each node once and build a compressed, indexed coordinate store;
|
||
3. resolve ways, retain relation-member way references, and clip tagged ways;
|
||
4. assemble tagged relation geometry, including multipolygon rings; and
|
||
5. externally sort 256×256 tile-block spools, encode MVT, gzip each tile, and
|
||
stream the tiles into MBTiles.
|
||
|
||
No feature/tag allow-list or geometry simplification is applied. Every tagged
|
||
node is emitted as a point. Every tagged way is emitted as a line; a closed way
|
||
is also emitted as a polygon candidate so the conversion does not make an
|
||
irreversible `area` interpretation. Tagged relation members are emitted as
|
||
relation point/line geometry, and `type=multipolygon` relations also receive
|
||
assembled polygon geometry.
|
||
|
||
The six MVT layers are:
|
||
|
||
- `osm_points`
|
||
- `osm_lines`
|
||
- `osm_polygons`
|
||
- `osm_relation_points`
|
||
- `osm_relation_lines`
|
||
- `osm_relation_polygons`
|
||
|
||
Each feature retains every original OSM tag as a string plus its MVT feature ID
|
||
and the internal fields `__makepad_osm_id`, `__makepad_osm_type`, and
|
||
`__makepad_osm_closed`.
|
||
|
||
Simple 3D Buildings tags are copied verbatim, including `building`,
|
||
`building:part`, `height`, `min_height`, `building:levels`,
|
||
`building:min_level`, roof shape/height/levels/direction/orientation/angle,
|
||
building and roof materials, and building and roof colours. Keeping the raw
|
||
strings avoids silently changing values with units; the renderer can normalize
|
||
them once when preparing a 2.5D building mesh and fall back from explicit
|
||
height to level counts.
|
||
|
||
The scratch directory is deliberately retained after a successful build. It
|
||
contains compressed node/way indexes, the relation-way bitset, source block
|
||
spools, `native-detail-audit.txt`, and an atomic `spool.complete.json` marker.
|
||
If final MBTiles writing is interrupted, move the partial MBTiles aside and
|
||
repeat the same command: the converter validates the source size and zoom, then
|
||
restarts only the external-sort/MBTiles pass. An incomplete earlier pass is
|
||
never guessed at or silently reused.
|
||
|
||
## Performance notes
|
||
|
||
The hot path is designed around the scale of the pinned Europe extract:
|
||
3.79 billion nodes, 464 million ways, and 4.79 billion way-node references.
|
||
|
||
- node/way group indexes use bounded direct lookup tables;
|
||
- compressed coordinate groups are decoded once into an O(1) cache;
|
||
- Web Mercator projection happens once per node, not once per way reference;
|
||
- ordinary ways resolve directly from the PBF iterator without allocating a
|
||
second node-reference vector;
|
||
- source tags borrow the PBF string table and use inline storage for the common
|
||
case, avoiding per-tag string copies and most small heap allocations;
|
||
- tile-block writers use hash lookup and retain enough buffered handles to
|
||
cover the Europe block set without per-feature reopen churn;
|
||
- one reusable scratch buffer encodes records without allocating a payload
|
||
vector for every tile record;
|
||
- relation-member way references are borrowed from decoded groups rather than
|
||
cloned; and
|
||
- MVT dictionaries use deterministic insertion order with fast lookup.
|
||
|
||
On the same machine used for development, the release converter processed the
|
||
47,228,875-byte Luxembourg extract (4,208,128 nodes, 6,313,908 way-node
|
||
references, and 1,617,855 tile records) into 3,953 checked tiles in 12.3
|
||
seconds. Pass 3 sustained about 1.29 million resolved references per second.
|
||
Scaling the measured stages by the audited Europe object/reference counts gives
|
||
a practical preflight estimate of roughly 2–3 hours, rather than four-plus,
|
||
though final time depends on SSD throughput and tile density.
|
||
|
||
## Other commands
|
||
|
||
```sh
|
||
# Inspect source ordering and bounds.
|
||
target/release/makepad-map-tiles inspect-pbf source.osm.pbf
|
||
|
||
# Count all source tags plus benches, playgrounds, buildings and 2.5D fields.
|
||
target/release/makepad-map-tiles audit-pbf source.osm.pbf
|
||
|
||
# Verify direct MBTiles lookup, gzip/MVT validity, layers, and detail tag keys.
|
||
target/release/makepad-map-tiles probe-mbtiles map.mbtiles 14/8414/5384
|
||
|
||
# Extract the Europe rectangle from a VersaTiles Shortbread planet archive.
|
||
target/release/makepad-map-tiles versatiles \
|
||
source.versatiles output.mbtiles \
|
||
--bbox=-32.683233,29.635548,46.753480,81.472990 \
|
||
--max-zoom 14
|
||
```
|
||
|
||
`download_map.sh` pins and verifies the large upstream snapshots, builds the
|
||
release utility, validates SQLite, probes a real MVT tile, and only then
|
||
atomically renames a `.partial.mbtiles` output.
|